EEW Evaluation

From SCECpedia
Revision as of 04:10, 1 May 2011 by Maechlin (talk | contribs)
Jump to navigationJump to search

Example Event Forecast Performance

Event ID: 10922333 Magnitude: 4.13 Origin Time: 2011/04/09 08:58:49.22 Hypocenter: 32.6055 -115.7237 9.38 km Forecast Source: smi://caltech.gps.edu/onsite/cinnabar/vs/eewvs Seconds after Origin: 17 Network:Station Forecast Time Dist(km) PGV(obs) PGV(fcst)m/s Int(obs) Int(fcst)MMI abs(Io-If) fcst Skill obs GM fcst GM 

 CI:WES   2011-04-09T08:59:06Z    17.3   0.0204    0.0021     5.22     5.13       0.09   Very Good       Very strong shaking/moderate damage       Very strong shaking/moderate damage
 CI:DRE   2011-04-09T08:59:06Z    34.0   0.0109    0.0003     4.89     3.30       1.59        Poor        Moderate shaking/very light damage     Strong shaking/light damage potential
 CI:SWS   2011-04-09T08:59:06Z    38.6   0.0021    0.0004     2.96     3.44       0.48   Very Good        Moderate shaking/very light damage         Light shaking/no damage potential
 CI:ERR   2011-04-09T08:59:06Z    57.7   0.0046    0.0002     3.88     2.70       1.19    Moderate         Light shaking/no damage potential        Moderate shaking/very light damage
 CI:RXH   2011-04-09T08:59:06Z    64.9   0.0018    0.0002     2.77     2.46       0.31   Very Good         Light shaking/no damage potential         Light shaking/no damage potential
 CI:BTC   2011-04-09T08:59:06Z    65.2   0.0021    0.0001     3.06     2.32       0.74        Good         Light shaking/no damage potential        Moderate shaking/very light damage

AZ:MONP2 2011-04-09T08:59:06Z 73.1 0.0014 0.0001 2.36 2.22 0.14 Very Good Light shaking/no damage potential Light shaking/no damage potential 

 CI:SAL   2011-04-09T08:59:06Z    79.0   0.0058    0.0001     3.78     2.15       1.63        Poor         Light shaking/no damage potential        Moderate shaking/very light damage
 CI:JEM   2011-04-09T08:59:06Z    97.7   0.0017    0.0001     2.59     1.75       0.83        Good          Weak shaking/no damage potential         Light shaking/no damage potential
 CI:BOR   2011-04-09T08:59:06Z    98.4   0.0011    0.0001     2.74     1.76       0.97        Good          Weak shaking/no damage potential         Light shaking/no damage potential
CI:NSS2   2011-04-09T08:59:06Z   107.6   0.0006    0.0001     1.49     1.62       0.13   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 CI:EML   2011-04-09T08:59:06Z   110.2   0.0011    0.0001     3.22     1.52       1.69        Poor          Weak shaking/no damage potential        Moderate shaking/very light damage
 CI:OLP   2011-04-09T08:59:06Z   113.6   0.0021    0.0001     3.20     1.43       1.76        Poor          Weak shaking/no damage potential        Moderate shaking/very light damage
AZ:LVA2   2011-04-09T08:59:06Z   114.3   0.0011    0.0001     2.11     1.51       0.61        Good          Weak shaking/no damage potential         Light shaking/no damage potential
 CI:SDR   2011-04-09T08:59:06Z   115.5   0.0009    0.0001     2.62     1.42       1.20    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
 CI:TOR   2011-04-09T08:59:06Z   117.6   0.0004    0.0001     1.42     1.47       0.05   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 CI:BC3   2011-04-09T08:59:06Z   119.2   0.0004    0.0001     1.53     1.42       0.11   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 CI:CTC   2011-04-09T08:59:06Z   119.3   0.0008    0.0001     1.52     1.44       0.08   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 AZ:TRO   2011-04-09T08:59:06Z   121.4   0.0005    0.0001     1.64     1.41       0.23   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 CI:DPP   2011-04-09T08:59:06Z   122.5   0.0012    0.0001     2.68     1.35       1.32    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
 AZ:HWB   2011-04-09T08:59:06Z   125.2   0.0003    0.0001     1.53     1.32       0.21   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 AZ:FRD   2011-04-09T08:59:06Z   128.6   0.0003    0.0001     0.85     1.31       0.45   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
 CI:PMD   2011-04-09T08:59:06Z   131.0   0.0007    0.0001     2.23     1.28       0.95        Good          Weak shaking/no damage potential         Light shaking/no damage potential
 AZ:PFO   2011-04-09T08:59:06Z   131.4   0.0002    0.0000     1.01     1.27       0.26   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 AZ:BZN   2011-04-09T08:59:06Z   132.3   0.0009    0.0000     1.99     1.26       0.73        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 AZ:CPE   2011-04-09T08:59:06Z   133.6   0.0018    0.0000     2.80     1.19       1.61        Poor          Weak shaking/no damage potential         Light shaking/no damage potential
 AZ:SND   2011-04-09T08:59:06Z   134.2   0.0011    0.0000     1.77     1.23       0.53        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 CI:SDG   2011-04-09T08:59:06Z   134.4   0.0020    0.0000     2.53     1.17       1.35    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
 CI:PLM   2011-04-09T08:59:06Z   135.3   0.0008    0.0000     1.65     1.21       0.44   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 CI:DNR   2011-04-09T08:59:06Z   136.5   0.0020    0.0000     2.34     1.20       1.13    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
 AZ:WMC   2011-04-09T08:59:06Z   139.7   0.0009    0.0000     2.22     1.16       1.06    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
 AZ:CRY   2011-04-09T08:59:06Z   142.8   0.0003    0.0000     0.77     1.12       0.36   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
 AZ:SOL   2011-04-09T08:59:06Z   145.6   0.0025    0.0000     2.46     1.04       1.42    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
 CI:MGE   2011-04-09T08:59:06Z   147.7   0.0005    0.0000     1.81     1.07       0.74        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
 CI:GOR   2011-04-09T08:59:06Z   153.8   0.0004    0.0000     1.73     0.97       0.76        Good              Not felt/no damage potential          Weak shaking/no damage potential
 AZ:KNW   2011-04-09T08:59:06Z   154.0   0.0002    0.0000     0.69     0.99       0.30   Very Good              Not felt/no damage potential              Not felt/no damage potential
 AZ:RDM   2011-04-09T08:59:06Z   155.0   0.0013    0.0000     1.92     0.98       0.94        Good              Not felt/no damage potential          Weak shaking/no damage potential
 CI:PSD   2011-04-09T08:59:06Z   155.9   0.0003    0.0000     1.33     0.98       0.35   Very Good              Not felt/no damage potential          Weak shaking/no damage potential
 CI:BEL   2011-04-09T08:59:06Z   157.1   0.0001    0.0000     0.00     0.96       0.96        Good              Not felt/no damage potential              Not felt/no damage potential
 CI:DEV   2011-04-09T08:59:06Z   168.0   0.0004    0.0000     1.51     0.84       0.67        Good              Not felt/no damage potential          Weak shaking/no damage potential
 CI:BLY   2011-04-09T08:59:06Z   169.2   0.0001    0.0000     0.39     0.78       0.38   Very Good              Not felt/no damage potential              Not felt/no damage potential
 CI:SLR   2011-04-09T08:59:06Z   169.4   0.0006    0.0000     1.56     0.83       0.74        Good              Not felt/no damage potential          Weak shaking/no damage potential
 CI:MUR   2011-04-09T08:59:06Z   176.5   0.0021    0.0000     2.56     0.75       1.82        Poor              Not felt/no damage potential         Light shaking/no damage potential
 CI:MSJ   2011-04-09T08:59:06Z   177.1   0.0012    0.0000     2.12     0.75       1.37    Moderate              Not felt/no damage potential         Light shaking/no damage potential
 CI:IRM   2011-04-09T08:59:06Z   180.4   0.0001    0.0000     0.00     0.70       0.70        Good              Not felt/no damage potential              Not felt/no damage potential
 CI:MCT   2011-04-09T08:59:06Z   182.4   0.0000    0.0000     0.00     0.70       0.70        Good              Not felt/no damage potential              Not felt/no damage potential
 CI:BBS   2011-04-09T08:59:06Z   187.5   0.0005    0.0000     0.72     0.65       0.07   Very Good              Not felt/no damage potential              Not felt/no damage potential
 CI:PER   2011-04-09T08:59:06Z   196.5   0.0004    0.0000     1.23     0.56       0.67        Good              Not felt/no damage potential          Weak shaking/no damage potential

Summary of 48 site-specific forecast ratings for event:10922333 Very Good: 37.5% Good : 33.3% Moderate : 16.7% Poor : 12.5% Very Poor: 0.0%


Performance Evaluation Summaries

Two descriptions of performance summaries include:

  • CISN Testing Document
  • CTC Group Evaluation Collection

We separate the summaries into "Event" Summaries and "Catalog" Summaries where catalog means any set of earthquakes, and the summary must make it clear what criteria were used to define the catalog.

CISN On Site Trigger

An example CISN Trigger file is posted here:

We noted several things about the current formatting.

  1. The report id never changes. The intention was that this uniquely identified the file. But it is currently fixed.
  2. Forecasts PGV and Mag only
  3. Reports both algorithm and alert time, which always seem to have a delta of 2 seconds. This may mean difference between the two is the amount of data needed to produce the alert
  4. Multiple forecasts per trigger file
  5. Multiple events per trigger file
  6. Seems to report duplicate forecast information for each trigger. One copy of the trigger also contains observational data for the trigger. It is unclear how the trigger is match to the event id. We noted that in some cases the event id reported in the trigger file is not the final ANSS event id.
  7. The event id does not have a network id so we assume it is CI network event

VS Trigger Reports

VS is running on southern California Events. We noted the following.

  1. Reports one event per trigger file
  2. Each trigger file reports source of report
  3. Each trigger file contains a set of time series
  4. It seems that all time series in the trigger report have the same timestamps
  5. Event oriented time series include:
    1. magnitude
    2. lat
    3. lon
    4. origin time
  6. Region-oriented time series include
    1. pgv
    2. pga

ShakeMap Data

We are currently working with the ShakeMap stationlist txt file. We may convert to the stationlist xml files, but it was easier to parse the column oriented to start.

We use the ShakeMap data as reference observational data for station amplitude values.

We have identified several characteristics to the ShakeMap observational reports.

  1. Two main formats are the CI: and NN: events, and ECX events. ECX events have their pgv and pga values in different columns than the others, so this type of ShakeMap report must be checked for while parsing.
  2. ShakeMap files also have different number of columns populated. Currently we require that both PGA and PGV are listed before we use it as a authorized data observation. The goal is to maximize the number of observations we have to compare to forecasts. Right now, we don't have forecasts at PGV or PGA only sites. Once we do, we should add small modification to track observations even if they only have one component of data.
  3. We pull event ID from the header of the Shakemap file. However, it does not list the network, which we need to match it up in the ANSS catalog.

Processing Flow

Current design is based on processing one or more trigger reports. Currently two types of trigger reports are supported CIT On Site, and ETH Virtual Seismologist.

  1. Init CISN Station list, ANSS EQ catalog, ShakeMap catalog
  2. For each Trigger File
    1. Mapped to One or More ANSS Events and observed data from ShakeMap
  3. For each event in trigger file
    1. write event summary
    2. Add to catalog summaries
  4. write catalog summaries

ANSS Catalog Files

We create a catalog of ANSS events based on the ANSS web-based catalog search capabilities. The query parameters for the ANSS catalog are in the header of the file. This catalog represents the authorized reference data for events in the testing region. While we use ShakeMap data files to get station information, we use ANSS catalog as the reference event catalog.

  • Example ANSS Catalog File

Software Download

CTC Software is posted in git repository on github. maechlin:ctc

Related Entries

See Also

Retrieved from "https://strike.scec.org/scecwiki/index.php?title=EEW_Evaluation&oldid=2623"