EEW Evaluation
From SCECpedia
Contents
- 1 Example On Site Trigger Forecast Performance
- 2 Example Virtual Seismologist Forecast Performance
- 3 Intensity Equations
- 4 Performance Evaluation Summaries
- 5 CISN On Site Trigger
- 6 VS Trigger Reports
- 7 ShakeMap Data
- 8 Processing Flow
- 9 ANSS Catalog Files
- 10 Software Download
- 11 Related Entries
- 12 See Also
Example On Site Trigger Forecast Performance
Event ID: 10922333 Magnitude: 4.13 Origin Time: 2011/04/09 08:58:49.22 Hypocenter: 32.6055 -115.7237 9.38 km Forecast Source: ssi://caltech.gps.edu/onsite/2.1.0 Forecast Time: 2011-4-9T8:59:21Z Seconds after Origin: 32 Net:Sta Dist(km) PGV(obs) PGV(fcst)cm/second I(obs) I(fcst)MMI abs(Io-If) Skill obs GM fcst GM CI:WES 17.3 0.0204 0.3816 0.20 3.82 3.61 Very Poor Moderate shaking/very light damage Not felt/no damage potential CI:DRE 34.0 0.0109 0.2022 0.11 2.02 1.91 Poor Light shaking/no damage potential Not felt/no damage potential CI:IBP 35.5 0.0043 0.1042 0.04 1.04 1.00 Good Weak shaking/no damage potential Not felt/no damage potential CI:ERR 57.7 0.0046 0.1317 0.05 1.32 1.27 Moderate Weak shaking/no damage potential Not felt/no damage potential Summary of 4 site-specific forecast ratings for event:10922333 Very Good: 0.0% Good : 25.0% Moderate : 25.0% Poor : 25.0% Very Poor: 25.0%
Example Virtual Seismologist Forecast Performance
Event ID: 10922333 Magnitude: 4.13 Origin Time: 2011/04/09 08:58:49.22 Hypocenter: 32.6055 -115.7237 9.38 km Forecast Source: smi://caltech.gps.edu/onsite/cinnabar/vs/eewvs Seconds after Origin: 17 Network:Station Forecast Time Dist(km) PGV(obs) PGV(fcst)m/s Int(obs) Int(fcst)MMI abs(Io-If) fcst Skill obs GM fcst GM CI:WES 2011-04-09T08:59:06Z 17.3 0.0204 0.0021 5.22 5.13 0.09 Very Good Very strong shaking/moderate damage Very strong shaking/moderate damage CI:DRE 2011-04-09T08:59:06Z 34.0 0.0109 0.0003 4.89 3.30 1.59 Poor Moderate shaking/very light damage Strong shaking/light damage potential CI:SWS 2011-04-09T08:59:06Z 38.6 0.0021 0.0004 2.96 3.44 0.48 Very Good Moderate shaking/very light damage Light shaking/no damage potential CI:ERR 2011-04-09T08:59:06Z 57.7 0.0046 0.0002 3.88 2.70 1.19 Moderate Light shaking/no damage potential Moderate shaking/very light damage CI:RXH 2011-04-09T08:59:06Z 64.9 0.0018 0.0002 2.77 2.46 0.31 Very Good Light shaking/no damage potential Light shaking/no damage potential CI:BTC 2011-04-09T08:59:06Z 65.2 0.0021 0.0001 3.06 2.32 0.74 Good Light shaking/no damage potential Moderate shaking/very light damage AZ:MONP2 2011-04-09T08:59:06Z 73.1 0.0014 0.0001 2.36 2.22 0.14 Very Good Light shaking/no damage potential Light shaking/no damage potential CI:SAL 2011-04-09T08:59:06Z 79.0 0.0058 0.0001 3.78 2.15 1.63 Poor Light shaking/no damage potential Moderate shaking/very light damage CI:JEM 2011-04-09T08:59:06Z 97.7 0.0017 0.0001 2.59 1.75 0.83 Good Weak shaking/no damage potential Light shaking/no damage potential CI:BOR 2011-04-09T08:59:06Z 98.4 0.0011 0.0001 2.74 1.76 0.97 Good Weak shaking/no damage potential Light shaking/no damage potential CI:NSS2 2011-04-09T08:59:06Z 107.6 0.0006 0.0001 1.49 1.62 0.13 Very Good Weak shaking/no damage potential Weak shaking/no damage potential CI:EML 2011-04-09T08:59:06Z 110.2 0.0011 0.0001 3.22 1.52 1.69 Poor Weak shaking/no damage potential Moderate shaking/very light damage CI:OLP 2011-04-09T08:59:06Z 113.6 0.0021 0.0001 3.20 1.43 1.76 Poor Weak shaking/no damage potential Moderate shaking/very light damage AZ:LVA2 2011-04-09T08:59:06Z 114.3 0.0011 0.0001 2.11 1.51 0.61 Good Weak shaking/no damage potential Light shaking/no damage potential CI:SDR 2011-04-09T08:59:06Z 115.5 0.0009 0.0001 2.62 1.42 1.20 Moderate Weak shaking/no damage potential Light shaking/no damage potential CI:TOR 2011-04-09T08:59:06Z 117.6 0.0004 0.0001 1.42 1.47 0.05 Very Good Weak shaking/no damage potential Weak shaking/no damage potential CI:BC3 2011-04-09T08:59:06Z 119.2 0.0004 0.0001 1.53 1.42 0.11 Very Good Weak shaking/no damage potential Weak shaking/no damage potential CI:CTC 2011-04-09T08:59:06Z 119.3 0.0008 0.0001 1.52 1.44 0.08 Very Good Weak shaking/no damage potential Weak shaking/no damage potential AZ:TRO 2011-04-09T08:59:06Z 121.4 0.0005 0.0001 1.64 1.41 0.23 Very Good Weak shaking/no damage potential Weak shaking/no damage potential CI:DPP 2011-04-09T08:59:06Z 122.5 0.0012 0.0001 2.68 1.35 1.32 Moderate Weak shaking/no damage potential Light shaking/no damage potential AZ:HWB 2011-04-09T08:59:06Z 125.2 0.0003 0.0001 1.53 1.32 0.21 Very Good Weak shaking/no damage potential Weak shaking/no damage potential AZ:FRD 2011-04-09T08:59:06Z 128.6 0.0003 0.0001 0.85 1.31 0.45 Very Good Weak shaking/no damage potential Not felt/no damage potential CI:PMD 2011-04-09T08:59:06Z 131.0 0.0007 0.0001 2.23 1.28 0.95 Good Weak shaking/no damage potential Light shaking/no damage potential AZ:PFO 2011-04-09T08:59:06Z 131.4 0.0002 0.0000 1.01 1.27 0.26 Very Good Weak shaking/no damage potential Weak shaking/no damage potential AZ:BZN 2011-04-09T08:59:06Z 132.3 0.0009 0.0000 1.99 1.26 0.73 Good Weak shaking/no damage potential Weak shaking/no damage potential AZ:CPE 2011-04-09T08:59:06Z 133.6 0.0018 0.0000 2.80 1.19 1.61 Poor Weak shaking/no damage potential Light shaking/no damage potential AZ:SND 2011-04-09T08:59:06Z 134.2 0.0011 0.0000 1.77 1.23 0.53 Good Weak shaking/no damage potential Weak shaking/no damage potential CI:SDG 2011-04-09T08:59:06Z 134.4 0.0020 0.0000 2.53 1.17 1.35 Moderate Weak shaking/no damage potential Light shaking/no damage potential CI:PLM 2011-04-09T08:59:06Z 135.3 0.0008 0.0000 1.65 1.21 0.44 Very Good Weak shaking/no damage potential Weak shaking/no damage potential CI:DNR 2011-04-09T08:59:06Z 136.5 0.0020 0.0000 2.34 1.20 1.13 Moderate Weak shaking/no damage potential Light shaking/no damage potential AZ:WMC 2011-04-09T08:59:06Z 139.7 0.0009 0.0000 2.22 1.16 1.06 Moderate Weak shaking/no damage potential Light shaking/no damage potential AZ:CRY 2011-04-09T08:59:06Z 142.8 0.0003 0.0000 0.77 1.12 0.36 Very Good Weak shaking/no damage potential Not felt/no damage potential AZ:SOL 2011-04-09T08:59:06Z 145.6 0.0025 0.0000 2.46 1.04 1.42 Moderate Weak shaking/no damage potential Light shaking/no damage potential CI:MGE 2011-04-09T08:59:06Z 147.7 0.0005 0.0000 1.81 1.07 0.74 Good Weak shaking/no damage potential Weak shaking/no damage potential CI:GOR 2011-04-09T08:59:06Z 153.8 0.0004 0.0000 1.73 0.97 0.76 Good Not felt/no damage potential Weak shaking/no damage potential AZ:KNW 2011-04-09T08:59:06Z 154.0 0.0002 0.0000 0.69 0.99 0.30 Very Good Not felt/no damage potential Not felt/no damage potential AZ:RDM 2011-04-09T08:59:06Z 155.0 0.0013 0.0000 1.92 0.98 0.94 Good Not felt/no damage potential Weak shaking/no damage potential CI:PSD 2011-04-09T08:59:06Z 155.9 0.0003 0.0000 1.33 0.98 0.35 Very Good Not felt/no damage potential Weak shaking/no damage potential CI:BEL 2011-04-09T08:59:06Z 157.1 0.0001 0.0000 0.00 0.96 0.96 Good Not felt/no damage potential Not felt/no damage potential CI:DEV 2011-04-09T08:59:06Z 168.0 0.0004 0.0000 1.51 0.84 0.67 Good Not felt/no damage potential Weak shaking/no damage potential CI:BLY 2011-04-09T08:59:06Z 169.2 0.0001 0.0000 0.39 0.78 0.38 Very Good Not felt/no damage potential Not felt/no damage potential CI:SLR 2011-04-09T08:59:06Z 169.4 0.0006 0.0000 1.56 0.83 0.74 Good Not felt/no damage potential Weak shaking/no damage potential CI:MUR 2011-04-09T08:59:06Z 176.5 0.0021 0.0000 2.56 0.75 1.82 Poor Not felt/no damage potential Light shaking/no damage potential CI:MSJ 2011-04-09T08:59:06Z 177.1 0.0012 0.0000 2.12 0.75 1.37 Moderate Not felt/no damage potential Light shaking/no damage potential CI:IRM 2011-04-09T08:59:06Z 180.4 0.0001 0.0000 0.00 0.70 0.70 Good Not felt/no damage potential Not felt/no damage potential CI:MCT 2011-04-09T08:59:06Z 182.4 0.0000 0.0000 0.00 0.70 0.70 Good Not felt/no damage potential Not felt/no damage potential CI:BBS 2011-04-09T08:59:06Z 187.5 0.0005 0.0000 0.72 0.65 0.07 Very Good Not felt/no damage potential Not felt/no damage potential CI:PER 2011-04-09T08:59:06Z 196.5 0.0004 0.0000 1.23 0.56 0.67 Good Not felt/no damage potential Weak shaking/no damage potential Summary of 48 site-specific forecast ratings for event:10922333 Very Good: 37.5% Good : 33.3% Moderate : 16.7% Poor : 12.5% Very Poor: 0.0%
Intensity Equations
There are formulas presented in the CISN testing document that provide formula's to convert PGV and PGA, or just PGV to intensity. We use first for VS intenstiy and other for On Site.
The formulas were somewhat vague on what type of logs to use, and what units to use in the input PGV values. We ended up with mms and that seems to be the right formula. Below is psuedo code showing how we implemented the algorithms.
These algorithms seems to consider non-integer intensity values so we support those here.
def convertVelocity2Intensity(pgv_val):
pgv_in_mms = pgv_val*10.0
ite = (2.10 * math.log10(pgv_in_mms)) + 3.40
if ite >= 5:
ite = (3.47 * math.log(pgv_in_mms)) + 2.35
return ite
def convertToIntensity(pgv,pga):
#
# inputs to floats, returns a float
#
pgv_in_mms = pgv*1000.0
pga_in_mms2 = pga*1000.0
inten = (2.20 * math.log10(pga_in_mms2)) + 1
if inten < 0.0:
inten = 0.0
if int(inten) > 5.0:
inten = (3.66 * math.log10(pga_in_mms2)) - 1.66
if int(inten) > 7.0:
inten = (3.47 * math.log10(pgv_in_mms)) + 2.35
return inten
Performance Evaluation Summaries
Two descriptions of performance summaries include:
- CISN Testing Document
- CTC Group Evaluation Collection
We separate the summaries into "Event" Summaries and "Catalog" Summaries where catalog means any set of earthquakes, and the summary must make it clear what criteria were used to define the catalog.
CISN On Site Trigger
An example CISN Trigger file is posted here:
We noted several things about the current formatting.
- The report id never changes. The intention was that this uniquely identified the file. But it is currently fixed.
- Forecasts PGV and Mag only
- Reports both algorithm and alert time, which always seem to have a delta of 2 seconds. This may mean difference between the two is the amount of data needed to produce the alert
- Multiple forecasts per trigger file
- Multiple events per trigger file
- Seems to report duplicate forecast information for each trigger. One copy of the trigger also contains observational data for the trigger. It is unclear how the trigger is match to the event id. We noted that in some cases the event id reported in the trigger file is not the final ANSS event id.
- The event id does not have a network id so we assume it is CI network event
VS Trigger Reports
VS is running on southern California Events. We noted the following.
- Reports one event per trigger file
- Each trigger file reports source of report
- Each trigger file contains a set of time series
- It seems that all time series in the trigger report have the same timestamps
- Event oriented time series include:
- magnitude
- lat
- lon
- origin time
- Region-oriented time series include
- pgv
- pga
ShakeMap Data
We are currently working with the ShakeMap stationlist txt file. We may convert to the stationlist xml files, but it was easier to parse the column oriented to start.
We use the ShakeMap data as reference observational data for station amplitude values.
We have identified several characteristics to the ShakeMap observational reports.
- Two main formats are the CI: and NN: events, and ECX events. ECX events have their pgv and pga values in different columns than the others, so this type of ShakeMap report must be checked for while parsing.
- ShakeMap files also have different number of columns populated. Currently we require that both PGA and PGV are listed before we use it as a authorized data observation. The goal is to maximize the number of observations we have to compare to forecasts. Right now, we don't have forecasts at PGV or PGA only sites. Once we do, we should add small modification to track observations even if they only have one component of data.
- We pull event ID from the header of the Shakemap file. However, it does not list the network, which we need to match it up in the ANSS catalog.
Processing Flow
Current design is based on processing one or more trigger reports. Currently two types of trigger reports are supported CIT On Site, and ETH Virtual Seismologist.
- Init CISN Station list, ANSS EQ catalog, ShakeMap catalog
- For each Trigger File
- Mapped to One or More ANSS Events and observed data from ShakeMap
- For each event in trigger file
- write event summary
- Add to catalog summaries
- write catalog summaries
ANSS Catalog Files
We create a catalog of ANSS events based on the ANSS web-based catalog search capabilities. The query parameters for the ANSS catalog are in the header of the file. This catalog represents the authorized reference data for events in the testing region. While we use ShakeMap data files to get station information, we use ANSS catalog as the reference event catalog.
- Example ANSS Catalog File
Software Download
CTC Software is posted in git repository on github. maechlin:ctc
