EEW Evaluation

CISN Testing Center (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.

CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.

1 Example Ground Motion Forecast Summaries
2 Example Virtual Seismologist Forecast Performance
3 Interpretation of Results
4 Trigger Matching Algorithms
5 Input Configuration Files
6 Usage Overview
7 Intensity Equations
8 Performance Evaluation Summaries
9 CISN On Site Trigger
10 VS Trigger Reports
11 ShakeMap Data
12 Migration to ShakeAlert Testing
13 Processing Flow
14 ANSS Catalog Files
15 Software Download
16 Contributors
17 Related Entries
18 See Also

Example Ground Motion Forecast Summaries

Event ID: 10922333
Magnitude: 4.13
Origin Time: 2011/04/09 08:58:49.22
Hypocenter: 32.6055 -115.7237 9.38 km
Forecast Source: ssi://caltech.gps.edu/onsite/2.1.0
Forecast Time: 2011-4-9T8:59:21Z
Seconds after Origin: 32
Net:Sta  Dist(km) PGV(obs) PGV(fcst)cm/second  I(obs) I(fcst)MMI   abs(Io-If)  Skill   obs GM    fcst GM    
  CI:WES    17.3   0.0204    0.3816             0.20     3.82       3.61   Very Poor        Moderate shaking/very light damage              Not felt/no damage potential
  CI:DRE    34.0   0.0109    0.2022             0.11     2.02       1.91        Poor         Light shaking/no damage potential              Not felt/no damage potential
  CI:IBP    35.5   0.0043    0.1042             0.04     1.04       1.00        Good          Weak shaking/no damage potential              Not felt/no damage potential
  CI:ERR    57.7   0.0046    0.1317             0.05     1.32       1.27    Moderate          Weak shaking/no damage potential              Not felt/no damage potential

Summary of 4 site-specific forecast ratings for event:10922333
Very Good:  0.0%
Good     : 25.0%
Moderate : 25.0%
Poor     : 25.0%
Very Poor: 25.0%

Example Virtual Seismologist Forecast Performance

Event ID: 10922333
Magnitude: 4.13
Origin Time: 2011/04/09 08:58:49.22
Hypocenter: 32.6055 -115.7237 9.38 km
Forecast Source: smi://caltech.gps.edu/onsite/cinnabar/vs/eewvs
Seconds after Origin: 17
Network:Station   Forecast Time  Dist(km) PGV(obs) PGV(fcst)m/s  Int(obs) Int(fcst)MMI abs(Io-If)  fcst Skill obs GM    fcst GM    
  CI:WES   2011-04-09T08:59:06Z    17.3   0.0204    0.0021     5.22     5.13       0.09   Very Good       Very strong shaking/moderate damage       Very strong shaking/moderate damage
  CI:DRE   2011-04-09T08:59:06Z    34.0   0.0109    0.0003     4.89     3.30       1.59        Poor        Moderate shaking/very light damage     Strong shaking/light damage potential
  CI:SWS   2011-04-09T08:59:06Z    38.6   0.0021    0.0004     2.96     3.44       0.48   Very Good        Moderate shaking/very light damage         Light shaking/no damage potential
  CI:ERR   2011-04-09T08:59:06Z    57.7   0.0046    0.0002     3.88     2.70       1.19    Moderate         Light shaking/no damage potential        Moderate shaking/very light damage
  CI:RXH   2011-04-09T08:59:06Z    64.9   0.0018    0.0002     2.77     2.46       0.31   Very Good         Light shaking/no damage potential         Light shaking/no damage potential
  CI:BTC   2011-04-09T08:59:06Z    65.2   0.0021    0.0001     3.06     2.32       0.74        Good         Light shaking/no damage potential        Moderate shaking/very light damage
AZ:MONP2   2011-04-09T08:59:06Z    73.1   0.0014    0.0001     2.36     2.22       0.14   Very Good         Light shaking/no damage potential         Light shaking/no damage potential
  CI:SAL   2011-04-09T08:59:06Z    79.0   0.0058    0.0001     3.78     2.15       1.63        Poor         Light shaking/no damage potential        Moderate shaking/very light damage
  CI:JEM   2011-04-09T08:59:06Z    97.7   0.0017    0.0001     2.59     1.75       0.83        Good          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:BOR   2011-04-09T08:59:06Z    98.4   0.0011    0.0001     2.74     1.76       0.97        Good          Weak shaking/no damage potential         Light shaking/no damage potential
 CI:NSS2   2011-04-09T08:59:06Z   107.6   0.0006    0.0001     1.49     1.62       0.13   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:EML   2011-04-09T08:59:06Z   110.2   0.0011    0.0001     3.22     1.52       1.69        Poor          Weak shaking/no damage potential        Moderate shaking/very light damage
  CI:OLP   2011-04-09T08:59:06Z   113.6   0.0021    0.0001     3.20     1.43       1.76        Poor          Weak shaking/no damage potential        Moderate shaking/very light damage
 AZ:LVA2   2011-04-09T08:59:06Z   114.3   0.0011    0.0001     2.11     1.51       0.61        Good          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:SDR   2011-04-09T08:59:06Z   115.5   0.0009    0.0001     2.62     1.42       1.20    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:TOR   2011-04-09T08:59:06Z   117.6   0.0004    0.0001     1.42     1.47       0.05   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:BC3   2011-04-09T08:59:06Z   119.2   0.0004    0.0001     1.53     1.42       0.11   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:CTC   2011-04-09T08:59:06Z   119.3   0.0008    0.0001     1.52     1.44       0.08   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:TRO   2011-04-09T08:59:06Z   121.4   0.0005    0.0001     1.64     1.41       0.23   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:DPP   2011-04-09T08:59:06Z   122.5   0.0012    0.0001     2.68     1.35       1.32    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:HWB   2011-04-09T08:59:06Z   125.2   0.0003    0.0001     1.53     1.32       0.21   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:FRD   2011-04-09T08:59:06Z   128.6   0.0003    0.0001     0.85     1.31       0.45   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
  CI:PMD   2011-04-09T08:59:06Z   131.0   0.0007    0.0001     2.23     1.28       0.95        Good          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:PFO   2011-04-09T08:59:06Z   131.4   0.0002    0.0000     1.01     1.27       0.26   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:BZN   2011-04-09T08:59:06Z   132.3   0.0009    0.0000     1.99     1.26       0.73        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:CPE   2011-04-09T08:59:06Z   133.6   0.0018    0.0000     2.80     1.19       1.61        Poor          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:SND   2011-04-09T08:59:06Z   134.2   0.0011    0.0000     1.77     1.23       0.53        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:SDG   2011-04-09T08:59:06Z   134.4   0.0020    0.0000     2.53     1.17       1.35    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:PLM   2011-04-09T08:59:06Z   135.3   0.0008    0.0000     1.65     1.21       0.44   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:DNR   2011-04-09T08:59:06Z   136.5   0.0020    0.0000     2.34     1.20       1.13    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:WMC   2011-04-09T08:59:06Z   139.7   0.0009    0.0000     2.22     1.16       1.06    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:CRY   2011-04-09T08:59:06Z   142.8   0.0003    0.0000     0.77     1.12       0.36   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
  AZ:SOL   2011-04-09T08:59:06Z   145.6   0.0025    0.0000     2.46     1.04       1.42    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:MGE   2011-04-09T08:59:06Z   147.7   0.0005    0.0000     1.81     1.07       0.74        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:GOR   2011-04-09T08:59:06Z   153.8   0.0004    0.0000     1.73     0.97       0.76        Good              Not felt/no damage potential          Weak shaking/no damage potential
  AZ:KNW   2011-04-09T08:59:06Z   154.0   0.0002    0.0000     0.69     0.99       0.30   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:RDM   2011-04-09T08:59:06Z   155.0   0.0013    0.0000     1.92     0.98       0.94        Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:PSD   2011-04-09T08:59:06Z   155.9   0.0003    0.0000     1.33     0.98       0.35   Very Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:BEL   2011-04-09T08:59:06Z   157.1   0.0001    0.0000     0.00     0.96       0.96        Good              Not felt/no damage potential              Not felt/no damage potential
  CI:DEV   2011-04-09T08:59:06Z   168.0   0.0004    0.0000     1.51     0.84       0.67        Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:BLY   2011-04-09T08:59:06Z   169.2   0.0001    0.0000     0.39     0.78       0.38   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:SLR   2011-04-09T08:59:06Z   169.4   0.0006    0.0000     1.56     0.83       0.74        Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:MUR   2011-04-09T08:59:06Z   176.5   0.0021    0.0000     2.56     0.75       1.82        Poor              Not felt/no damage potential         Light shaking/no damage potential
  CI:MSJ   2011-04-09T08:59:06Z   177.1   0.0012    0.0000     2.12     0.75       1.37    Moderate              Not felt/no damage potential         Light shaking/no damage potential
  CI:IRM   2011-04-09T08:59:06Z   180.4   0.0001    0.0000     0.00     0.70       0.70        Good              Not felt/no damage potential              Not felt/no damage potential
  CI:MCT   2011-04-09T08:59:06Z   182.4   0.0000    0.0000     0.00     0.70       0.70        Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BBS   2011-04-09T08:59:06Z   187.5   0.0005    0.0000     0.72     0.65       0.07   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:PER   2011-04-09T08:59:06Z   196.5   0.0004    0.0000     1.23     0.56       0.67        Good              Not felt/no damage potential          Weak shaking/no damage potential

Summary of 48 site-specific forecast ratings for event:10922333
Very Good: 37.5%
Good     : 33.3%
Moderate : 16.7%
Poor     : 12.5%
Very Poor:  0.0%

Interpretation of Results

When comparing perfmance of these two EEW algorithms here are few things to consider. First, VS is an associator-based system. It uses phase picks to get a hypocenter and some other technique to get the magnitude. Once the event is declared, VS starts to forecast time series of parameters. In the current summaries, we use first reports from VS. Other plots showing more details of the time series will be developed. Once the VS forecasting starts, it updates event parameters lat,lon,mag,origin time. It also forecast pgv and pga at a set of CISN station locations. We are not sure how the set of stations to forecast for are decided, but it is fixed for the full trigger report. Event parameters and pgv and pga for all sites are updated every timestep (5 seconds). The number of sites varies by event we've seen between 20 and 50 per M4-5 events.

The OnSite algorithm is intended to be a faster algorithm because it is a single station approach. It produces a trigger, a final mag, a peak pgv.

We currently produce the intensity-skill summary using all available onsite triggers for each event. We combine the pgv forecasts to create the map.

However, a better comparison to the VS may be to use the magnitude forecast at each trigger, and use that to forecast ground motions at station sites, using an attenuation relationship.

VS must use this kind of approach to produce it's groundmotion forecasts. I think the On Site trigger might have enough information to use VS ground motion forecast to produce ground motion forecasts at multiple stations for each on-site trigger.

Trigger Matching Algorithms

The CTC software is designed to start processing when given one, or more, EEW trigger files produced by CISN EEW systems. The intent of the EEW Evaluation system is that it will compare earthquake and ground motion forecasts from these trigger files against the observational data collected by the CISN and other network operators. The reference observational catalog is the ANSS catalog. We are uncertain the time lag before recent earthquakes appear in this catalog. Given a trigger file, each forecast is matched to a ANSS event. Then observational data about the event is used to produce goodness of fit metrics for the forecasting system.

Developing the algorithm testing system, we learned there are many ways to match triggers to ANSS events. The system should automatically do the matching and also allow users to override automatic matching results.

The matching for current CTC is done differently for VS triggers and On Site Triggers.

The On Site triggers contain an associated CI event. We assume a network CI and match CI:EventID to the ANSS catalog. This usually works, but we have seen a Mamonth event trigger with a CI event id, but in the ANSS catalog, it was under an nothern california event id. To process this event, we added a duplicate of the norcal ANSS event, and put the duplicate into the anss_catalog file, and then the trigger could be processed.

The VS trigger does not have an associated event id. This is more typical case, or the expected case because the EEW algorithms are declaring their own events before CISN events are declared.

The current system takes the vs trigger file. VS trigger files have one earthquake per file. The system finds the Vs trigger time, and then finds an ANSS event with the closest origin time to the trigger time. The trigger and event that get matched must be within configuration.parameter 240 seconds of each other. The matches are put into an output file.

The matching system is separated from the other parts of the processing. It may require humans to correctly match triggers to events. As designed, humans can update or correct the automatically generated matching file. The rest of the CTC codes only use the matches in the matching file, regardless of how they were made.

Input Configuration Files

File-based inputs of CISN configuration information is needed.

CISN station list with network,station,channel,location,lat,lon

NP 1821 HNZ -- 37.798599 -122.268402

Set of Shakemap files current in stationlist.txt format
ANSS catalog file of events in format from web page

2010/06/02 09:30:47.55  32.0432 -115.2220  10.00  3.96   ML   12          0.21  CI      14731660

Usage Overview

To process EEW trigger files for current or future events you can take the following steps.

Install ctc software distribution from scec's github repository. Before any formal release, the expectation is that the current version is best available. Once we have release versions, select the one most appropriate for your use.
Setup CISN confirmation files. Default files are included in distribution, but may need to be updated as network changes. These include stations list, ANSS catalog for a specific region, and time period, and magnitude limit.
Collect ShakeMap event files. For the CTC performanc analysis processing to work, it requires the observational data from ShakeMap stationlist.txt files. We currently use the .txt format files, but we believe the information content is equivalent to the shakemap xml files, so either can be used.
If your trigger files are VS files, you need to trigger2anssevent.txt file to process VS triggers. Default programming is to generate this from ANSS catalog and trigger files. Since we get EEW triggers only for larger events, matching isn't currently a problem. If it is, the system supports manually creating the trigger to event matching file to ensure each trigger is matched and compared to the correct ANSS event.
the main directory is called ctc
In ctc directory, there are three subdirectories:
1. Inputs
2. Outputs
3. src
In the Inputs directory. There are three subdirectories:
1. authdata
2. vs_triggers
3. onsite_triggers
Put the VS trigger files into the vs_triggers subdirectory. Put the OnSite trigger files into the onsite_triggers subdirectory.
In authdata, there are two files and a subdirectory
1. anss_catalog_2011.txt
2. cisn_stations_13April2011.txt
3. shakemaps
The anss_catalog contains events retrieved through the anss web site
The cisn catalog contains sncl and lat,lon for most cisn stations. This should include all stations at which ground motions are forecast by any of the EEW algorithms. When processing a trigger file, a station may be reported as missing. Find out the station information from somewhere, and add it into this catalog to continue processing
Shakemap files in order to produce performance summaries for any eew triggers. Shakemap files have observed pga and pgv for many cisn stations for larger california earthquakes. CTC matches a trigger to a shakemap through the anss event id. So to process a trigger, match the trigger to anss event, and match the anss event to a shakemap file.
All configurations in place and all trigger files to process posted in inputs move to the src directory
gen_ctc.py
This generates a results and puts the results into the output directory.
Updates are sequenced to avoid overwriting. Update files that can be viewed as an html file with links to images and data files used in the summaries.
Currently, the system will produces one summary per event per EEW algorith. The ground motion forecast evaluation report is an event specific forecast skill summary defined in a 2008 CISN testing meeting. Example shown below. Further performance summaries will be added.

Intensity Equations

There are formulas presented in the CISN testing document that provide formula's to convert PGV and PGA, or just PGV to intensity. We use first for VS intenstiy and other for On Site.

The formulas were somewhat vague on what type of logs to use, and what units to use in the input PGV values. We ended up with mms and that seems to be the right formula. Below is psuedo code showing how we implemented the algorithms.

These algorithms seems to consider non-integer intensity values so we support those here.

def convertVelocity2Intensity(pgv_val):
    pgv_in_mms = pgv_val*10.0
    ite = (2.10 * math.log10(pgv_in_mms)) + 3.40
    if ite >= 5:
        ite = (3.47 * math.log(pgv_in_mms)) + 2.35
    return ite

def convertToIntensity(pgv,pga):
    #
    # inputs to floats, returns a float
    #
    pgv_in_mms = pgv*1000.0
    pga_in_mms2 = pga*1000.0
    inten = (2.20 * math.log10(pga_in_mms2)) + 1
    if inten < 0.0:
        inten = 0.0
    if int(inten) > 5.0:
        inten = (3.66 * math.log10(pga_in_mms2)) - 1.66
    if int(inten) > 7.0:
        inten = (3.47 * math.log10(pgv_in_mms)) + 2.35
    return inten

Performance Evaluation Summaries

Two descriptions of performance summaries include:

CISN Testing Document
CTC Group Evaluation Collection

We separate the summaries into "Event" Summaries and "Catalog" Summaries where catalog means any set of earthquakes, and the summary must make it clear what criteria were used to define the catalog.

CISN On Site Trigger

An example CISN Trigger file is posted here:

We noted several things about the current formatting.

The report id never changes. The intention was that this uniquely identified the file. But it is currently fixed.
Forecasts PGV and Mag only
Reports both algorithm and alert time, which always seem to have a delta of 2 seconds. This may mean difference between the two is the amount of data needed to produce the alert
Multiple forecasts per trigger file
Multiple events per trigger file
Seems to report duplicate forecast information for each trigger. One copy of the trigger also contains observational data for the trigger. It is unclear how the trigger is match to the event id. We noted that in some cases the event id reported in the trigger file is not the final ANSS event id.
The event id does not have a network id so we assume it is CI network event

VS Trigger Reports

VS is running on southern California Events. We noted the following.

Reports one event per trigger file
Each trigger file reports source of report
Each trigger file contains a set of time series
It seems that all time series in the trigger report have the same timestamps
Event oriented time series include:
1. magnitude
2. lat
3. lon
4. origin time
Region-oriented time series include
1. pgv
2. pga

ShakeMap Data

We are currently working with the ShakeMap stationlist txt file. We may convert to the stationlist xml files, but it was easier to parse the column oriented to start.

We use the ShakeMap data as reference observational data for station amplitude values.

We have identified several characteristics to the ShakeMap observational reports.

Two main formats are the CI: and NN: events, and ECX events. ECX events have their pgv and pga values in different columns than the others, so this type of ShakeMap report must be checked for while parsing.
ShakeMap files also have different number of columns populated. Currently we require that both PGA and PGV are listed before we use it as a authorized data observation. The goal is to maximize the number of observations we have to compare to forecasts. Right now, we don't have forecasts at PGV or PGA only sites. Once we do, we should add small modification to track observations even if they only have one component of data.
We pull event ID from the header of the Shakemap file. However, it does not list the network, which we need to match it up in the ANSS catalog.

Migration to ShakeAlert Testing

Current capabilities are retrospective.

Processing Flow

Current design is based on processing one or more trigger reports. Currently two types of trigger reports are supported CIT On Site, and ETH Virtual Seismologist.

Init CISN Station list, ANSS EQ catalog, ShakeMap catalog
For each Trigger File
1. Mapped to One or More ANSS Events and observed data from ShakeMap
For each event in trigger file
1. write event summary
2. Add to catalog summaries
write catalog summaries

ANSS Catalog Files

We create a catalog of ANSS events based on the ANSS web-based catalog search capabilities. The query parameters for the ANSS catalog are in the header of the file. This catalog represents the authorized reference data for events in the testing region. While we use ShakeMap data files to get station information, we use ANSS catalog as the reference event catalog.

Example ANSS Catalog File

Software Download

CTC software is intended to be general purpose and will be posted online as open-source. SCEC software version is date-based, something like v11.4.0

Development Version Version:11.5.0

CTC Software is posted in git repository on github.

Git Hub Home Page
Project Name: git@github.com:maechlin/ctc
Read-Access Source Code

If you have git installed, and have entered a public key created, and uploaded to github, you can retrieve the codes with the following commands.

%git remote git@github.com:maechlin/ctc

EEW Evaluation

Contents

Example Ground Motion Forecast Summaries

Example Virtual Seismologist Forecast Performance

Interpretation of Results

Trigger Matching Algorithms

Input Configuration Files

Usage Overview

Intensity Equations

Performance Evaluation Summaries

CISN On Site Trigger

VS Trigger Reports

ShakeMap Data

Migration to ShakeAlert Testing

Processing Flow

ANSS Catalog Files

Software Download

Contributors

Related Entries

See Also

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

Tools