Difference between revisions of "EEW Evaluation"

From SCECpedia
Jump to navigationJump to search
Line 181: Line 181:
 
== Software Download ==
 
== Software Download ==
 
CTC Software is posted in git repository on github.
 
CTC Software is posted in git repository on github.
maechlin:ctc
+
 
 +
*[https://github.com/ Git Hub Home Page]
 +
*Project Name: git@github.com:maechlin/ctc
 +
 
 +
If you have git installed, and have entered a public key created, and uploaded to github, you can retrieve the codes with the following commands.
 +
 
 +
%git remote git@github.com:maechlin/ctc
  
 
== Related Entries ==
 
== Related Entries ==

Revision as of 21:30, 1 May 2011

Example On Site Trigger Forecast Performance

Event ID: 10922333
Magnitude: 4.13
Origin Time: 2011/04/09 08:58:49.22
Hypocenter: 32.6055 -115.7237 9.38 km
Forecast Source: ssi://caltech.gps.edu/onsite/2.1.0
Forecast Time: 2011-4-9T8:59:21Z
Seconds after Origin: 32
Net:Sta  Dist(km) PGV(obs) PGV(fcst)cm/second  I(obs) I(fcst)MMI   abs(Io-If)  Skill   obs GM    fcst GM    
  CI:WES    17.3   0.0204    0.3816             0.20     3.82       3.61   Very Poor        Moderate shaking/very light damage              Not felt/no damage potential
  CI:DRE    34.0   0.0109    0.2022             0.11     2.02       1.91        Poor         Light shaking/no damage potential              Not felt/no damage potential
  CI:IBP    35.5   0.0043    0.1042             0.04     1.04       1.00        Good          Weak shaking/no damage potential              Not felt/no damage potential
  CI:ERR    57.7   0.0046    0.1317             0.05     1.32       1.27    Moderate          Weak shaking/no damage potential              Not felt/no damage potential

Summary of 4 site-specific forecast ratings for event:10922333
Very Good:  0.0%
Good     : 25.0%
Moderate : 25.0%
Poor     : 25.0%
Very Poor: 25.0%

Example Virtual Seismologist Forecast Performance

Event ID: 10922333
Magnitude: 4.13
Origin Time: 2011/04/09 08:58:49.22
Hypocenter: 32.6055 -115.7237 9.38 km
Forecast Source: smi://caltech.gps.edu/onsite/cinnabar/vs/eewvs
Seconds after Origin: 17
Network:Station   Forecast Time  Dist(km) PGV(obs) PGV(fcst)m/s  Int(obs) Int(fcst)MMI abs(Io-If)  fcst Skill obs GM    fcst GM    
  CI:WES   2011-04-09T08:59:06Z    17.3   0.0204    0.0021     5.22     5.13       0.09   Very Good       Very strong shaking/moderate damage       Very strong shaking/moderate damage
  CI:DRE   2011-04-09T08:59:06Z    34.0   0.0109    0.0003     4.89     3.30       1.59        Poor        Moderate shaking/very light damage     Strong shaking/light damage potential
  CI:SWS   2011-04-09T08:59:06Z    38.6   0.0021    0.0004     2.96     3.44       0.48   Very Good        Moderate shaking/very light damage         Light shaking/no damage potential
  CI:ERR   2011-04-09T08:59:06Z    57.7   0.0046    0.0002     3.88     2.70       1.19    Moderate         Light shaking/no damage potential        Moderate shaking/very light damage
  CI:RXH   2011-04-09T08:59:06Z    64.9   0.0018    0.0002     2.77     2.46       0.31   Very Good         Light shaking/no damage potential         Light shaking/no damage potential
  CI:BTC   2011-04-09T08:59:06Z    65.2   0.0021    0.0001     3.06     2.32       0.74        Good         Light shaking/no damage potential        Moderate shaking/very light damage
AZ:MONP2   2011-04-09T08:59:06Z    73.1   0.0014    0.0001     2.36     2.22       0.14   Very Good         Light shaking/no damage potential         Light shaking/no damage potential
  CI:SAL   2011-04-09T08:59:06Z    79.0   0.0058    0.0001     3.78     2.15       1.63        Poor         Light shaking/no damage potential        Moderate shaking/very light damage
  CI:JEM   2011-04-09T08:59:06Z    97.7   0.0017    0.0001     2.59     1.75       0.83        Good          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:BOR   2011-04-09T08:59:06Z    98.4   0.0011    0.0001     2.74     1.76       0.97        Good          Weak shaking/no damage potential         Light shaking/no damage potential
 CI:NSS2   2011-04-09T08:59:06Z   107.6   0.0006    0.0001     1.49     1.62       0.13   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:EML   2011-04-09T08:59:06Z   110.2   0.0011    0.0001     3.22     1.52       1.69        Poor          Weak shaking/no damage potential        Moderate shaking/very light damage
  CI:OLP   2011-04-09T08:59:06Z   113.6   0.0021    0.0001     3.20     1.43       1.76        Poor          Weak shaking/no damage potential        Moderate shaking/very light damage
 AZ:LVA2   2011-04-09T08:59:06Z   114.3   0.0011    0.0001     2.11     1.51       0.61        Good          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:SDR   2011-04-09T08:59:06Z   115.5   0.0009    0.0001     2.62     1.42       1.20    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:TOR   2011-04-09T08:59:06Z   117.6   0.0004    0.0001     1.42     1.47       0.05   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:BC3   2011-04-09T08:59:06Z   119.2   0.0004    0.0001     1.53     1.42       0.11   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:CTC   2011-04-09T08:59:06Z   119.3   0.0008    0.0001     1.52     1.44       0.08   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:TRO   2011-04-09T08:59:06Z   121.4   0.0005    0.0001     1.64     1.41       0.23   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:DPP   2011-04-09T08:59:06Z   122.5   0.0012    0.0001     2.68     1.35       1.32    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:HWB   2011-04-09T08:59:06Z   125.2   0.0003    0.0001     1.53     1.32       0.21   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:FRD   2011-04-09T08:59:06Z   128.6   0.0003    0.0001     0.85     1.31       0.45   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
  CI:PMD   2011-04-09T08:59:06Z   131.0   0.0007    0.0001     2.23     1.28       0.95        Good          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:PFO   2011-04-09T08:59:06Z   131.4   0.0002    0.0000     1.01     1.27       0.26   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:BZN   2011-04-09T08:59:06Z   132.3   0.0009    0.0000     1.99     1.26       0.73        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  AZ:CPE   2011-04-09T08:59:06Z   133.6   0.0018    0.0000     2.80     1.19       1.61        Poor          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:SND   2011-04-09T08:59:06Z   134.2   0.0011    0.0000     1.77     1.23       0.53        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:SDG   2011-04-09T08:59:06Z   134.4   0.0020    0.0000     2.53     1.17       1.35    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:PLM   2011-04-09T08:59:06Z   135.3   0.0008    0.0000     1.65     1.21       0.44   Very Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:DNR   2011-04-09T08:59:06Z   136.5   0.0020    0.0000     2.34     1.20       1.13    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:WMC   2011-04-09T08:59:06Z   139.7   0.0009    0.0000     2.22     1.16       1.06    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  AZ:CRY   2011-04-09T08:59:06Z   142.8   0.0003    0.0000     0.77     1.12       0.36   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
  AZ:SOL   2011-04-09T08:59:06Z   145.6   0.0025    0.0000     2.46     1.04       1.42    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
  CI:MGE   2011-04-09T08:59:06Z   147.7   0.0005    0.0000     1.81     1.07       0.74        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:GOR   2011-04-09T08:59:06Z   153.8   0.0004    0.0000     1.73     0.97       0.76        Good              Not felt/no damage potential          Weak shaking/no damage potential
  AZ:KNW   2011-04-09T08:59:06Z   154.0   0.0002    0.0000     0.69     0.99       0.30   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:RDM   2011-04-09T08:59:06Z   155.0   0.0013    0.0000     1.92     0.98       0.94        Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:PSD   2011-04-09T08:59:06Z   155.9   0.0003    0.0000     1.33     0.98       0.35   Very Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:BEL   2011-04-09T08:59:06Z   157.1   0.0001    0.0000     0.00     0.96       0.96        Good              Not felt/no damage potential              Not felt/no damage potential
  CI:DEV   2011-04-09T08:59:06Z   168.0   0.0004    0.0000     1.51     0.84       0.67        Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:BLY   2011-04-09T08:59:06Z   169.2   0.0001    0.0000     0.39     0.78       0.38   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:SLR   2011-04-09T08:59:06Z   169.4   0.0006    0.0000     1.56     0.83       0.74        Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:MUR   2011-04-09T08:59:06Z   176.5   0.0021    0.0000     2.56     0.75       1.82        Poor              Not felt/no damage potential         Light shaking/no damage potential
  CI:MSJ   2011-04-09T08:59:06Z   177.1   0.0012    0.0000     2.12     0.75       1.37    Moderate              Not felt/no damage potential         Light shaking/no damage potential
  CI:IRM   2011-04-09T08:59:06Z   180.4   0.0001    0.0000     0.00     0.70       0.70        Good              Not felt/no damage potential              Not felt/no damage potential
  CI:MCT   2011-04-09T08:59:06Z   182.4   0.0000    0.0000     0.00     0.70       0.70        Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BBS   2011-04-09T08:59:06Z   187.5   0.0005    0.0000     0.72     0.65       0.07   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:PER   2011-04-09T08:59:06Z   196.5   0.0004    0.0000     1.23     0.56       0.67        Good              Not felt/no damage potential          Weak shaking/no damage potential

Summary of 48 site-specific forecast ratings for event:10922333
Very Good: 37.5%
Good     : 33.3%
Moderate : 16.7%
Poor     : 12.5%
Very Poor:  0.0%

Intensity Equations

There are formulas presented in the CISN testing document that provide formula's to convert PGV and PGA, or just PGV to intensity. We use first for VS intenstiy and other for On Site.

The formulas were somewhat vague on what type of logs to use, and what units to use in the input PGV values. We ended up with mms and that seems to be the right formula. Below is psuedo code showing how we implemented the algorithms.

These algorithms seems to consider non-integer intensity values so we support those here.

def convertVelocity2Intensity(pgv_val):
    pgv_in_mms = pgv_val*10.0
    ite = (2.10 * math.log10(pgv_in_mms)) + 3.40
    if ite >= 5:
        ite = (3.47 * math.log(pgv_in_mms)) + 2.35
    return ite

def convertToIntensity(pgv,pga):
    #
    # inputs to floats, returns a float
    #
    pgv_in_mms = pgv*1000.0
    pga_in_mms2 = pga*1000.0
    inten = (2.20 * math.log10(pga_in_mms2)) + 1
    if inten < 0.0:
        inten = 0.0
    if int(inten) > 5.0:
        inten = (3.66 * math.log10(pga_in_mms2)) - 1.66
    if int(inten) > 7.0:
        inten = (3.47 * math.log10(pgv_in_mms)) + 2.35
    return inten

Performance Evaluation Summaries

Two descriptions of performance summaries include:

  • CISN Testing Document
  • CTC Group Evaluation Collection

We separate the summaries into "Event" Summaries and "Catalog" Summaries where catalog means any set of earthquakes, and the summary must make it clear what criteria were used to define the catalog.

CISN On Site Trigger

An example CISN Trigger file is posted here:

We noted several things about the current formatting.

  1. The report id never changes. The intention was that this uniquely identified the file. But it is currently fixed.
  2. Forecasts PGV and Mag only
  3. Reports both algorithm and alert time, which always seem to have a delta of 2 seconds. This may mean difference between the two is the amount of data needed to produce the alert
  4. Multiple forecasts per trigger file
  5. Multiple events per trigger file
  6. Seems to report duplicate forecast information for each trigger. One copy of the trigger also contains observational data for the trigger. It is unclear how the trigger is match to the event id. We noted that in some cases the event id reported in the trigger file is not the final ANSS event id.
  7. The event id does not have a network id so we assume it is CI network event

VS Trigger Reports

VS is running on southern California Events. We noted the following.

  1. Reports one event per trigger file
  2. Each trigger file reports source of report
  3. Each trigger file contains a set of time series
  4. It seems that all time series in the trigger report have the same timestamps
  5. Event oriented time series include:
    1. magnitude
    2. lat
    3. lon
    4. origin time
  6. Region-oriented time series include
    1. pgv
    2. pga

ShakeMap Data

We are currently working with the ShakeMap stationlist txt file. We may convert to the stationlist xml files, but it was easier to parse the column oriented to start.

We use the ShakeMap data as reference observational data for station amplitude values.

We have identified several characteristics to the ShakeMap observational reports.

  1. Two main formats are the CI: and NN: events, and ECX events. ECX events have their pgv and pga values in different columns than the others, so this type of ShakeMap report must be checked for while parsing.
  2. ShakeMap files also have different number of columns populated. Currently we require that both PGA and PGV are listed before we use it as a authorized data observation. The goal is to maximize the number of observations we have to compare to forecasts. Right now, we don't have forecasts at PGV or PGA only sites. Once we do, we should add small modification to track observations even if they only have one component of data.
  3. We pull event ID from the header of the Shakemap file. However, it does not list the network, which we need to match it up in the ANSS catalog.

Processing Flow

Current design is based on processing one or more trigger reports. Currently two types of trigger reports are supported CIT On Site, and ETH Virtual Seismologist.

  1. Init CISN Station list, ANSS EQ catalog, ShakeMap catalog
  2. For each Trigger File
    1. Mapped to One or More ANSS Events and observed data from ShakeMap
  3. For each event in trigger file
    1. write event summary
    2. Add to catalog summaries
  4. write catalog summaries

ANSS Catalog Files

We create a catalog of ANSS events based on the ANSS web-based catalog search capabilities. The query parameters for the ANSS catalog are in the header of the file. This catalog represents the authorized reference data for events in the testing region. While we use ShakeMap data files to get station information, we use ANSS catalog as the reference event catalog.

  • Example ANSS Catalog File

Software Download

CTC Software is posted in git repository on github.

If you have git installed, and have entered a public key created, and uploaded to github, you can retrieve the codes with the following commands.

%git remote git@github.com:maechlin/ctc

Related Entries

See Also