Difference between revisions of "EEW Evaluation"

From SCECpedia
Jump to navigationJump to search
 
(25 intermediate revisions by the same user not shown)
Line 1: Line 1:
CISN Testing Center (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.
+
[[Image:SCEC_NSF-USGS-words_logom.png|left|150px]][[Image:CISNlogos.png|right|250px]][[Image:usgs-logo-color.jpg|right|150px]]
 +
 
 +
[[CISN Testing Center]] (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.
  
 
CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.
 
CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.
 +
== Software Requirements ==
 +
*[http://www.python.org/download/releases/2.6/ Python 2.6]
 +
*[http://code.google.com/p/pysqlite/ Pysqlite Home Page]
 +
 +
== CISN Testing Documents ==
 +
*'''2011'''
 +
*[http://hypocenter.usc.edu/research/EEW/CTC_Summaries_26May2011.pptx CTC Example Summaries 2Mb pptx file]
 +
*[http://scec.usc.edu/scecwiki/images/5/5d/CTC_Update_12May2011.ppt Summary of CTC Development 12 May 2011]
 +
 +
*'''2010'''
 +
*[http://scec.usc.edu/scecwiki/images/c/c8/CTC_14Oct2010_v6.ppt CISN EEW Workshop Oct 2010] (7MB ppt File)
 +
*[http://scec.usc.edu/scecwiki/images/3/3f/EEW_Algor_testing_v01b.pdf CISN EEW Testing Document] (Updated July 2010)
  
 
== Example Ground Motion Forecast Summaries ==
 
== Example Ground Motion Forecast Summaries ==
 
<pre>
 
<pre>
Event ID: 10922333
+
ANSS Event ID: 10922333
 
Magnitude: 4.13
 
Magnitude: 4.13
Origin Time: 2011/04/09 08:58:49.22
+
Origin Time: 2011/04/09 08:58:49
 
Hypocenter: 32.6055 -115.7237 9.38 km
 
Hypocenter: 32.6055 -115.7237 9.38 km
Forecast Source: ssi://caltech.gps.edu/onsite/2.1.0
+
Forecast Sources: ssi://caltech.gps.edu/onsite/2.1.0
Forecast Time: 2011-4-9T8:59:21Z
+
First Forecast Time: 2011/04/09 08:59:21
Seconds after Origin: 32
+
First Forecast Secs after Origin: 32
Net:Sta  Dist(km) PGV(obs) PGV(fcst)cm/second I(obs) I(fcst)MMI   abs(Io-If)  Skill   obs GM   fcst GM     
+
Net:Sta  Dist(km) Wd(secs) PGV(obs-cm/s) PGV(fcst) PGA(obs-cm/s2)  PGA(fcst) I(obs-MMI)     I(fcst)  abs(Io-If)  Skill           obs GM             fcst GM     
   CI:WES    17.3   0.0204    0.3816             0.20    3.82       3.61  Very Poor       Moderate shaking/very light damage             Not felt/no damage potential
+
   CI:WES    17.3   32.0    0.2000      0.3816                                     1.93          2.52       0.59       Good        Light shaking/no damage potential          Weak shaking/no damage potential
   CI:DRE    34.0   0.0109    0.2022             0.11    2.02       1.91       Poor        Light shaking/no damage potential             Not felt/no damage potential
+
   CI:DRE    34.0   35.0    0.1071      0.2022                                     1.36          1.94       0.58       Good          Weak shaking/no damage potential         Weak shaking/no damage potential
   CI:IBP    35.5   0.0043    0.1042             0.04    1.04       1.00        Good          Weak shaking/no damage potential              Not felt/no damage potential
+
   CI:IBP    35.5   35.0    0.0422      0.1042                                     1.00          1.34       0.34  Very Good          Weak shaking/no damage potential              Not felt/no damage potential
   CI:ERR    57.7   0.0046    0.1317             0.05    1.32       1.27    Moderate         Weak shaking/no damage potential              Not felt/no damage potential
+
   CI:ERR    57.7   39.0    0.0449      0.1317                                     1.00          1.55       0.55        Good         Weak shaking/no damage potential              Not felt/no damage potential
  
 
Summary of 4 site-specific forecast ratings for event:10922333
 
Summary of 4 site-specific forecast ratings for event:10922333
Very Good:  0.0%
+
Very Good:  25.0%
Good    : 25.0%
+
Good    : 75.0%
Moderate : 25.0%
+
Moderate :   0.0%
Poor    : 25.0%
+
Poor    :   0.0%
Very Poor: 25.0%
+
Very Poor:   0.0%
  
 
</pre>
 
</pre>
 +
 
== Example Virtual Seismologist Forecast Performance ==
 
== Example Virtual Seismologist Forecast Performance ==
 
<pre>
 
<pre>
Event ID: 10922333
+
ANSS Event ID: 14937372
Magnitude: 4.13
+
Magnitude: 5.07
Origin Time: 2011/04/09 08:58:49.22
+
Origin Time: 2011/02/18 17:47:35
Hypocenter: 32.6055 -115.7237 9.38 km
+
Hypocenter: 32.047 -115.0622 15.0 km
Forecast Source: smi://caltech.gps.edu/onsite/cinnabar/vs/eewvs
+
Forecast Sources: smi://caltech.gps.edu/onsite/cinnabar/vs/eewvs
Seconds after Origin: 17
+
First Forecast Time: 2011/02/18 17:48:12
Network:Station  Forecast Time  Dist(km) PGV(obs) PGV(fcst)m/s Int(obs) Int(fcst)MMI abs(Io-If)  fcst Skill obs GM   fcst GM     
+
First Forecast Secs after Origin: 37
   CI:WES  2011-04-09T08:59:06Z   17.0.0204    0.0021    5.22    5.13       0.09   Very Good       Very strong shaking/moderate damage       Very strong shaking/moderate damage
+
Net:Sta  Dist(km) Wd(secs) PGV(obs-cm/s) PGV(fcst) PGA(obs-cm/s2)  PGA(fcst) I(obs-MMI)     I(fcst)   abs(Io-If)  Skill           obs GM             fcst GM     
   CI:DRE   2011-04-09T08:59:06Z   34.0   0.0109    0.0003    4.89    3.30      1.59        Poor        Moderate shaking/very light damage     Strong shaking/light damage potential
+
   CI:WES  101.0   37.0    0.2711      0.1067        3.06        3.66          2.07          2.24       0.17   Very Good         Light shaking/no damage potential        Light shaking/no damage potential
   CI:SWS  2011-04-09T08:59:06Z   38.0.0021    0.0004    2.96    3.44       0.48   Very Good       Moderate shaking/very light damage        Light shaking/no damage potential
+
   CI:BTC   108.0   37.0     0.1491      0.0123        1.33        0.35          1.27          1.00      0.27  Very Good              Not felt/no damage potential          Weak shaking/no damage potential
   CI:ERR   2011-04-09T08:59:06Z   57.0.0046    0.0002    3.88    2.70       1.19    Moderate        Light shaking/no damage potential       Moderate shaking/very light damage
+
   CI:SWS  121.3   37.0    0.0371      0.0464        0.35        1.49          1.00          1.38       0.38   Very Good         Weak shaking/no damage potential             Not felt/no damage potential
   CI:RXH   2011-04-09T08:59:06Z   64.0.0018    0.0002    2.77    2.46      0.31  Very Good        Light shaking/no damage potential         Light shaking/no damage potential
+
   CI:RXH   136.5   37.0    0.0496      0.0161        0.27        0.47          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
  CI:BTC   2011-04-09T08:59:06Z   65.0.0021    0.0001    3.06    2.32       0.74        Good         Light shaking/no damage potential       Moderate shaking/very light damage
+
   CI:ERR   138.5   37.0    0.0743      0.0226        0.75        0.68          1.00          1.00      0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
AZ:MONP2   2011-04-09T08:59:06Z   73.0.0014    0.0001    2.36    2.22       0.14   Very Good         Light shaking/no damage potential         Light shaking/no damage potential
+
AZ:MONP2   158.5   37.0    0.0296      0.0175        0.51        0.52          1.00          1.00       0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:SAL   2011-04-09T08:59:06Z   79.0   0.0058    0.0001    3.78    2.15      1.63        Poor        Light shaking/no damage potential       Moderate shaking/very light damage
+
  CI:SAL   161.9   37.0    0.0918      0.0139        0.71        0.40          1.00          1.00       0.00   Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:JEM  2011-04-09T08:59:06Z   97.0.0017    0.0001    2.59    1.75       0.83        Good         Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:BC3   182.1   37.0     0.0266      0.0061        0.24        0.16          1.00          1.00      0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:BOR  2011-04-09T08:59:06Z    98.0.0011    0.0001    2.74    1.76       0.97        Good         Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:JEM  184.2   37.0    0.0438      0.0103        0.61        0.29          1.00          1.00       0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
CI:NSS2  2011-04-09T08:59:06Z   107.6   0.0006    0.0001     1.49    1.62       0.13  Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   CI:BOR  185.7    37.0    0.0111      0.0101        0.16        0.28          1.00          1.00       0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
  CI:EML   2011-04-09T08:59:06Z  110.0.0011    0.0001    3.22     1.52      1.69        Poor          Weak shaking/no damage potential       Moderate shaking/very light damage
+
  CI:OLP   186.6   37.0    0.0296       0.0074        0.78        0.20          1.00          1.00      0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:OLP   2011-04-09T08:59:06Z  113.0.0021    0.0001     3.20    1.43       1.76        Poor          Weak shaking/no damage potential       Moderate shaking/very light damage
+
CI:NSS2   186.7    37.0    0.0358      0.0079        0.24        0.22           1.00          1.00      0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
AZ:LVA2   2011-04-09T08:59:06Z   114.3  0.0011    0.0001     2.11    1.51       0.61        Good         Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:EML   192.0   37.0    0.0257      0.0081        0.47        0.22          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:SDR   2011-04-09T08:59:06Z  115.0.0009    0.0001    2.62    1.42      1.20    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:SDR   192.7    37.0    0.0233       0.0073        0.46        0.20          1.00         1.00      0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:TOR   2011-04-09T08:59:06Z  117.6  0.0004    0.0001    1.42    1.47       0.05   Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   CI:BLY   195.5    37.0    0.0274      0.0032        0.32        0.08          1.00          1.00      0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:BC3   2011-04-09T08:59:06Z  119.0.0004    0.0001    1.53    1.42       0.11   Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   CI:CTC   198.4    37.0    0.0208      0.0067        0.18        0.18          1.00          1.00       0.00   Very Good             Not felt/no damage potential             Not felt/no damage potential
  CI:CTC   2011-04-09T08:59:06Z  119.0.0008    0.0001    1.52    1.44       0.08   Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   CI:TOR   201.5    37.0    0.0133      0.0071        0.07        0.19          1.00          1.00       0.00   Very Good             Not felt/no damage potential             Not felt/no damage potential
   AZ:TRO   2011-04-09T08:59:06Z  121.0.0005    0.0001    1.64    1.41       0.23   Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
AZ:LVA2   201.7    37.0    0.0238      0.0077        0.25        0.21          1.00          1.00       0.00   Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:DPP   2011-04-09T08:59:06Z  122.0.0012    0.0001    2.68    1.35       1.32    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:DPP   205.7    37.0    0.0322      0.0068        0.77        0.18          1.00          1.00       0.00   Very Good             Not felt/no damage potential             Not felt/no damage potential
   AZ:HWB  2011-04-09T08:59:06Z  125.0.0003    0.0001    1.53    1.32       0.21   Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   AZ:TRO   207.7    37.0    0.0121      0.0068        0.17        0.18          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   AZ:FRD  2011-04-09T08:59:06Z   128.6  0.0003    0.0001     0.85    1.31       0.45  Very Good         Weak shaking/no damage potential              Not felt/no damage potential
+
   AZ:HWB  208.7    37.0    0.0091      0.0066        0.19        0.18          1.00          1.00       0.00   Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:PMD   2011-04-09T08:59:06Z  131.0   0.0007    0.0001    2.23    1.28       0.95        Good         Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:SDG   211.7    37.0    0.0344       0.0057        0.47        0.15          1.00          1.00      0.00  Very Good             Not felt/no damage potential              Not felt/no damage potential
   AZ:PFO   2011-04-09T08:59:06Z  131.4  0.0002    0.0000    1.01    1.27      0.26  Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   AZ:CPE   213.6    37.0     0.0366      0.0058        0.86        0.15          1.00          1.00       0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   AZ:BZN   2011-04-09T08:59:06Z  132.0.0009    0.0000    1.99    1.26       0.73        Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   AZ:FRD   215.9    37.0    0.0085      0.0063        0.07        0.17          1.00          1.00      0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   AZ:CPE  2011-04-09T08:59:06Z   133.6  0.0018    0.0000    2.80    1.19      1.61        Poor         Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:PMD   216.1    37.0    0.0079      0.0059        0.11        0.16          1.00          1.00       0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   AZ:SND   2011-04-09T08:59:06Z  134.0.0011    0.0000    1.77    1.23       0.53        Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   AZ:PFO   217.3    37.0    0.0056      0.0060        0.04        0.16          1.00         1.00      0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:SDG   2011-04-09T08:59:06Z  134.4  0.0020    0.0000    2.53    1.17       1.35    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
+
   AZ:BZN   219.7    37.0    0.0130      0.0060        0.22        0.16          1.00          1.00       0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:PLM  2011-04-09T08:59:06Z  135.3  0.0008    0.0000    1.65    1.21      0.44  Very Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   AZ:SND   221.2    37.0    0.0157      0.0058        0.18        0.15          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:DNR   2011-04-09T08:59:06Z  136.0.0020    0.0000    2.34    1.20      1.13    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
+
   CI:PLM  222.5    37.0    0.0439      0.0058        0.41        0.15          1.00         1.00      0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   AZ:WMC   2011-04-09T08:59:06Z  139.7  0.0009   0.0000    2.22    1.16       1.06    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
+
   AZ:SOL   223.6    37.0    0.0889      0.0049        1.40        0.13          1.32          1.00      0.32  Very Good              Not felt/no damage potential         Weak shaking/no damage potential
   AZ:CRY   2011-04-09T08:59:06Z  142.0.0003    0.0000    0.77    1.12       0.36   Very Good         Weak shaking/no damage potential              Not felt/no damage potential
+
   CI:DNR   223.6   37.0    0.0353      0.0056        0.34        0.15          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   AZ:SOL   2011-04-09T08:59:06Z  145.0.0025    0.0000    2.46    1.04       1.42    Moderate          Weak shaking/no damage potential         Light shaking/no damage potential
+
   AZ:WMC   226.9    37.0    0.0187      0.0054        0.22        0.14          1.00          1.00       0.00   Very Good             Not felt/no damage potential              Not felt/no damage potential
   CI:MGE  2011-04-09T08:59:06Z  147.7  0.0005    0.0000    1.81    1.07       0.74        Good         Weak shaking/no damage potential         Weak shaking/no damage potential
+
   AZ:CRY   230.2    37.0    0.0041      0.0052        0.04        0.14          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:GOR   2011-04-09T08:59:06Z  153.0.0004    0.0000    1.73    0.97       0.76        Good              Not felt/no damage potential         Weak shaking/no damage potential
+
   CI:MGE  231.4    37.0    0.0252       0.0048        0.22        0.12          1.00          1.00      0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   AZ:KNW  2011-04-09T08:59:06Z   154.0  0.0002    0.0000    0.69    0.99      0.30  Very Good              Not felt/no damage potential              Not felt/no damage potential
+
   CI:BEL   233.6    37.0    0.0109      0.0042        0.04        0.11          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   AZ:RDM   2011-04-09T08:59:06Z  155.0   0.0013    0.0000    1.92    0.98       0.94        Good              Not felt/no damage potential         Weak shaking/no damage potential
+
   CI:IRM   234.2    37.0    0.0142      0.0031        0.08        0.08          1.00          1.00      0.00  Very Good              Not felt/no damage potential              Not felt/no damage potential
   CI:PSD   2011-04-09T08:59:06Z  155.0.0003    0.0000    1.33    0.98       0.35   Very Good              Not felt/no damage potential         Weak shaking/no damage potential
+
   CI:GOR   237.6    37.0     0.0176      0.0046        0.31        0.12          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:BEL   2011-04-09T08:59:06Z  157.1  0.0001    0.0000    0.00    0.96      0.96        Good              Not felt/no damage potential              Not felt/no damage potential
+
   AZ:KNW   240.9    37.0    0.0056      0.0045        0.04        0.12          1.00          1.00       0.00   Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:DEV   2011-04-09T08:59:06Z  168.0.0004    0.0000    1.51    0.84       0.67        Good              Not felt/no damage potential         Weak shaking/no damage potential
+
   CI:PSD   241.2    37.0    0.0057      0.0044        0.08        0.11          1.00          1.00      0.00  Very Good              Not felt/no damage potential              Not felt/no damage potential
   CI:BLY   2011-04-09T08:59:06Z  169.0.0001    0.0000    0.39    0.78       0.38   Very Good              Not felt/no damage potential              Not felt/no damage potential
+
   AZ:RDM   242.4    37.0    0.0114      0.0045        0.17        0.12          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:SLR  2011-04-09T08:59:06Z  169.0.0006    0.0000    1.56    0.83       0.74        Good              Not felt/no damage potential         Weak shaking/no damage potential
+
   CI:DEV   252.9    37.0    0.0112      0.0039        0.13        0.10          1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
   CI:MUR   2011-04-09T08:59:06Z  176.0.0021    0.0000    2.56    0.75      1.82        Poor             Not felt/no damage potential         Light shaking/no damage potential
+
   CI:SLR  256.1    37.0    0.0117      0.0039        0.16        0.10          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
   CI:MSJ   2011-04-09T08:59:06Z  177.1  0.0012    0.0000    2.12    0.75       1.37    Moderate             Not felt/no damage potential         Light shaking/no damage potential
+
   CI:MCT   258.4    37.0    0.0082      0.0032        0.03        0.08          1.00          1.00      0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:IRM   2011-04-09T08:59:06Z  180.4   0.0001    0.0000    0.00     0.70       0.70        Good              Not felt/no damage potential              Not felt/no damage potential
+
   CI:MUR   263.8    37.0    0.0225      0.0036        0.38        0.09          1.00          1.00       0.00  Very Good             Not felt/no damage potential             Not felt/no damage potential
   CI:MCT   2011-04-09T08:59:06Z  182.4  0.0000    0.0000    0.00    0.70      0.70        Good              Not felt/no damage potential              Not felt/no damage potential
+
   CI:MSJ   264.4   37.0    0.0409      0.0036        0.40        0.09          1.00         1.00       0.00  Very Good              Not felt/no damage potential              Not felt/no damage potential
   CI:BBS   2011-04-09T08:59:06Z  187.0.0005   0.0000    0.72    0.65       0.07   Very Good              Not felt/no damage potential              Not felt/no damage potential
+
   CI:BBS   274.5    37.0    0.0219      0.0032        0.17        0.08          1.00          1.00      0.00  Very Good              Not felt/no damage potential              Not felt/no damage potential
   CI:PER   2011-04-09T08:59:06Z  196.5  0.0004    0.0000    1.23    0.56       0.67        Good              Not felt/no damage potential         Weak shaking/no damage potential
+
   CI:PER   284.0    37.0    0.0125      0.0030        0.10        0.07          1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
 +
   CI:JVA   295.1    37.0    0.0116      0.0025        0.10        0.06          1.00          1.00       0.00  Very Good              Not felt/no damage potential             Not felt/no damage potential
 +
  CI:SDD  295.2    37.0    0.0520      0.0027        0.32        0.07          1.00          1.00      0.00  Very Good              Not felt/no damage potential              Not felt/no damage potential
 +
  CI:BBR  300.7    37.0    0.0123      0.0025        0.09        0.06          1.00          1.00      0.00  Very Good              Not felt/no damage potential              Not felt/no damage potential
 +
  CI:RSS  300.7    37.0    0.0144      0.0026        0.19        0.06          1.00          1.00      0.00  Very Good              Not felt/no damage potential              Not felt/no damage potential
 +
 
 +
Summary of 51 site-specific forecast ratings for event:14937372
 +
Very Good: 100.0%
 +
Good    :  0.0%
 +
Moderate :  0.0%
 +
Poor    :  0.0%
 +
Very Poor:  0.0%
  
Summary of 48 site-specific forecast ratings for event:10922333
 
Very Good: 37.5%
 
Good    : 33.3%
 
Moderate : 16.7%
 
Poor    : 12.5%
 
Very Poor:  0.0%
 
 
</pre>
 
</pre>
  
Line 153: Line 173:
 
#Shakemap files in order to produce performance summaries for any eew triggers. Shakemap files have observed pga and pgv for many cisn stations for larger california earthquakes. CTC matches a trigger to a shakemap through the anss event id. So to process a trigger, match the trigger to anss event, and match the anss event to a shakemap file.
 
#Shakemap files in order to produce performance summaries for any eew triggers. Shakemap files have observed pga and pgv for many cisn stations for larger california earthquakes. CTC matches a trigger to a shakemap through the anss event id. So to process a trigger, match the trigger to anss event, and match the anss event to a shakemap file.
 
#All configurations in place and all trigger files to process posted in inputs move to the src directory
 
#All configurations in place and all trigger files to process posted in inputs move to the src directory
#gen_ctc_update.py
+
#gen_ctc.py
 
#This generates a results and puts the results into the output directory.
 
#This generates a results and puts the results into the output directory.
 
#Updates are sequenced to avoid overwriting. Update files that can be viewed as an html file with links to images and data files used in the summaries.
 
#Updates are sequenced to avoid overwriting. Update files that can be viewed as an html file with links to images and data files used in the summaries.
#Currently, the system will produce only one summary. This is an event specific forecast skill summary defined in a 2008 CISN testing meeting. Example shown below. Further performance summaries will be added.
+
#Currently, the system will produces one summary per event per EEW algorith. The ground motion forecast evaluation report is an event specific forecast skill summary defined in a 2008 CISN testing meeting. Example shown below. Further performance summaries will be added.
  
 
== Intensity Equations ==
 
== Intensity Equations ==
Line 202: Line 222:
 
#The report id never changes. The intention was that this uniquely identified the file. But it is currently fixed.
 
#The report id never changes. The intention was that this uniquely identified the file. But it is currently fixed.
 
#Forecasts PGV and Mag only
 
#Forecasts PGV and Mag only
#Reports both algorithm and alert time, which always seem to have a delta of 2 seconds. This may mean difference between the two is the amount of data needed to produce the alert
+
#Reports both algorithm and alert time, the delta seems to vary from 2 to 5 seconds. This may represent the processing time of algorithm.
 
#Multiple forecasts per trigger file
 
#Multiple forecasts per trigger file
 
#Multiple events per trigger file
 
#Multiple events per trigger file
Line 234: Line 254:
  
 
== Migration to ShakeAlert Testing ==
 
== Migration to ShakeAlert Testing ==
Current capabilities are retrospective.
+
Current capabilities are retrospective. They can be downloaded and used to process Trigger files and produce retrospective reports. The intention is that the current tools will be useful in prospective testing of ShakeAlert. Current trigger exchange format may be changed to use a version of ShakeMap format which would be more standardized.
  
 
== Processing Flow ==
 
== Processing Flow ==
Line 252: Line 272:
 
*Example ANSS Catalog File
 
*Example ANSS Catalog File
  
== Software Download ==
+
== Central California Event ==
CTC software is intended to be general purpose and will be posted online as open-source. SCEC software version is date-based, something like v11.4.0
+
 
 +
During a recent central california event several issues emerged. First the shakemap results that arrived via the shakereader seemed to have only two rows of stations.
 +
Then, via the ShakeMap web site, we were able to find a complete ShakeMap file.
 +
Next, the event arrived by email but was not available in the ANSS catalog. We had to manually enter the event information from the email into an anss catalog entry for use in processing.
 +
Then we saw that all the stations in the ShakeMap observation file were NorCal stations, however the forecast stations in the VS trigger were all southern California stations. This looks to be a limitation of using ShakeMap files for source of observational data
 +
 
 +
== ANSS Catalog Format ==
 +
 
 +
Your search parameters are:
 +
 
 +
    catalog=ANSS
 +
    start_time=2011/01/01,00:00:00
 +
    end_time=2011/06/20,00:00:00
 +
    minimum_latitude=31.00
 +
    maximum_latitude=42.00
 +
    minimum_longitude=-125.00
 +
    maximum_longitude=-112.00
 +
    minimum_magnitude=4.0
 +
    maximum_magnitude=10
 +
    event_type=E
 +
 
 +
Date      Time            Lat      Lon  Depth  Mag Magt  Nst Gap  Clo  RMS  SRC  Event ID
 +
----------------------------------------------------------------------------------------------
 +
<pre>
 +
2011/01/03 12:06:36.88  38.2473 -112.3398  5.40  4.55  ML  25  46  27 0.23  UU             
 +
2011/01/08 00:10:16.74  37.2872 -121.6582  10.07  4.10  Mw  223  55    6 0.18  NC      71506865
 +
2011/01/11 01:36:19.82  37.1658 -117.3770  12.41  4.06  ML  16 161  18 0.12  NN        324385
 +
2011/01/12 08:51:03.68  36.7712 -121.4963  8.04  4.50  Mw  105  35    2 0.19  NC      71508850
 +
2011/01/22 17:26:09.19  38.5496 -118.3800  13.94  4.01  ML  16 229  15 0.11  NN        325137
 +
2011/01/22 17:29:25.70  38.5387 -118.3798  8.35  4.32  ML  18 188  16 0.12  NN        325135
 +
2011/01/22 19:21:04.62  38.5443 -118.3781  14.20  4.15  ML  16 227  16 0.10  NN        325134
 +
2011/01/23 05:29:59.80  31.2170 -115.7000  16.00  4.50  ML  155          0.00  ECX 201101231009
 +
2011/01/23 05:55:30.20  31.0600 -115.6900  17.40  4.00  Mc  29          0.00  ECX 201101231011
 +
2011/01/23 06:47:24.55  32.2165 -115.3213  1.87  4.15  ML  15          0.20  CI      14920876
 +
2011/01/26 00:11:28.00  31.2360 -115.6530  5.00  4.50  Mb  55          0.00  ECX 201101261001
 +
2011/02/13 23:25:09.54  32.1707 -115.2835  13.46  4.08  ML  15          0.19  CI      10892285
 +
2011/02/18 17:47:35.77  32.0470 -115.0622  15.00  5.07  Mw  12          0.26  CI      14937372
 +
2011/02/20 19:34:39.98  32.0468 -115.2518  10.00  4.50  Mw    9          0.18  CI      14937988
 +
2011/02/23 04:49:41.51  39.4993 -122.9488  12.75  4.29  Mw  60  48    2 0.13  NC      71527390
 +
2011/02/23 04:50:14.36  39.4938 -122.9565  11.26  4.21  Mw  41  69    2 0.10  NC      71048354
 +
2011/03/01 02:19:47.01  38.8153 -122.8200  2.95  4.46  Mw  76  24    1 0.08  NC      71530230
 +
2011/03/31 20:22:55.70  32.3392 -115.2282  10.69  4.35  ML  15          0.29  CI      14960652
 +
2011/04/07 13:40:57.80  31.2350 -115.6170  5.00  4.10  Mb  134          0.00  ECX 201104071036
 +
2011/04/09 08:58:49.22  32.6055 -115.7237  9.38  4.13  ML  43          0.36  CI      10922333
 +
2011/04/11 13:00:12.60  31.2170 -115.5920  5.00  4.10  ML  15          0.00  ECX 201104111065
 +
2011/04/13 22:10:08.34  38.3730 -118.7450  16.00  4.30  Mw  128          0.00  REN 201104131069
 +
2011/04/13 22:16:08.71  38.3860 -118.7420  15.40  4.10  Mw  26          0.00  REN 201104131071
 +
2011/04/17 00:45:37.00  38.3940 -118.7340  15.40  4.60  ML  94          0.00  REN 201104171002
 +
2011/04/27 19:19:20.58  38.4140 -118.7180  2.60  5.50  Mb  37          0.00  REN 201104271026
 +
2011/05/03 21:41:49.82  36.3953 -117.8703  2.24  4.45  ML  44          0.18  CI      14978940
 +
2011/06/09 15:22:12.37  32.6630 -115.7365  15.08  4.42  ML  64          0.44  CI      14998780
 +
2011/06/15 12:29:07.64  31.1920 -115.6160  10.00  4.20  ML  40          1.22  NEI 201106151008
 +
</pre>
 +
 
 +
 
 +
== CTC Software Versions ==
 +
 
 +
Tag Descriptions:
 +
*11_05_24
 +
**This is the first tagged version with skill summary reports considered accurate. Summaries produced with versions prior to this have units problems when converting from PGA to acceleration (cm/s/s).
  
Development Version
+
Tag Descriptions:
Version:11.5.0
+
*11.5.0
 +
**This distribution is capable of processing performance summaries for new events. Retrieve this distribution and setup the authdata inputs correctly, put the vs trigger file or onsite trigger file in the inputs directory, and run gen_ctc.py and the software should produce a number of performance summaries without errors. This is the distribution described in the SCEC Year 2 report to USGS.
  
CTC Software is posted in git repository on github.
 
  
*[https://github.com/ Git Hub Home Page]
+
== Software Download ==
*Project Name: git@github.com:maechlin/ctc
+
CTC software is intended to be general purpose and will be posted online as open-source. SCEC software version is date-based, something like 11_05_24
*[git://github.com/maechlin/ctc.git Read-Access Source Code]
 
  
If you have git installed, and have entered a public key created, and uploaded to github, you can retrieve the codes with the following commands.
+
If you have a SCEC account you can retrieve the project files with the following commands.
  
%git remote git@github.com:maechlin/ctc
+
CTC Software is posted in repository on SCEC SVN repository
 +
*svn co http://source.usc.edu/svn/ctc/tags/11_05_24 ctc
 +
*[http://source.usc.edu/svn/ctc CTC SVN Trunk]
 +
 
 +
Latest development version is available with the command:
 +
*svn co http://source.usc.edu/svn/ctc ctc
  
 
== Contributors ==
 
== Contributors ==
Line 272: Line 355:
 
*[http://www.seismolab.caltech.edu/index.html Caltech Seismology]
 
*[http://www.seismolab.caltech.edu/index.html Caltech Seismology]
 
*[http://seismo.berkeley.edu/ UC Berkeley Seismological Lab]
 
*[http://seismo.berkeley.edu/ UC Berkeley Seismological Lab]
 +
*[http://earthquake.usgs.gov/earthquakes/recenteqscanv/ CISN Recent Earthquakes]
  
 
== Related Entries ==
 
== Related Entries ==
 
*[[Earthquake Early Warning]]
 
*[[Earthquake Early Warning]]
 
*[[M8 EEW Analysis]]
 
*[[M8 EEW Analysis]]
 +
*[http://www.ncedc.org/cnss/catalog-search.html ANSS Catalog Search Site]
  
 
== See Also ==
 
== See Also ==
 
*[[Main Page]]
 
*[[Main Page]]
 +
 +
== CISN EEW Testing Data Sets ==
 +
 +
'''CISN EEW Trigger Summary'''
 +
*[http://scec.usc.edu/research/eew/summary/California/shakecast-tests/ CTC Test Results Directory]
 +
*[http://scecdata.usc.edu/earlywarning/ CISN EEW Algorithm Testing Site]
 +
*[http://scecdata.usc.edu/maechling/Cumulative_Trigger_Summary.zip CISN Cumulative Trigger Summary] (2008-2010) (15MB Zip File containing 75MB html File)
 +
 +
'''CISN EEW Station Lists'''
 +
 +
CISN provided the following list of stations used in current CISN EEW processing
 +
 +
*[http://scecdata.usc.edu/maechling/stations_nc_04.13.2010.txt Northern CISN Station List]
 +
*[http://scecdata.usc.edu/maechling/stations_sc_04.13.2010.txt Southern CISN Station List]
 +
*[http://scecdata.usc.edu/maechling/cisn_eew_10Oct_2010_v1.kml Combined CISN EEW Station List (KML Format)]
 +
*[http://scec.usc.edu/websims/repo/m8/data/trace.kml Southern San Andreas Fault Trace lon/lat (KML format)]
 +
 +
The ShakeMap authorized data source may provide the observational data used to validate short term ground motion forecasts.
 +
 +
*[http://earthquake.usgs.gov/earthquakes/shakemap/rss_info.php ShakeMap Rss System]
 +
*[http://www.ncedc.org/anss/catalog-search.html ANSS Seismic Data Center]
 +
*[http://www.data.scec.org/ SCEC Data Center at Caltech]
 +
*[http://www.ncedc.org/ Northern California Data Center at UC Berkeley]
 +
 +
'''ShakeAlert CTC Trac System'''
 +
 +
*[http://northridge.usc.edu/trac/eew CISN EEW Trac System]
 +
 +
== EEW-Related SCEC Research Documents and Links ==
 +
 +
'''2010'''
 +
*[http://scec.usc.edu/scecwiki/images/c/c8/CTC_14Oct2010_v6.ppt CISN EEW Workshop Oct 2010] (7MB ppt File)
 +
*[http://scecdata.usc.edu/maechling/CTC_Testing_Plans_1April2010.pptx CTC Development Plans April 2010 (12Mb pptx)]
 +
*[[M8_EEW_Analysis]]
 +
*[http://scec.usc.edu/scecwiki/images/3/3f/EEW_Algor_testing_v01b.pdf CISN EEW Testing Document] (Updated July 2010)
 +
 +
'''2009'''
 +
*[http://scec.usc.edu/scecwiki/images/f/ff/EEW_Testing_summary_v2.pdf CISN EEW Testing Summary 2009]
 +
*[http://scecdata.usc.edu/maechling/ShakeAlert_3YrSummary.pdf Fall AGU 2009 (Cua et al)]
 +
**[[File:Cua_ShakeAlert_3yrSummary.pdf]] Alternative Download Site
 +
*[http://scecdata.usc.edu/maechling/ShakeAlert_User_Poster.pdf Fall AGU 2009 (Hellweg et al)]
 +
*[http://scecdata.usc.edu/maechling/Maechling_EEW_AGU_Fall_2009_v4.pdf Fall AGU 2009 (Maechling et al)]
 +
*[http://scecdata.usc.edu/maechling/Maechling_EEW_SCEC_AM_2009_v2.pdf SCEC Annual Meeting Poster 2009 (Maechling et al)]
 +
*[http://scecdata.usc.edu/maechling/mboese_IST_LunchBunch2009.pdf Maren Bose EEW Presentation (2009)]
 +
 +
'''2008'''
 +
*[http://scec.usc.edu/scecwiki/images/9/99/CISN_AGU08F.ppt Fall 2008 AGU EEW Presentation (Hellweg et al)]
 +
*[http://scecdata.usc.edu/maechling/Maechling_CSEP_System_Overview_120508_v3.ppt CSEP Testing Center Overview (Dec 2008) (Maechling et al) (3MB ppt)]
 +
*[http://scecdata.usc.edu/maechling/Maechling_Prospective_Testing_06102008.ppt Prospective Testing Overview (EEW and CSEP) (Maechling et al) (3MB ppt)]
 +
*[http://scecdata.usc.edu/maechling/Maechling_EEW_SCEC_Japan_2009_v1.pdf EEW Meeting In Kyoto Poster 2008 (Maechling et al) (PDF)]
 +
 +
'''2006'''
 +
 +
*[http://scecdata.usc.edu/maechling/K4_Erdik.pdf Mustafa Erdik EEW Presentation (2006)]
 +
*[http://scecdata.usc.edu/maechling/O06_UJNR200611081.pdf Shin'ya TSUKADA EEW Presentation (2006)]
 +
 +
[[File:Eew_stations_cal_m8.png|340px|thumb|right|Fig 1: California Station List with M8 Epicenter.]]
 +
 +
 +
[[Image:CaliforniaNatLab.png|340px|thumb|right|Fig 2: California Testing Region (inner) and Catalog Region (outer).]]
 +
 +
[[File:scec.eew.EEWTest.eewTest_SHAKEMAP-Test_Event10736069.png|340px|thumb|right|Fig 3: Plotting of ShakeMap observation data for Event in Southern California.]]
 +
 +
[[File:scec.eew.EEWTest.eewTest_SHAKEMAP-Test_ZoomEvent10736069.png|340px|thumb|right|Fig 4: Plotting of ShakeMap observation data for Event in Southern California.]]
 +
 +
[[File:Define_Reference_Events.png|340px|thumb|right|Fig 5: Travel time curves show region for which no warnings are possible because of travel time and processing delays.]]
 +
 +
[[File:shakemap_color_scale.png|340px|thumb|right|Fig 6: ShakeMap color scale shows instrumental intensity estimates or observations.]]
 +
 +
[[Image:GroundMotionEvaluation.png|340px|thumb|right|Fig 6: Scheme for comparing PGV and PGA values.]]
 +
 +
[[Image:EEWDisplay.png|340px|thumb|right|Fig 6: Prototype ShakeAlert Display (Image Credit: Kalpesh Solanki)]]
 +
 +
[[Image:CA_Eq_Probability_by_Mag.jpg|340px|thumb|right|Fig 7: CA EQ Probabilities by Magnitude]]
 +
 +
[[Image:CISN_EQ_Example.jpg|340px|thumb|right|Fig 8: CISN EQ Example of Warning Times.]]
 +
 +
 +
== Catalog Comparison ==
 +
We looked at current eew results on the CISN algorithm testing site [http://scecdata.usc.edu/earlywarning/ CISN EEW Algorithm Testing Site]]. The web system provides several summaries. We compared the earthquakes on two summaries available on the web site which are: (1) Catalog summary and (2) Cumulative Trigger Summary. EEW system list of ANSS events against ANSS list of triggers.
 +
 +
To interpret these trigger summaries were designed to compare eew performance anywhere in california. The ANSS earthquake catalog is the reference observational data.
 +
 +
The EEW algorithm testing system filters the catalog to include only earthquakes within California region. We based the EEW California region on the CSEP region. There are two regions defined. California testing region. In this region, the systems are expected to forecast earthquakes, or provide EEW alerts. There is a slightly larger region which is the catalog region. When we retrieve the catalog and we restrict our request to a region, we may miss earthquakes right on the edge due to uncertainties in the earthquake locations. So we use a reference earthquake data set from a slightly larger region, the catalog region.
 +
 +
These cumulative summaries work on the site, but they are so large now, with multple years of info that it produces multi-MB htlm files. We don't have good parsing of these, but we worked with data from these two summaries retrieved on 12 Oct 2010.
 +
 +
We found that he catalog list has two events not included in the trigger list. I believe these in the region between the testing region and the catalog region. The two catalog events are listed:
 +
 +
NC 71330090 1/4/2010 14:24:54 4.12 40.2773 -123.868 30.74
 +
CI 14858988 10/12/2010 3:00:34 4.27 32.1582 -115.29 5.58
 +
 +
We need to confirm this one is plausible (32.1582,-115.29) but this is in Humbolt (40.2773,-123.868)so it is not out of the teting region. This discrpency between ANSS events in the two lists remains.
 +
 +
The total events in the trigger list is 146 (1Jan2010 through 12Oct2010)
 +
 +
== CTC Abstract for SCEC Annual Meeting 2010 ==
 +
 +
Testing Regional-scale Ground Motion Estimates for the CISN ShakeAlert Project
 +
Philip Maechling, Maria Liukis, Thomas H. Jordan
 +
 +
Our SCEC research team has developed initial capabilities of the CISN Testing Center (CTC) as a part of the USGS-funded CISN ShakeAlert Project. The CTC is designed to compare regional-scale ground motion estimates, such as peak ground motion estimates, against observed ground motion data, for significant California Earthquakes. The CTC is designed to identify which ShakeAlert ground motion estimation methods produce the best fit to observed ground motions.
 +
 +
Scientific goals of the CTC include (a) development of consensus among SCEC researchers on meaningful measures of comparison between ground motion estimates and observations, (b) collection and analysis of ShakeAlert performance information, and (c) integration of standard Probabilistic Seismic Hazard Analysis (PSHA) terms and practices into EEW. The technical goal of the CTC is to develop an automated system that can comparing ground motion estimates against ground motion observations. The CTC system used the CSEP testing framework to automate the evaluation of CISN ShakeAlert rapid earthquake parameters including ground motion forecasts for each event.
 +
 +
As part of this year’s CTC development, we updated the CSEP Testing Framework to use the USGS ShakeMap RSS Feed as an authorized source of event-based ground motion observations. As ShakeAlert systems produces rapid ground motion estimates for significant California events, the CTC will evaluate those ShakeAlert ground motion estimates against CISN ground motion observation using goodness-of-fit algorithms defined by CISN scientific groups.
 +
 +
Our ShakeAlert ground motion forecast testing shares scientific and technical goals with several other SCEC activities including validation of attenuation relationships, synthetic ShakeMaps, velocity model and wave propagation model goodness of fit measures, and short term ground motion forecast testing. The CTC system is designed to support CISN ShakeAlert testing and to also be applicable for evaluation of other ground motion estimation methods.
 +
 +
[[File:eew1.png|340px|thumb|right|Fig 7: JMA diagram showing EEW concepts and types of EEW system implementations.]]
 +
 +
[[File:eew2.png|340px|thumb|right|Fig 8: JMA diagram showing potential users of EEW ground motion forecasts.]]
 +
 +
[[File:SanFranciscoChronicleSciSecFig.jpg|340px|thumb|right|Fig 9: San Francisco Chronicle graphic showing EEW concepts in California.]]
 +
 +
[[File:EEWsuperhero.jpg|340px|thumb|right|Fig 10: Japanese Public EEW Awareness Information.]]
 +
 +
[[File:CAstations_Optimal.png|340px|thumb|right|Fig 11: Current and proposed station locations for CISN.]]
 +
 +
== Additional ShakeAlert Information ==
 +
*[http://www.cisn.org/ CISN Home Page]
 +
*[http://www.usgs.gov/newsroom/article.asp?ID=2366&from=rss_home USGS ShakeAlert Press Release]
 +
*[http://www.seismolab.caltech.edu/ Caltech Seismo Lab]
 +
*[http://seismo.berkeley.edu/ U.C. Berkeley Seismo Lab]
 +
*[http://earthquake.usgs.gov/regional/sca/ USGS Pasadena Seismic Network Operations]
 +
*[http://earthquake.usgs.gov/regional/nca/ USGS Menlo Park Seismic Network Operations]
 +
*[http://qcn.stanford.edu/ Quake Catcher Network]
 +
 +
== ShakeAlert CTC Software Stack Description ==
 +
 +
The following packages contribute to the software stack used by EEW software:
 +
 +
*awk
 +
*sed
 +
*java
 +
*python and it's external libraries: matplotlib, basemap, numpy, scipy, xml
 +
 +
== See Also ==
 +
 +
*[http://qcn.stanford.edu/ Quake Catcher Network]

Latest revision as of 06:32, 21 June 2011

SCEC NSF-USGS-words logom.png
CISNlogos.png
Usgs-logo-color.jpg

CISN Testing Center (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.

CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.

Software Requirements

CISN Testing Documents

Example Ground Motion Forecast Summaries

ANSS Event ID: 10922333
Magnitude: 4.13
Origin Time: 2011/04/09 08:58:49
Hypocenter: 32.6055 -115.7237 9.38 km
Forecast Sources: ssi://caltech.gps.edu/onsite/2.1.0
First Forecast Time: 2011/04/09 08:59:21
First Forecast Secs after Origin: 32
Net:Sta  Dist(km) Wd(secs) PGV(obs-cm/s) PGV(fcst)  PGA(obs-cm/s2)  PGA(fcst)  I(obs-MMI)     I(fcst)   abs(Io-If)  Skill           obs GM              fcst GM    
  CI:WES    17.3    32.0     0.2000       0.3816                                     1.93          2.52       0.59        Good         Light shaking/no damage potential          Weak shaking/no damage potential
  CI:DRE    34.0    35.0     0.1071       0.2022                                     1.36          1.94       0.58        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
  CI:IBP    35.5    35.0     0.0422       0.1042                                     1.00          1.34       0.34   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
  CI:ERR    57.7    39.0     0.0449       0.1317                                     1.00          1.55       0.55        Good          Weak shaking/no damage potential              Not felt/no damage potential

Summary of 4 site-specific forecast ratings for event:10922333
Very Good:  25.0%
Good     :  75.0%
Moderate :   0.0%
Poor     :   0.0%
Very Poor:   0.0%

Example Virtual Seismologist Forecast Performance

ANSS Event ID: 14937372
Magnitude: 5.07
Origin Time: 2011/02/18 17:47:35
Hypocenter: 32.047 -115.0622 15.0 km
Forecast Sources: smi://caltech.gps.edu/onsite/cinnabar/vs/eewvs
First Forecast Time: 2011/02/18 17:48:12
First Forecast Secs after Origin: 37
Net:Sta  Dist(km) Wd(secs) PGV(obs-cm/s) PGV(fcst)  PGA(obs-cm/s2)  PGA(fcst)  I(obs-MMI)     I(fcst)   abs(Io-If)  Skill           obs GM              fcst GM    
  CI:WES   101.0    37.0     0.2711       0.1067         3.06         3.66           2.07          2.24       0.17   Very Good         Light shaking/no damage potential         Light shaking/no damage potential
  CI:BTC   108.0    37.0     0.1491       0.0123         1.33         0.35           1.27          1.00       0.27   Very Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:SWS   121.3    37.0     0.0371       0.0464         0.35         1.49           1.00          1.38       0.38   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
  CI:RXH   136.5    37.0     0.0496       0.0161         0.27         0.47           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:ERR   138.5    37.0     0.0743       0.0226         0.75         0.68           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
AZ:MONP2   158.5    37.0     0.0296       0.0175         0.51         0.52           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:SAL   161.9    37.0     0.0918       0.0139         0.71         0.40           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BC3   182.1    37.0     0.0266       0.0061         0.24         0.16           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:JEM   184.2    37.0     0.0438       0.0103         0.61         0.29           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BOR   185.7    37.0     0.0111       0.0101         0.16         0.28           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:OLP   186.6    37.0     0.0296       0.0074         0.78         0.20           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
 CI:NSS2   186.7    37.0     0.0358       0.0079         0.24         0.22           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:EML   192.0    37.0     0.0257       0.0081         0.47         0.22           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:SDR   192.7    37.0     0.0233       0.0073         0.46         0.20           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BLY   195.5    37.0     0.0274       0.0032         0.32         0.08           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:CTC   198.4    37.0     0.0208       0.0067         0.18         0.18           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:TOR   201.5    37.0     0.0133       0.0071         0.07         0.19           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
 AZ:LVA2   201.7    37.0     0.0238       0.0077         0.25         0.21           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:DPP   205.7    37.0     0.0322       0.0068         0.77         0.18           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:TRO   207.7    37.0     0.0121       0.0068         0.17         0.18           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:HWB   208.7    37.0     0.0091       0.0066         0.19         0.18           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:SDG   211.7    37.0     0.0344       0.0057         0.47         0.15           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:CPE   213.6    37.0     0.0366       0.0058         0.86         0.15           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:FRD   215.9    37.0     0.0085       0.0063         0.07         0.17           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:PMD   216.1    37.0     0.0079       0.0059         0.11         0.16           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:PFO   217.3    37.0     0.0056       0.0060         0.04         0.16           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:BZN   219.7    37.0     0.0130       0.0060         0.22         0.16           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:SND   221.2    37.0     0.0157       0.0058         0.18         0.15           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:PLM   222.5    37.0     0.0439       0.0058         0.41         0.15           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:SOL   223.6    37.0     0.0889       0.0049         1.40         0.13           1.32          1.00       0.32   Very Good              Not felt/no damage potential          Weak shaking/no damage potential
  CI:DNR   223.6    37.0     0.0353       0.0056         0.34         0.15           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:WMC   226.9    37.0     0.0187       0.0054         0.22         0.14           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:CRY   230.2    37.0     0.0041       0.0052         0.04         0.14           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:MGE   231.4    37.0     0.0252       0.0048         0.22         0.12           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BEL   233.6    37.0     0.0109       0.0042         0.04         0.11           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:IRM   234.2    37.0     0.0142       0.0031         0.08         0.08           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:GOR   237.6    37.0     0.0176       0.0046         0.31         0.12           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:KNW   240.9    37.0     0.0056       0.0045         0.04         0.12           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:PSD   241.2    37.0     0.0057       0.0044         0.08         0.11           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  AZ:RDM   242.4    37.0     0.0114       0.0045         0.17         0.12           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:DEV   252.9    37.0     0.0112       0.0039         0.13         0.10           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:SLR   256.1    37.0     0.0117       0.0039         0.16         0.10           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:MCT   258.4    37.0     0.0082       0.0032         0.03         0.08           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:MUR   263.8    37.0     0.0225       0.0036         0.38         0.09           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:MSJ   264.4    37.0     0.0409       0.0036         0.40         0.09           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BBS   274.5    37.0     0.0219       0.0032         0.17         0.08           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:PER   284.0    37.0     0.0125       0.0030         0.10         0.07           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:JVA   295.1    37.0     0.0116       0.0025         0.10         0.06           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:SDD   295.2    37.0     0.0520       0.0027         0.32         0.07           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:BBR   300.7    37.0     0.0123       0.0025         0.09         0.06           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential
  CI:RSS   300.7    37.0     0.0144       0.0026         0.19         0.06           1.00          1.00       0.00   Very Good              Not felt/no damage potential              Not felt/no damage potential

Summary of 51 site-specific forecast ratings for event:14937372
Very Good: 100.0%
Good     :   0.0%
Moderate :   0.0%
Poor     :   0.0%
Very Poor:   0.0%

Interpretation of Results

When comparing perfmance of these two EEW algorithms here are few things to consider. First, VS is an associator-based system. It uses phase picks to get a hypocenter and some other technique to get the magnitude. Once the event is declared, VS starts to forecast time series of parameters. In the current summaries, we use first reports from VS. Other plots showing more details of the time series will be developed. Once the VS forecasting starts, it updates event parameters lat,lon,mag,origin time. It also forecast pgv and pga at a set of CISN station locations. We are not sure how the set of stations to forecast for are decided, but it is fixed for the full trigger report. Event parameters and pgv and pga for all sites are updated every timestep (5 seconds). The number of sites varies by event we've seen between 20 and 50 per M4-5 events.

The OnSite algorithm is intended to be a faster algorithm because it is a single station approach. It produces a trigger, a final mag, a peak pgv.

We currently produce the intensity-skill summary using all available onsite triggers for each event. We combine the pgv forecasts to create the map.

However, a better comparison to the VS may be to use the magnitude forecast at each trigger, and use that to forecast ground motions at station sites, using an attenuation relationship.

VS must use this kind of approach to produce it's groundmotion forecasts. I think the On Site trigger might have enough information to use VS ground motion forecast to produce ground motion forecasts at multiple stations for each on-site trigger.

Trigger Matching Algorithms

The CTC software is designed to start processing when given one, or more, EEW trigger files produced by CISN EEW systems. The intent of the EEW Evaluation system is that it will compare earthquake and ground motion forecasts from these trigger files against the observational data collected by the CISN and other network operators. The reference observational catalog is the ANSS catalog. We are uncertain the time lag before recent earthquakes appear in this catalog. Given a trigger file, each forecast is matched to a ANSS event. Then observational data about the event is used to produce goodness of fit metrics for the forecasting system.

Developing the algorithm testing system, we learned there are many ways to match triggers to ANSS events. The system should automatically do the matching and also allow users to override automatic matching results.

The matching for current CTC is done differently for VS triggers and On Site Triggers.

The On Site triggers contain an associated CI event. We assume a network CI and match CI:EventID to the ANSS catalog. This usually works, but we have seen a Mamonth event trigger with a CI event id, but in the ANSS catalog, it was under an nothern california event id. To process this event, we added a duplicate of the norcal ANSS event, and put the duplicate into the anss_catalog file, and then the trigger could be processed.

The VS trigger does not have an associated event id. This is more typical case, or the expected case because the EEW algorithms are declaring their own events before CISN events are declared.

The current system takes the vs trigger file. VS trigger files have one earthquake per file. The system finds the Vs trigger time, and then finds an ANSS event with the closest origin time to the trigger time. The trigger and event that get matched must be within configuration.parameter 240 seconds of each other. The matches are put into an output file.

The matching system is separated from the other parts of the processing. It may require humans to correctly match triggers to events. As designed, humans can update or correct the automatically generated matching file. The rest of the CTC codes only use the matches in the matching file, regardless of how they were made.

Input Configuration Files

File-based inputs of CISN configuration information is needed.

  • CISN station list with network,station,channel,location,lat,lon
NP 1821 HNZ -- 37.798599 -122.268402 
  • Set of Shakemap files current in stationlist.txt format
  • ANSS catalog file of events in format from web page
2010/06/02 09:30:47.55  32.0432 -115.2220  10.00  3.96   ML   12          0.21  CI      14731660

Usage Overview

To process EEW trigger files for current or future events you can take the following steps.

  1. Install ctc software distribution from scec's github repository. Before any formal release, the expectation is that the current version is best available. Once we have release versions, select the one most appropriate for your use.
  2. Setup CISN confirmation files. Default files are included in distribution, but may need to be updated as network changes. These include stations list, ANSS catalog for a specific region, and time period, and magnitude limit.
  3. Collect ShakeMap event files. For the CTC performanc analysis processing to work, it requires the observational data from ShakeMap stationlist.txt files. We currently use the .txt format files, but we believe the information content is equivalent to the shakemap xml files, so either can be used.
  4. If your trigger files are VS files, you need to trigger2anssevent.txt file to process VS triggers. Default programming is to generate this from ANSS catalog and trigger files. Since we get EEW triggers only for larger events, matching isn't currently a problem. If it is, the system supports manually creating the trigger to event matching file to ensure each trigger is matched and compared to the correct ANSS event.
  5. the main directory is called ctc
  6. In ctc directory, there are three subdirectories:
    1. Inputs
    2. Outputs
    3. src
  7. In the Inputs directory. There are three subdirectories:
    1. authdata
    2. vs_triggers
    3. onsite_triggers
  8. Put the VS trigger files into the vs_triggers subdirectory. Put the OnSite trigger files into the onsite_triggers subdirectory.
  9. In authdata, there are two files and a subdirectory
    1. anss_catalog_2011.txt
    2. cisn_stations_13April2011.txt
    3. shakemaps
  10. The anss_catalog contains events retrieved through the anss web site
  11. The cisn catalog contains sncl and lat,lon for most cisn stations. This should include all stations at which ground motions are forecast by any of the EEW algorithms. When processing a trigger file, a station may be reported as missing. Find out the station information from somewhere, and add it into this catalog to continue processing
  12. Shakemap files in order to produce performance summaries for any eew triggers. Shakemap files have observed pga and pgv for many cisn stations for larger california earthquakes. CTC matches a trigger to a shakemap through the anss event id. So to process a trigger, match the trigger to anss event, and match the anss event to a shakemap file.
  13. All configurations in place and all trigger files to process posted in inputs move to the src directory
  14. gen_ctc.py
  15. This generates a results and puts the results into the output directory.
  16. Updates are sequenced to avoid overwriting. Update files that can be viewed as an html file with links to images and data files used in the summaries.
  17. Currently, the system will produces one summary per event per EEW algorith. The ground motion forecast evaluation report is an event specific forecast skill summary defined in a 2008 CISN testing meeting. Example shown below. Further performance summaries will be added.

Intensity Equations

There are formulas presented in the CISN testing document that provide formula's to convert PGV and PGA, or just PGV to intensity. We use first for VS intenstiy and other for On Site.

The formulas were somewhat vague on what type of logs to use, and what units to use in the input PGV values. We ended up with mms and that seems to be the right formula. Below is psuedo code showing how we implemented the algorithms.

These algorithms seems to consider non-integer intensity values so we support those here.

def convertVelocity2Intensity(pgv_val):
    pgv_in_mms = pgv_val*10.0
    ite = (2.10 * math.log10(pgv_in_mms)) + 3.40
    if ite >= 5:
        ite = (3.47 * math.log(pgv_in_mms)) + 2.35
    return ite

def convertToIntensity(pgv,pga):
    #
    # inputs to floats, returns a float
    #
    pgv_in_mms = pgv*1000.0
    pga_in_mms2 = pga*1000.0
    inten = (2.20 * math.log10(pga_in_mms2)) + 1
    if inten < 0.0:
        inten = 0.0
    if int(inten) > 5.0:
        inten = (3.66 * math.log10(pga_in_mms2)) - 1.66
    if int(inten) > 7.0:
        inten = (3.47 * math.log10(pgv_in_mms)) + 2.35
    return inten

Performance Evaluation Summaries

Two descriptions of performance summaries include:

  • CISN Testing Document
  • CTC Group Evaluation Collection

We separate the summaries into "Event" Summaries and "Catalog" Summaries where catalog means any set of earthquakes, and the summary must make it clear what criteria were used to define the catalog.

CISN On Site Trigger

An example CISN Trigger file is posted here:

We noted several things about the current formatting.

  1. The report id never changes. The intention was that this uniquely identified the file. But it is currently fixed.
  2. Forecasts PGV and Mag only
  3. Reports both algorithm and alert time, the delta seems to vary from 2 to 5 seconds. This may represent the processing time of algorithm.
  4. Multiple forecasts per trigger file
  5. Multiple events per trigger file
  6. Seems to report duplicate forecast information for each trigger. One copy of the trigger also contains observational data for the trigger. It is unclear how the trigger is match to the event id. We noted that in some cases the event id reported in the trigger file is not the final ANSS event id.
  7. The event id does not have a network id so we assume it is CI network event

VS Trigger Reports

VS is running on southern California Events. We noted the following.

  1. Reports one event per trigger file
  2. Each trigger file reports source of report
  3. Each trigger file contains a set of time series
  4. It seems that all time series in the trigger report have the same timestamps
  5. Event oriented time series include:
    1. magnitude
    2. lat
    3. lon
    4. origin time
  6. Region-oriented time series include
    1. pgv
    2. pga

ShakeMap Data

We are currently working with the ShakeMap stationlist txt file. We may convert to the stationlist xml files, but it was easier to parse the column oriented to start.

We use the ShakeMap data as reference observational data for station amplitude values.

We have identified several characteristics to the ShakeMap observational reports.

  1. Two main formats are the CI: and NN: events, and ECX events. ECX events have their pgv and pga values in different columns than the others, so this type of ShakeMap report must be checked for while parsing.
  2. ShakeMap files also have different number of columns populated. Currently we require that both PGA and PGV are listed before we use it as a authorized data observation. The goal is to maximize the number of observations we have to compare to forecasts. Right now, we don't have forecasts at PGV or PGA only sites. Once we do, we should add small modification to track observations even if they only have one component of data.
  3. We pull event ID from the header of the Shakemap file. However, it does not list the network, which we need to match it up in the ANSS catalog.

Migration to ShakeAlert Testing

Current capabilities are retrospective. They can be downloaded and used to process Trigger files and produce retrospective reports. The intention is that the current tools will be useful in prospective testing of ShakeAlert. Current trigger exchange format may be changed to use a version of ShakeMap format which would be more standardized.

Processing Flow

Current design is based on processing one or more trigger reports. Currently two types of trigger reports are supported CIT On Site, and ETH Virtual Seismologist.

  1. Init CISN Station list, ANSS EQ catalog, ShakeMap catalog
  2. For each Trigger File
    1. Mapped to One or More ANSS Events and observed data from ShakeMap
  3. For each event in trigger file
    1. write event summary
    2. Add to catalog summaries
  4. write catalog summaries

ANSS Catalog Files

We create a catalog of ANSS events based on the ANSS web-based catalog search capabilities. The query parameters for the ANSS catalog are in the header of the file. This catalog represents the authorized reference data for events in the testing region. While we use ShakeMap data files to get station information, we use ANSS catalog as the reference event catalog.

  • Example ANSS Catalog File

Central California Event

During a recent central california event several issues emerged. First the shakemap results that arrived via the shakereader seemed to have only two rows of stations. Then, via the ShakeMap web site, we were able to find a complete ShakeMap file. Next, the event arrived by email but was not available in the ANSS catalog. We had to manually enter the event information from the email into an anss catalog entry for use in processing. Then we saw that all the stations in the ShakeMap observation file were NorCal stations, however the forecast stations in the VS trigger were all southern California stations. This looks to be a limitation of using ShakeMap files for source of observational data

ANSS Catalog Format

Your search parameters are:

   catalog=ANSS
   start_time=2011/01/01,00:00:00
   end_time=2011/06/20,00:00:00
   minimum_latitude=31.00
   maximum_latitude=42.00
   minimum_longitude=-125.00
   maximum_longitude=-112.00
   minimum_magnitude=4.0
   maximum_magnitude=10
   event_type=E 

Date Time Lat Lon Depth Mag Magt Nst Gap Clo RMS SRC Event ID


2011/01/03 12:06:36.88  38.2473 -112.3398   5.40  4.55   ML   25  46   27 0.23  UU              
2011/01/08 00:10:16.74  37.2872 -121.6582  10.07  4.10   Mw  223  55    6 0.18  NC      71506865
2011/01/11 01:36:19.82  37.1658 -117.3770  12.41  4.06   ML   16 161   18 0.12  NN        324385
2011/01/12 08:51:03.68  36.7712 -121.4963   8.04  4.50   Mw  105  35    2 0.19  NC      71508850
2011/01/22 17:26:09.19  38.5496 -118.3800  13.94  4.01   ML   16 229   15 0.11  NN        325137
2011/01/22 17:29:25.70  38.5387 -118.3798   8.35  4.32   ML   18 188   16 0.12  NN        325135
2011/01/22 19:21:04.62  38.5443 -118.3781  14.20  4.15   ML   16 227   16 0.10  NN        325134
2011/01/23 05:29:59.80  31.2170 -115.7000  16.00  4.50   ML  155          0.00  ECX 201101231009
2011/01/23 05:55:30.20  31.0600 -115.6900  17.40  4.00   Mc   29          0.00  ECX 201101231011
2011/01/23 06:47:24.55  32.2165 -115.3213   1.87  4.15   ML   15          0.20  CI      14920876
2011/01/26 00:11:28.00  31.2360 -115.6530   5.00  4.50   Mb   55          0.00  ECX 201101261001
2011/02/13 23:25:09.54  32.1707 -115.2835  13.46  4.08   ML   15          0.19  CI      10892285
2011/02/18 17:47:35.77  32.0470 -115.0622  15.00  5.07   Mw   12          0.26  CI      14937372
2011/02/20 19:34:39.98  32.0468 -115.2518  10.00  4.50   Mw    9          0.18  CI      14937988
2011/02/23 04:49:41.51  39.4993 -122.9488  12.75  4.29   Mw   60  48    2 0.13  NC      71527390
2011/02/23 04:50:14.36  39.4938 -122.9565  11.26  4.21   Mw   41  69    2 0.10  NC      71048354
2011/03/01 02:19:47.01  38.8153 -122.8200   2.95  4.46   Mw   76  24    1 0.08  NC      71530230
2011/03/31 20:22:55.70  32.3392 -115.2282  10.69  4.35   ML   15          0.29  CI      14960652
2011/04/07 13:40:57.80  31.2350 -115.6170   5.00  4.10   Mb  134          0.00  ECX 201104071036
2011/04/09 08:58:49.22  32.6055 -115.7237   9.38  4.13   ML   43          0.36  CI      10922333
2011/04/11 13:00:12.60  31.2170 -115.5920   5.00  4.10   ML   15          0.00  ECX 201104111065
2011/04/13 22:10:08.34  38.3730 -118.7450  16.00  4.30   Mw  128          0.00  REN 201104131069
2011/04/13 22:16:08.71  38.3860 -118.7420  15.40  4.10   Mw   26          0.00  REN 201104131071
2011/04/17 00:45:37.00  38.3940 -118.7340  15.40  4.60   ML   94          0.00  REN 201104171002
2011/04/27 19:19:20.58  38.4140 -118.7180   2.60  5.50   Mb   37          0.00  REN 201104271026
2011/05/03 21:41:49.82  36.3953 -117.8703   2.24  4.45   ML   44          0.18  CI      14978940
2011/06/09 15:22:12.37  32.6630 -115.7365  15.08  4.42   ML   64          0.44  CI      14998780
2011/06/15 12:29:07.64  31.1920 -115.6160  10.00  4.20   ML   40          1.22  NEI 201106151008


CTC Software Versions

Tag Descriptions:

  • 11_05_24
    • This is the first tagged version with skill summary reports considered accurate. Summaries produced with versions prior to this have units problems when converting from PGA to acceleration (cm/s/s).

Tag Descriptions:

  • 11.5.0
    • This distribution is capable of processing performance summaries for new events. Retrieve this distribution and setup the authdata inputs correctly, put the vs trigger file or onsite trigger file in the inputs directory, and run gen_ctc.py and the software should produce a number of performance summaries without errors. This is the distribution described in the SCEC Year 2 report to USGS.


Software Download

CTC software is intended to be general purpose and will be posted online as open-source. SCEC software version is date-based, something like 11_05_24

If you have a SCEC account you can retrieve the project files with the following commands.

CTC Software is posted in repository on SCEC SVN repository

Latest development version is available with the command:

Contributors

Related Entries

See Also

CISN EEW Testing Data Sets

CISN EEW Trigger Summary

CISN EEW Station Lists

CISN provided the following list of stations used in current CISN EEW processing

The ShakeMap authorized data source may provide the observational data used to validate short term ground motion forecasts.

ShakeAlert CTC Trac System

EEW-Related SCEC Research Documents and Links

2010

2009

2008

2006

Fig 1: California Station List with M8 Epicenter.


Fig 2: California Testing Region (inner) and Catalog Region (outer).
Fig 3: Plotting of ShakeMap observation data for Event in Southern California.
Fig 4: Plotting of ShakeMap observation data for Event in Southern California.
Fig 5: Travel time curves show region for which no warnings are possible because of travel time and processing delays.
Fig 6: ShakeMap color scale shows instrumental intensity estimates or observations.
Fig 6: Scheme for comparing PGV and PGA values.
Fig 6: Prototype ShakeAlert Display (Image Credit: Kalpesh Solanki)
Fig 7: CA EQ Probabilities by Magnitude
Fig 8: CISN EQ Example of Warning Times.


Catalog Comparison

We looked at current eew results on the CISN algorithm testing site CISN EEW Algorithm Testing Site]. The web system provides several summaries. We compared the earthquakes on two summaries available on the web site which are: (1) Catalog summary and (2) Cumulative Trigger Summary. EEW system list of ANSS events against ANSS list of triggers.

To interpret these trigger summaries were designed to compare eew performance anywhere in california. The ANSS earthquake catalog is the reference observational data.

The EEW algorithm testing system filters the catalog to include only earthquakes within California region. We based the EEW California region on the CSEP region. There are two regions defined. California testing region. In this region, the systems are expected to forecast earthquakes, or provide EEW alerts. There is a slightly larger region which is the catalog region. When we retrieve the catalog and we restrict our request to a region, we may miss earthquakes right on the edge due to uncertainties in the earthquake locations. So we use a reference earthquake data set from a slightly larger region, the catalog region.

These cumulative summaries work on the site, but they are so large now, with multple years of info that it produces multi-MB htlm files. We don't have good parsing of these, but we worked with data from these two summaries retrieved on 12 Oct 2010.

We found that he catalog list has two events not included in the trigger list. I believe these in the region between the testing region and the catalog region. The two catalog events are listed:

NC	71330090	1/4/2010	14:24:54	4.12	40.2773	-123.868	30.74
CI	14858988	10/12/2010	3:00:34	4.27	32.1582	-115.29	5.58

We need to confirm this one is plausible (32.1582,-115.29) but this is in Humbolt (40.2773,-123.868)so it is not out of the teting region. This discrpency between ANSS events in the two lists remains.

The total events in the trigger list is 146 (1Jan2010 through 12Oct2010)

CTC Abstract for SCEC Annual Meeting 2010

Testing Regional-scale Ground Motion Estimates for the CISN ShakeAlert Project Philip Maechling, Maria Liukis, Thomas H. Jordan

Our SCEC research team has developed initial capabilities of the CISN Testing Center (CTC) as a part of the USGS-funded CISN ShakeAlert Project. The CTC is designed to compare regional-scale ground motion estimates, such as peak ground motion estimates, against observed ground motion data, for significant California Earthquakes. The CTC is designed to identify which ShakeAlert ground motion estimation methods produce the best fit to observed ground motions.

Scientific goals of the CTC include (a) development of consensus among SCEC researchers on meaningful measures of comparison between ground motion estimates and observations, (b) collection and analysis of ShakeAlert performance information, and (c) integration of standard Probabilistic Seismic Hazard Analysis (PSHA) terms and practices into EEW. The technical goal of the CTC is to develop an automated system that can comparing ground motion estimates against ground motion observations. The CTC system used the CSEP testing framework to automate the evaluation of CISN ShakeAlert rapid earthquake parameters including ground motion forecasts for each event.

As part of this year’s CTC development, we updated the CSEP Testing Framework to use the USGS ShakeMap RSS Feed as an authorized source of event-based ground motion observations. As ShakeAlert systems produces rapid ground motion estimates for significant California events, the CTC will evaluate those ShakeAlert ground motion estimates against CISN ground motion observation using goodness-of-fit algorithms defined by CISN scientific groups.

Our ShakeAlert ground motion forecast testing shares scientific and technical goals with several other SCEC activities including validation of attenuation relationships, synthetic ShakeMaps, velocity model and wave propagation model goodness of fit measures, and short term ground motion forecast testing. The CTC system is designed to support CISN ShakeAlert testing and to also be applicable for evaluation of other ground motion estimation methods.

Fig 7: JMA diagram showing EEW concepts and types of EEW system implementations.
Fig 8: JMA diagram showing potential users of EEW ground motion forecasts.
Fig 9: San Francisco Chronicle graphic showing EEW concepts in California.
Fig 10: Japanese Public EEW Awareness Information.
Fig 11: Current and proposed station locations for CISN.

Additional ShakeAlert Information

ShakeAlert CTC Software Stack Description

The following packages contribute to the software stack used by EEW software:

  • awk
  • sed
  • java
  • python and it's external libraries: matplotlib, basemap, numpy, scipy, xml

See Also