EEW Evaluation - Revision history

Maechlin at 06:32, 21 June 2011

2011-06-21T06:32:07Z

Maechlin at 05:33, 20 June 2011

2011-06-20T05:33:15Z

Maechlin at 05:32, 20 June 2011

2011-06-20T05:32:59Z

Maechlin at 04:14, 18 June 2011

2011-06-18T04:14:56Z

Maechlin at 04:51, 11 June 2011

2011-06-11T04:51:00Z

Maechlin: /* Example Virtual Seismologist Forecast Performance */

2011-05-26T08:44:49Z

Example Virtual Seismologist Forecast Performance

Show changes

Maechlin: /* Example Ground Motion Forecast Summaries */

2011-05-26T08:44:06Z

Example Ground Motion Forecast Summaries

Maechlin: /* CISN Testing Documents */

2011-05-26T08:43:02Z

CISN Testing Documents

Maechlin at 08:40, 26 May 2011

2011-05-26T08:40:49Z

Maechlin at 04:46, 25 May 2011

2011-05-25T04:46:39Z

← Older revision		Revision as of 06:32, 21 June 2011
Line 364:		Line 364:
	== See Also ==		== See Also ==
	*[[Main Page]]		*[[Main Page]]
		+
		+	== CISN EEW Testing Data Sets ==
		+
		+	'''CISN EEW Trigger Summary'''
		+	*[http://scec.usc.edu/research/eew/summary/California/shakecast-tests/ CTC Test Results Directory]
		+	*[http://scecdata.usc.edu/earlywarning/ CISN EEW Algorithm Testing Site]
		+	*[http://scecdata.usc.edu/maechling/Cumulative_Trigger_Summary.zip CISN Cumulative Trigger Summary] (2008-2010) (15MB Zip File containing 75MB html File)
		+
		+	'''CISN EEW Station Lists'''
		+
		+	CISN provided the following list of stations used in current CISN EEW processing
		+
		+	*[http://scecdata.usc.edu/maechling/stations_nc_04.13.2010.txt Northern CISN Station List]
		+	*[http://scecdata.usc.edu/maechling/stations_sc_04.13.2010.txt Southern CISN Station List]
		+	*[http://scecdata.usc.edu/maechling/cisn_eew_10Oct_2010_v1.kml Combined CISN EEW Station List (KML Format)]
		+	*[http://scec.usc.edu/websims/repo/m8/data/trace.kml Southern San Andreas Fault Trace lon/lat (KML format)]
		+
		+	The ShakeMap authorized data source may provide the observational data used to validate short term ground motion forecasts.
		+
		+	*[http://earthquake.usgs.gov/earthquakes/shakemap/rss_info.php ShakeMap Rss System]
		+	*[http://www.ncedc.org/anss/catalog-search.html ANSS Seismic Data Center]
		+	*[http://www.data.scec.org/ SCEC Data Center at Caltech]
		+	*[http://www.ncedc.org/ Northern California Data Center at UC Berkeley]
		+
		+	'''ShakeAlert CTC Trac System'''
		+
		+	*[http://northridge.usc.edu/trac/eew CISN EEW Trac System]
		+
		+	== EEW-Related SCEC Research Documents and Links ==
		+
		+	'''2010'''
		+	*[http://scec.usc.edu/scecwiki/images/c/c8/CTC_14Oct2010_v6.ppt CISN EEW Workshop Oct 2010] (7MB ppt File)
		+	*[http://scecdata.usc.edu/maechling/CTC_Testing_Plans_1April2010.pptx CTC Development Plans April 2010 (12Mb pptx)]
		+	*[[M8_EEW_Analysis]]
		+	*[http://scec.usc.edu/scecwiki/images/3/3f/EEW_Algor_testing_v01b.pdf CISN EEW Testing Document] (Updated July 2010)
		+
		+	'''2009'''
		+	*[http://scec.usc.edu/scecwiki/images/f/ff/EEW_Testing_summary_v2.pdf CISN EEW Testing Summary 2009]
		+	*[http://scecdata.usc.edu/maechling/ShakeAlert_3YrSummary.pdf Fall AGU 2009 (Cua et al)]
		+	**[[File:Cua_ShakeAlert_3yrSummary.pdf]] Alternative Download Site
		+	*[http://scecdata.usc.edu/maechling/ShakeAlert_User_Poster.pdf Fall AGU 2009 (Hellweg et al)]
		+	*[http://scecdata.usc.edu/maechling/Maechling_EEW_AGU_Fall_2009_v4.pdf Fall AGU 2009 (Maechling et al)]
		+	*[http://scecdata.usc.edu/maechling/Maechling_EEW_SCEC_AM_2009_v2.pdf SCEC Annual Meeting Poster 2009 (Maechling et al)]
		+	*[http://scecdata.usc.edu/maechling/mboese_IST_LunchBunch2009.pdf Maren Bose EEW Presentation (2009)]
		+
		+	'''2008'''
		+	*[http://scec.usc.edu/scecwiki/images/9/99/CISN_AGU08F.ppt Fall 2008 AGU EEW Presentation (Hellweg et al)]
		+	*[http://scecdata.usc.edu/maechling/Maechling_CSEP_System_Overview_120508_v3.ppt CSEP Testing Center Overview (Dec 2008) (Maechling et al) (3MB ppt)]
		+	*[http://scecdata.usc.edu/maechling/Maechling_Prospective_Testing_06102008.ppt Prospective Testing Overview (EEW and CSEP) (Maechling et al) (3MB ppt)]
		+	*[http://scecdata.usc.edu/maechling/Maechling_EEW_SCEC_Japan_2009_v1.pdf EEW Meeting In Kyoto Poster 2008 (Maechling et al) (PDF)]
		+
		+	'''2006'''
		+
		+	*[http://scecdata.usc.edu/maechling/K4_Erdik.pdf Mustafa Erdik EEW Presentation (2006)]
		+	*[http://scecdata.usc.edu/maechling/O06_UJNR200611081.pdf Shin'ya TSUKADA EEW Presentation (2006)]
		+
		+	[[File:Eew_stations_cal_m8.png\|340px\|thumb\|right\|Fig 1: California Station List with M8 Epicenter.]]
		+
		+
		+	[[Image:CaliforniaNatLab.png\|340px\|thumb\|right\|Fig 2: California Testing Region (inner) and Catalog Region (outer).]]
		+
		+	[[File:scec.eew.EEWTest.eewTest_SHAKEMAP-Test_Event10736069.png\|340px\|thumb\|right\|Fig 3: Plotting of ShakeMap observation data for Event in Southern California.]]
		+
		+	[[File:scec.eew.EEWTest.eewTest_SHAKEMAP-Test_ZoomEvent10736069.png\|340px\|thumb\|right\|Fig 4: Plotting of ShakeMap observation data for Event in Southern California.]]
		+
		+	[[File:Define_Reference_Events.png\|340px\|thumb\|right\|Fig 5: Travel time curves show region for which no warnings are possible because of travel time and processing delays.]]
		+
		+	[[File:shakemap_color_scale.png\|340px\|thumb\|right\|Fig 6: ShakeMap color scale shows instrumental intensity estimates or observations.]]
		+
		+	[[Image:GroundMotionEvaluation.png\|340px\|thumb\|right\|Fig 6: Scheme for comparing PGV and PGA values.]]
		+
		+	[[Image:EEWDisplay.png\|340px\|thumb\|right\|Fig 6: Prototype ShakeAlert Display (Image Credit: Kalpesh Solanki)]]
		+
		+	[[Image:CA_Eq_Probability_by_Mag.jpg\|340px\|thumb\|right\|Fig 7: CA EQ Probabilities by Magnitude]]
		+
		+	[[Image:CISN_EQ_Example.jpg\|340px\|thumb\|right\|Fig 8: CISN EQ Example of Warning Times.]]
		+
		+
		+	== Catalog Comparison ==
		+	We looked at current eew results on the CISN algorithm testing site [http://scecdata.usc.edu/earlywarning/ CISN EEW Algorithm Testing Site]]. The web system provides several summaries. We compared the earthquakes on two summaries available on the web site which are: (1) Catalog summary and (2) Cumulative Trigger Summary. EEW system list of ANSS events against ANSS list of triggers.
		+
		+	To interpret these trigger summaries were designed to compare eew performance anywhere in california. The ANSS earthquake catalog is the reference observational data.
		+
		+	The EEW algorithm testing system filters the catalog to include only earthquakes within California region. We based the EEW California region on the CSEP region. There are two regions defined. California testing region. In this region, the systems are expected to forecast earthquakes, or provide EEW alerts. There is a slightly larger region which is the catalog region. When we retrieve the catalog and we restrict our request to a region, we may miss earthquakes right on the edge due to uncertainties in the earthquake locations. So we use a reference earthquake data set from a slightly larger region, the catalog region.
		+
		+	These cumulative summaries work on the site, but they are so large now, with multple years of info that it produces multi-MB htlm files. We don't have good parsing of these, but we worked with data from these two summaries retrieved on 12 Oct 2010.
		+
		+	We found that he catalog list has two events not included in the trigger list. I believe these in the region between the testing region and the catalog region. The two catalog events are listed:
		+
		+	NC 71330090 1/4/2010 14:24:54 4.12 40.2773 -123.868 30.74
		+	CI 14858988 10/12/2010 3:00:34 4.27 32.1582 -115.29 5.58
		+
		+	We need to confirm this one is plausible (32.1582,-115.29) but this is in Humbolt (40.2773,-123.868)so it is not out of the teting region. This discrpency between ANSS events in the two lists remains.
		+
		+	The total events in the trigger list is 146 (1Jan2010 through 12Oct2010)
		+
		+	== CTC Abstract for SCEC Annual Meeting 2010 ==
		+
		+	Testing Regional-scale Ground Motion Estimates for the CISN ShakeAlert Project
		+	Philip Maechling, Maria Liukis, Thomas H. Jordan
		+
		+	Our SCEC research team has developed initial capabilities of the CISN Testing Center (CTC) as a part of the USGS-funded CISN ShakeAlert Project. The CTC is designed to compare regional-scale ground motion estimates, such as peak ground motion estimates, against observed ground motion data, for significant California Earthquakes. The CTC is designed to identify which ShakeAlert ground motion estimation methods produce the best fit to observed ground motions.
		+
		+	Scientific goals of the CTC include (a) development of consensus among SCEC researchers on meaningful measures of comparison between ground motion estimates and observations, (b) collection and analysis of ShakeAlert performance information, and (c) integration of standard Probabilistic Seismic Hazard Analysis (PSHA) terms and practices into EEW. The technical goal of the CTC is to develop an automated system that can comparing ground motion estimates against ground motion observations. The CTC system used the CSEP testing framework to automate the evaluation of CISN ShakeAlert rapid earthquake parameters including ground motion forecasts for each event.
		+
		+	As part of this year’s CTC development, we updated the CSEP Testing Framework to use the USGS ShakeMap RSS Feed as an authorized source of event-based ground motion observations. As ShakeAlert systems produces rapid ground motion estimates for significant California events, the CTC will evaluate those ShakeAlert ground motion estimates against CISN ground motion observation using goodness-of-fit algorithms defined by CISN scientific groups.
		+
		+	Our ShakeAlert ground motion forecast testing shares scientific and technical goals with several other SCEC activities including validation of attenuation relationships, synthetic ShakeMaps, velocity model and wave propagation model goodness of fit measures, and short term ground motion forecast testing. The CTC system is designed to support CISN ShakeAlert testing and to also be applicable for evaluation of other ground motion estimation methods.
		+
		+	[[File:eew1.png\|340px\|thumb\|right\|Fig 7: JMA diagram showing EEW concepts and types of EEW system implementations.]]
		+
		+	[[File:eew2.png\|340px\|thumb\|right\|Fig 8: JMA diagram showing potential users of EEW ground motion forecasts.]]
		+
		+	[[File:SanFranciscoChronicleSciSecFig.jpg\|340px\|thumb\|right\|Fig 9: San Francisco Chronicle graphic showing EEW concepts in California.]]
		+
		+	[[File:EEWsuperhero.jpg\|340px\|thumb\|right\|Fig 10: Japanese Public EEW Awareness Information.]]
		+
		+	[[File:CAstations_Optimal.png\|340px\|thumb\|right\|Fig 11: Current and proposed station locations for CISN.]]
		+
		+	== Additional ShakeAlert Information ==
		+	*[http://www.cisn.org/ CISN Home Page]
		+	*[http://www.usgs.gov/newsroom/article.asp?ID=2366&from=rss_home USGS ShakeAlert Press Release]
		+	*[http://www.seismolab.caltech.edu/ Caltech Seismo Lab]
		+	*[http://seismo.berkeley.edu/ U.C. Berkeley Seismo Lab]
		+	*[http://earthquake.usgs.gov/regional/sca/ USGS Pasadena Seismic Network Operations]
		+	*[http://earthquake.usgs.gov/regional/nca/ USGS Menlo Park Seismic Network Operations]
		+	*[http://qcn.stanford.edu/ Quake Catcher Network]
		+
		+	== ShakeAlert CTC Software Stack Description ==
		+
		+	The following packages contribute to the software stack used by EEW software:
		+
		+	*awk
		+	*sed
		+	*java
		+	*python and it's external libraries: matplotlib, basemap, numpy, scipy, xml
		+
		+	== See Also ==
		+
		+	*[http://qcn.stanford.edu/ Quake Catcher Network]

@@ Line 1: / Line 1: @@
 [[Image:SCEC_NSF-USGS-words_logom.png|left|150px]][[Image:CISNlogos.png|right|250px]][[Image:usgs-logo-color.jpg|right|150px]]
-[CISN Testing Center] (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.
+[[CISN Testing Center]] (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.
 CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.

← Older revision		Revision as of 05:32, 20 June 2011
Line 1:		Line 1:
−	CISN Testing Center (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.	+	[[Image:SCEC_NSF-USGS-words_logom.png\|left\|150px]][[Image:CISNlogos.png\|right\|250px]][[Image:usgs-logo-color.jpg\|right\|150px]]
		+
		+	[CISN Testing Center] (CTC) code development project processes CISN EEW trigger reports and produces CISN EEW performance summaries. Current design goal is to produce a system capable of both retrospective testing of CISN reports and usable for prospective testing of ongoing CISN ShakeAlert results.

	CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.		CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.

@@ Line 276: / Line 276: @@
 Next, the event arrived by email but was not available in the ANSS catalog. We had to manually enter the event information from the email into an anss catalog entry for use in processing.
 Then we saw that all the stations in the ShakeMap observation file were NorCal stations, however the forecast stations in the VS trigger were all southern California stations. This looks to be a limitation of using ShakeMap files for source of observational data
@@ Line 282: / Line 330: @@
 Tag Descriptions:
 *11_05_24
-**This is the first tagged version with skill summary reports considered accurate. All versions prior to this a considered to have units problems when converting from PGA to acceleration (cm/s/s).
+**This is the first tagged version with skill summary reports considered accurate. Summaries produced with versions prior to this have units problems when converting from PGA to acceleration (cm/s/s).
 Tag Descriptions:

← Older revision		Revision as of 04:51, 11 June 2011
Line 2:		Line 2:

	CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.		CTC Software is designed to input EEW trigger information and produce performance summaries. It is designed to support EEW triggers in multiple formats and is intended support processing of ShakeAlert EEW trigger files. Currently two trigger types can be processed, Virtual Seismologist lead by Swiss Seismological Service at ETH Zurich, and On Site lead by Caltech seismology lab. Support for UC Berkeley ELarms and CISN ShakeAlert are planned.
		+	== Software Requirements ==
		+	*[http://www.python.org/download/releases/2.6/ Python 2.6]
		+	*[http://code.google.com/p/pysqlite/ Pysqlite Home Page]

	== CISN Testing Documents ==		== CISN Testing Documents ==

@@ Line 14: / Line 14: @@
 == Example Ground Motion Forecast Summaries ==
 <pre>
-Event ID: 10922333
+ANSS Event ID: 10922333
 Magnitude: 4.13
-Origin Time: 2011/04/09 08:58:49.22
+Origin Time: 2011/04/09 08:58:49
 Hypocenter: 32.6055 -115.7237 9.38 km
-Forecast Source: ssi://caltech.gps.edu/onsite/2.1.0
+Forecast Sources: ssi://caltech.gps.edu/onsite/2.1.0
-Forecast Time: 2011-4-9T8:59:21Z
+First Forecast Time: 2011/04/09 08:59:21
 First Forecast Secs after Origin: 32
-Net:Sta  Dist(km) Wd(secs) PGV(obs) PGV(fcst)cm/second  I(obs) I(fcst)MMI   abs(Io-If)  Skill   obs GM    fcst GM
+Net:Sta  Dist(km) Wd(secs) PGV(obs-cm/s) PGV(fcst)  PGA(obs-cm/s2)  PGA(fcst)  I(obs-MMI)     I(fcst)   abs(Io-If)  Skill           obs GM              fcst GM
-   CI:WES    17.3  32.0  0.0204    0.3816             1.95     4.62       2.67   Very Poor     Strong shaking/light damage potential          Weak shaking/no damage potential
+   CI:WES    17.3    32.0     0.2000       0.3816                                     1.93          2.52       0.59        Good         Light shaking/no damage potential          Weak shaking/no damage potential
-   CI:DRE    34.0  35.0  0.0109    0.2022             1.38     4.04       2.66   Very Poor     Strong shaking/light damage potential          Weak shaking/no damage potential
+   CI:DRE    34.0    35.0     0.1071       0.2022                                     1.36          1.94       0.58        Good          Weak shaking/no damage potential          Weak shaking/no damage potential
-   CI:IBP    35.5  35.0  0.0043    0.1042             0.53     3.44       2.91   Very Poor        Moderate shaking/very light damage              Not felt/no damage potential
+   CI:IBP    35.5    35.0     0.0422       0.1042                                     1.00          1.34       0.34   Very Good          Weak shaking/no damage potential              Not felt/no damage potential
-   CI:ERR    57.7  39.0  0.0046    0.1317             0.59     3.65       3.06   Very Poor        Moderate shaking/very light damage              Not felt/no damage potential
+   CI:ERR    57.7    39.0     0.0449       0.1317                                     1.00          1.55       0.55        Good          Weak shaking/no damage potential              Not felt/no damage potential
 Summary of 4 site-specific forecast ratings for event:10922333
-Very Good:   0.0%
+Very Good:  25.0%
-Good     :   0.0%
+Good     :  75.0%
 Moderate :   0.0%
 Poor     :   0.0%
-Very Poor: 100.0%
+Very Poor:   0.0%
 </pre>
 == Example Virtual Seismologist Forecast Performance ==
 <pre>

@@ Line 5: / Line 5: @@
 == CISN Testing Documents ==
 *'''2011'''
-*[http://hypocenter.usc.edu/research/EEW/CTC_Summaries_26May2011.pptx CTC Example Summaries]
+*[http://hypocenter.usc.edu/research/EEW/CTC_Summaries_26May2011.pptx CTC Example Summaries 2Mb pptx file]
 *[http://scec.usc.edu/scecwiki/images/5/5d/CTC_Update_12May2011.ppt Summary of CTC Development 12 May 2011]

@@ Line 10: / Line 10: @@
 *[http://scec.usc.edu/scecwiki/images/c/c8/CTC_14Oct2010_v6.ppt CISN EEW Workshop Oct 2010] (7MB ppt File)
 *[http://scec.usc.edu/scecwiki/images/3/3f/EEW_Algor_testing_v01b.pdf CISN EEW Testing Document] (Updated July 2010)
 == Example Ground Motion Forecast Summaries ==
@@ Line 276: / Line 270: @@
 Next, the event arrived by email but was not available in the ANSS catalog. We had to manually enter the event information from the email into an anss catalog entry for use in processing.
 Then we saw that all the stations in the ShakeMap observation file were NorCal stations, however the forecast stations in the VS trigger were all southern California stations. This looks to be a limitation of using ShakeMap files for source of observational data
 == Software Download ==