Difference between revisions of "Broadband CyberShake Validation"
(69 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
This page follows on [[CyberShake BBP Verification]], moving from 1D comparisons to 3D CyberShake comparisons with both the BBP and observations. | This page follows on [[CyberShake BBP Verification]], moving from 1D comparisons to 3D CyberShake comparisons with both the BBP and observations. | ||
+ | |||
+ | == Process == | ||
+ | |||
+ | These are the steps involved in comparing CyberShake and BBP results to observations for a verification event. | ||
+ | |||
+ | === BBP === | ||
+ | |||
+ | #Create a working directory. | ||
+ | #Create an src_files directory inside it. | ||
+ | #Copy the src file from the BBP validation directory into the src_files directory. | ||
+ | #Copy the station list into the working directory. | ||
+ | #Run run_bbp.py with the '--expert -g' arguments to create an XML file description of the run. Instead of using the defaults for the validation event, point to the station list file in the working directory and the src file in the src_files directory. Include the GP GOF, but not FAS. | ||
+ | #Copy the created XML file into the working directory. | ||
+ | #Run helper_scripts/create_srcs.sh, which will create 64 random seeds and then create 64 realizations by copying over the initial src file and changing the random seed | ||
+ | #Create 64 directories, one for each source. | ||
+ | #Run helper_scripts/prepare_lsf_files.sh script. This will copy both the XML file and the run_bbp_src.lsf script into each realization directory, and make the needed changes to each for that realization. | ||
+ | #Submit all 64 src_*/run_bbp_src*.lsf files. | ||
+ | #Once all 64 run successfully, make a purple_plot/Sims directory, and logs, indata, tmpdata, and outdata directories inside it. | ||
+ | #Edit stage_purple_plot.sh to have the right sim_id. | ||
+ | #Run stage_purple_plot.sh . | ||
+ | #Run utils/batch/combine_gof_gen.py -d purple_plot -o . -c gp | ||
+ | #Examine the .png file produced to make sure it looks OK. | ||
+ | |||
+ | === CyberShake === | ||
+ | |||
+ | ==== Set up and run BBP validation event ==== | ||
+ | |||
+ | <ol> | ||
+ | <b>Setup</b> | ||
+ | <li>Create working directory on local system.</li> | ||
+ | <li>Copy in the following scripts: | ||
+ | *insert_sites_from_list.py | ||
+ | *pop_seeds.py | ||
+ | *update_dist.py | ||
+ | *pop_rvs.py | ||
+ | *pop_site_ruptures.py</li> | ||
+ | |||
+ | <b><br/>Create the sites.</b> | ||
+ | <li>Copy over *.stl list from BBP validation event.</li> | ||
+ | <li>Copy stl list into a bbp_to_cs version of the stl list.</li> | ||
+ | <li>Run the script ./check_for_duplicate_stations.py to see if any stations used by this event are already in the CyberShake site list.</li> | ||
+ | <li>For the stations not in the CyberShake site list, determine their full name from https://www.strongmotioncenter.org/wserv/stations/builder/ . If no full name exists, look at Google maps and make one up. Add the abbreviation to the bbp_to_cs file, and add a full entry (lon, lat, short name, long name, 3) for each new site into a sites_to_insert.txt file.</li> | ||
+ | <li>Check each new site short name to make sure one doesn’t already exist in the DB.</li> | ||
+ | <li>Run insert_sites_from_list.py on the sites_to_insert.txt file.</li> | ||
+ | |||
+ | <b><br/>Set up the rupture variations remotely.</b> | ||
+ | <li>On the remote system, create a directory inside the Ruptures_erf60 directory corresponding to the source_id of the run.</li> | ||
+ | <li>Inside the source_id directory, create one directory for each realization, which will each be its own rupture.</li> | ||
+ | <li>Copy the realizations from the BBP tmpdirs in which the realizations were created, into the corresponding source directory.</li> | ||
+ | <li>Create a symlink to each SRF with the name <src_id>_<rup_id>_event<rup_id>.srf .</li> | ||
+ | <li>Create rupture geometries, <src_id>_<rup_id>.txt, for each file. Note that each of these rupture geometries will be slightly different due to the subfaults shifting. To create the rupture geometry files, run RuptureCodes/utils/extract_rup_geom on each rupture.</li> | ||
+ | <li>Determine the distances from each station to each rupture variation by running calc_distance.py for each variation on a site file in lon, lat, short name format.</li> | ||
+ | <li>Copy the rupture geometry files and the distance files back locally.</li> | ||
+ | <li>Copy the random_seeds.txt file from the BBP validation dir/src_files locally.</li> | ||
+ | |||
+ | <b><br/>Insert rupture variations into database.</b> | ||
+ | <li>Run pop_rvs.py, pointing to the rupture geometry files, to insert the rupture variations into the DB.</li> | ||
+ | <li>Run pop_site_ruptures.py, pointing to a site list file, the distance files, and rupture geometry files.</li> | ||
+ | <li>Run pop_seeds.py on the random_seeds.txt file to populate the seeds into the DB.</li> | ||
+ | <li>Create a new RLS file on shock with the LFNs for both the rupture geometries and the SRFs.</li> | ||
+ | <li>Run pegasus-rc-client to insert the LFNs into the DB.</li> | ||
+ | </ol> | ||
+ | |||
+ | ==== Perform GoF analysis on results ==== | ||
+ | |||
+ | <ol> | ||
+ | <li>Create a working directory.</li> | ||
+ | <li>Copy the src_files directory from the Broadband validation run directory.</li> | ||
+ | <li>Copy in the *.stl file from the BBP for this validation event.</li> | ||
+ | <li>Create site_list.txt, with one CyberShake site per line. | ||
+ | <li>From another CyberShake validation directory, copy in the following scripts: | ||
+ | <ul> | ||
+ | <li>import.lsf</li> | ||
+ | <li>prep.lsf</li> | ||
+ | <li>prep_stat.sh</li> | ||
+ | <li>rename.py</li> | ||
+ | <li>run_gof.sh</li> | ||
+ | <li>stage_purple_plot.sh</li> | ||
+ | <li>run_cs_src0.lsf</li> | ||
+ | </ul></li> | ||
+ | <li>Create a .tgz file on shock with the seismograms for all the stations and all the events.</li> | ||
+ | <li>Unzip this .tgz file into the working directory.</li> | ||
+ | <li>Submit prep.lsf. This will copy the seismograms from the station directories into the src directories, and then process them into BBP-friendly format.</li> | ||
+ | <li>Edit rename.py to point to the BB-to-CS STL file, and then run it to rename the seismograms to include the full BBP site names.</li> | ||
+ | <li>Edit import.lsf to use the correct Run ID prefix, src_id, directory name, and BBP STL file, then submit it. This script will import the seismograms into BBP working directories, so that the BBP can run GoF on them.</li> | ||
+ | <li>Run the BBP to create an XML file which includes running the GoF. To do this, run <pre>run_bbp.py --expert -g</pre> When prompted, select | ||
+ | <ul><li>'validation simulation'</li> | ||
+ | <li>the validation simulation created in the first part from the list</li> | ||
+ | <li>'GP'</li> | ||
+ | <li>yes custom source</li> | ||
+ | <li>point to BBP-provided source installed in src_files directory</li> | ||
+ | <li>yes rupture generator</li> | ||
+ | <li>custom list</li> | ||
+ | <li>point to list in the working directory</li> | ||
+ | <li>yes site response</li> | ||
+ | <li>yes velocity seismogram plots</li> | ||
+ | <li>yes acceleration seismogram plots</li> | ||
+ | <li>no GMPE comparisons</li> | ||
+ | <li>yes goodness-of-fit</li> | ||
+ | <li>'GP'</li> | ||
+ | <li>no additional metrics</li></ul> | ||
+ | Copy the created XML file into the working directory. | ||
+ | </li> | ||
+ | <li>Edit run_cs_src0.lsf and set SIM_ID_PREFIX, DIR_NAME, and XML_PREFIX to the correct values for this event.</li> | ||
+ | <li>Edit run_gof.sh to set DIR_NAME and XML_PREFIX to the same values, and modify the paths to the *.src and *.srf files used in the sed commands.</li> | ||
+ | <li>Run run_gof.sh, which will submit a job to calculate GoF for each of the 64 realizations.</li> | ||
+ | <li>Create the directory purple_plot/Sims. Inside there, create 4 directories: logs, indata, tmpdata, and outdata.</li> | ||
+ | <li>Edit stage_purple_plot.sh to use the correct sim id.</li> | ||
+ | <li>Run stage_purple_plot.sh.</li> | ||
+ | <li>Create the directory purple_plot_output.</li> | ||
+ | <li>Run bbp/utils/batch/combine_gof_gen.py -d purple_plot -o purple_plot_output -c gp to produce a purple plot, combining the GoF results from all stations and events.</li> | ||
+ | <li>Run bbp/utils/batch/compare_bbp_runs.py -d purple_plot -o purple_plot_output/ -c gp to generate the bias values.</li> | ||
+ | </ol> | ||
== Northridge == | == Northridge == | ||
Line 163: | Line 276: | ||
| [[File:CyberShake-GoF-NR-4_28_real47.png|thumb|400px]] | | [[File:CyberShake-GoF-NR-4_28_real47.png|thumb|400px]] | ||
| [[File:CyberShake-GoF-NR-4_28_real25.png|thumb|400px]] | | [[File:CyberShake-GoF-NR-4_28_real25.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== BBP, FAS ==== | ||
+ | |||
+ | {| | ||
+ | | [[File:CyberShake-GoF-NR-BBP_FAS.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== CyberShake v4_7, FAS ==== | ||
+ | |||
+ | {| | ||
+ | | [[File:CyberShake-GoF-NR-4_7_FAS.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === Observational Comparisons, BBP v22.4 === | ||
+ | |||
+ | We repeated the comparisons, using BBP v22.4 for both the 1D BBP and 3D CyberShake calculations. | ||
+ | |||
+ | ==== BBP ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best BBP realization (#6) !! GoF, worst BBP realization (#51) !! GoF, best CS realization (#40) !! GoF, worst CS realization (#51) | ||
+ | |- | ||
+ | | [[File:BBP-v22.4-GoF-NR-combined.png|thumb|400px]] | ||
+ | | [[File:BBP-v22.4-GoF-NR-real6.png|thumb|400px]] | ||
+ | | [[File:BBP-v22.4-GoF-NR-real51.png|thumb|400px]] | ||
+ | | [[File:BBP-v22.4-GoF-NR-real40.png|thumb|400px]] | ||
+ | | [[File:BBP-v22.4-GoF-NR-real51.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== CyberShake ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#40) !! GoF, worst CS realization (#51) !! GoF, best BBP realization (#6) !! GoF, worst BBP realization (#51) | ||
+ | |- | ||
+ | | [[File:CyberShake-GoF-NR-9_1_combined.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-9_1_real40.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-9_1_real51.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-9_1_real6.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-9_1_real51.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake, updated velocity model === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#33) !! GoF, worst CS realization (#51) | ||
+ | |- | ||
+ | | [[File:CyberShake-GoF-NR-11_28_combined.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-11_28_real33.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-11_28_real51.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === Study 22.12 configuration === | ||
+ | |||
+ | These are the validation results using the Study 22.12 configuration, with risetime_coef=2.3 and hb_high_v6.1.1. | ||
+ | |||
+ | ==== BBP (12/9/22) ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#40) !! GoF, worst CS realization (#9) !! GoF, best BBP realization (#6) !! GoF, worst BBP realization (#51) | ||
+ | |- | ||
+ | | [[File:CyberShake-GoF-NR-12_9_combined.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_9_real40.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_9_real9.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_9_real6.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_9_real51.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== CyberShake (12/11/22) ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#40) !! GoF, worst CS realization (#9) !! GoF, best BBP realization (#6) !! GoF, worst BBP realization (#51) | ||
+ | |- | ||
+ | | [[File:CyberShake-GoF-NR-12_11_combined.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_11_real40.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_11_real9.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_11_real6.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-NR-12_11_real51.png|thumb|400px]] | ||
|} | |} | ||
Line 171: | Line 362: | ||
=== Sites === | === Sites === | ||
− | The BBP validation event for Chino Hills has 40 stations. A KML file with the stations is available [[File:chino_hills_stations.kml|here]]. | + | The BBP validation event for Chino Hills has 40 stations. A KML file with the stations is available [[:File:chino_hills_stations.kml|here]]. |
+ | |||
+ | === BBP === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization (#24) !! GoF, worst realization (#35) !! GoF, worst CS realization (#34) | ||
+ | |- | ||
+ | | [[File:BBP-GoF-CH-combined.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-CH-real24.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-CH-real35.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-CH-real34.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | {| Overall FAS | ||
+ | | [[File:BBP-GoF-CH-FAS-combined.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization (#24) !! GoF, worst realization (#34) !! GoF, worst BBP realization (#35) | ||
+ | |- | ||
+ | | [[File:CS-GoF-CH-combined.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-CH-real24.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-CH-real34.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-CH-real35.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | {| Overall FAS | ||
+ | | [[File:CS-GoF-CH-FAS-combined.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake, BBP Vs30 values === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization (#24) !! GoF, worst realization (#34) !! GoF, worst BBP realization (#35) | ||
+ | |- | ||
+ | | [[File:CS-BBPvs30-GoF-CH-combined.png|thumb|400px]] | ||
+ | | [[File:CS-BBPvs30-GoF-CH-real24.png|thumb|400px]] | ||
+ | | [[File:CS-BBPvs30-GoF-CH-real34.png|thumb|400px]] | ||
+ | | [[File:CS-BBPvs30-GoF-CH-real35.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | {| Overall FAS | ||
+ | | [[File:CS-BBPvs30-GoF-CH-FAS-combined.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake, BBP v22.4 === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization (#24) !! GoF, worst realization (#34) !! GoF, best BBP realization (#24) !! GoF, worst BBP realization (#35) | ||
+ | |- | ||
+ | | [[File:CS-9_2-GoF-CH-combined.png|thumb|400px]] | ||
+ | | [[File:CS-9_2-GoF-CH-real24.png|thumb|400px]] | ||
+ | | [[File:CS-9_2-GoF-CH-real34.png|thumb|400px]] | ||
+ | | [[File:CS-9_2-GoF-CH-real24.png|thumb|400px]] | ||
+ | | [[File:CS-9_2-GoF-CH-real35.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake, BBP v22.4, BBP Vs30 === | ||
+ | |||
+ | == Whittier == | ||
+ | |||
+ | Our third event is Whittier. | ||
+ | |||
+ | === Sites === | ||
+ | |||
+ | The BBP validation event for Whittier has 39 stations, one of which is a duplicate. A KML file with the 38 stations used in these tests is available [[:File:whittier_stations.kml|here]]. | ||
+ | |||
+ | === BBP === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization (#15) !! GoF, worst realization (#7) !! GoF, best CS realization (#62) !! GoF, worst CS realization (#3) | ||
+ | |- | ||
+ | | [[File:BBP-GoF-WH-combined.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-WH-2022052315.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-WH-2022052307.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-WH-2022052362.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-WH-2022052303.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | {| Overall FAS | ||
+ | | [[File:BBP-GoF-WH-FAS-combined.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake === | ||
+ | |||
+ | These results are incomplete, without 2 of the stations (VER and PMN). | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization (#62) !! GoF, worst realization (#3) !! GoF, best BBP realization (#15) !! GoF, worst BBP realization (#7) | ||
+ | |- | ||
+ | | [[File:CS-GoF-WH-combined.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-WH-2022061362.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-WH-2022061303.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-WH-2022061315.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-WH-2022061307.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | {| Overall FAS | ||
+ | | [[File:CS-GoF-WH-FAS-combined.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake, BBP Vs30 values === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization !! GoF, worst realization !! GoF, best BBP realization !! GoF, worst BBP realization | ||
+ | |- | ||
+ | | [[File:CS-BBPvs30-GoF-WH-combined.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | {| Overall FAS | ||
+ | | [[File:CS-BBPvs30-GoF-WH-FAS-combined.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake, BBP v22.4 === | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best realization (#62) !! GoF, worst realization (#39) !! GoF, best BBP realization (#15) !! GoF, worst BBP realization (#7) | ||
+ | |- | ||
+ | | [[File:CS-9_4-GoF-WH-combined.png|thumb|400px]] | ||
+ | | [[File:CS-9_4-GoF-WH-real62.png|thumb|400px]] | ||
+ | | [[File:CS-9_4-GoF-WH-real39.png|thumb|400px]] | ||
+ | | [[File:CS-9_4-GoF-WH-real15.png|thumb|400px]] | ||
+ | | [[File:CS-9_4-GoF-WH-real7.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === CyberShake, BBP v22.4, BBP Vs30 values === | ||
+ | |||
+ | == Landers == | ||
+ | |||
+ | === Single fault realization === | ||
+ | |||
+ | ==== CS (2/15/22) ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#6) !! GoF, best BBP realization (#13) | ||
+ | |- | ||
+ | | [[File:CS-GoF-Landers-2_15_combined.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-Landers-2_15_real6.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-Landers-2_15_real13.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== BBP (12/10/22) ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#9) !! GoF, best BBP realization (#13) | ||
+ | |- | ||
+ | | [[File:BBP-GoF-Landers-12_10_combined.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-Landers-12_10_real9.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-Landers-12_10_real13.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== CyberShake (Study 22.12 configuration, 12/12/22) ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#9) !! GoF, worst CS realization (#22) !! GoF, best BBP realization (#13) !! GoF, worst BBP realization (#22) | ||
+ | |- | ||
+ | | [[File:CS-GoF-Landers-12_12_combined.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-Landers-12_12_real9.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-Landers-12_12_real22.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-Landers-12_12_real13.png|thumb|400px]] | ||
+ | | [[File:CS-GoF-Landers-12_12_real22.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== GMPE ==== | ||
+ | |||
+ | {| | ||
+ | | [[File:GMPE-GoF-Landers-single.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === Multi fault realization === | ||
+ | |||
+ | ==== BBP ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#54) !! GoF, worst CS realization (#11) !! GoF, best BBP realization (#55) !! GoF, worst BBP realization (#11) | ||
+ | |- | ||
+ | | [[File:BBP-GoF-Landers-9_22_combined.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-Landers-9_22_real54.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-Landers-9_22_real11.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-Landers-9_22_real55.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-Landers-9_22_real11.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== CyberShake ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#54) !! GoF, worst CS realization (#11) !! GoF, best BBP realization (#55) !! GoF, worst BBP realization (#11) | ||
+ | |- | ||
+ | | [[File:CyberShake-GoF-Landers-8_25_combined.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-Landers-8_25_real54.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-Landers-8_25_real11.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-Landers-8_25_real55.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-Landers-8_25_real11.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== GMPE ==== | ||
+ | |||
+ | Fabio notes that the GMPE codes we have on the BBP are not set up for multi-segment events, and probably only use the first segment in the rupture. | ||
+ | |||
+ | {| | ||
+ | | [[File:GMPE-GoF-Landers-multi.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | == Ridgecrest == | ||
+ | |||
+ | We looked at both the smaller Ridgecrest (7/4/19, M6.4) 'A' event and the larger (7/6/19, M7.1) multi-fault 'C' event. | ||
+ | |||
+ | === Ridgecrest A === | ||
+ | |||
+ | ==== BBP (6/19/23) ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#6) !! GoF, worst CS realization (#16) !! GoF, best BBP realization (#27) !! GoF, worst BBP realization (#16) | ||
+ | |- | ||
+ | | [[File:BBP-GoF-RidgecrestA-6_19_combined.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-RidgecrestA-6_19_real06.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-RidgecrestA-6_19_real16.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-RidgecrestA-6_19_real27.png|thumb|400px]] | ||
+ | | [[File:BBP-GoF-RidgecrestA-6_19_real16.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== CyberShake (9/29/23) ==== | ||
+ | |||
+ | {| | ||
+ | ! Overall GoF !! GoF, best CS realization (#6) !! GoF, worst CS realization (#16) !! GoF, best BBP realization (#27) !! GoF, worst BBP realization (#16) | ||
+ | |- | ||
+ | | [[File:CyberShake-GoF-RidgecrestA-9_29_combined.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-RidgecrestA-9_29_real06.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-RidgecrestA-9_29_real16.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-RidgecrestA-9_29_real27.png|thumb|400px]] | ||
+ | | [[File:CyberShake-GoF-RidgecrestA-9_29_real16.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | ==== GMPE ==== | ||
+ | |||
+ | {| | ||
+ | | [[File:GMPE-GoF-RidgecrestA.png|thumb|400px]] | ||
+ | |} | ||
+ | |||
+ | === Ridgecrest C multi-fault === | ||
+ | |||
+ | |||
+ | == North Palm Springs == |
Latest revision as of 18:17, 13 December 2023
This page follows on CyberShake BBP Verification, moving from 1D comparisons to 3D CyberShake comparisons with both the BBP and observations.
Contents
Process
These are the steps involved in comparing CyberShake and BBP results to observations for a verification event.
BBP
- Create a working directory.
- Create an src_files directory inside it.
- Copy the src file from the BBP validation directory into the src_files directory.
- Copy the station list into the working directory.
- Run run_bbp.py with the '--expert -g' arguments to create an XML file description of the run. Instead of using the defaults for the validation event, point to the station list file in the working directory and the src file in the src_files directory. Include the GP GOF, but not FAS.
- Copy the created XML file into the working directory.
- Run helper_scripts/create_srcs.sh, which will create 64 random seeds and then create 64 realizations by copying over the initial src file and changing the random seed
- Create 64 directories, one for each source.
- Run helper_scripts/prepare_lsf_files.sh script. This will copy both the XML file and the run_bbp_src.lsf script into each realization directory, and make the needed changes to each for that realization.
- Submit all 64 src_*/run_bbp_src*.lsf files.
- Once all 64 run successfully, make a purple_plot/Sims directory, and logs, indata, tmpdata, and outdata directories inside it.
- Edit stage_purple_plot.sh to have the right sim_id.
- Run stage_purple_plot.sh .
- Run utils/batch/combine_gof_gen.py -d purple_plot -o . -c gp
- Examine the .png file produced to make sure it looks OK.
CyberShake
Set up and run BBP validation event
-
Setup
- Create working directory on local system.
- Copy in the following scripts:
- insert_sites_from_list.py
- pop_seeds.py
- update_dist.py
- pop_rvs.py
- pop_site_ruptures.py
Create the sites. - Copy over *.stl list from BBP validation event.
- Copy stl list into a bbp_to_cs version of the stl list.
- Run the script ./check_for_duplicate_stations.py to see if any stations used by this event are already in the CyberShake site list.
- For the stations not in the CyberShake site list, determine their full name from https://www.strongmotioncenter.org/wserv/stations/builder/ . If no full name exists, look at Google maps and make one up. Add the abbreviation to the bbp_to_cs file, and add a full entry (lon, lat, short name, long name, 3) for each new site into a sites_to_insert.txt file.
- Check each new site short name to make sure one doesn’t already exist in the DB.
- Run insert_sites_from_list.py on the sites_to_insert.txt file.
- On the remote system, create a directory inside the Ruptures_erf60 directory corresponding to the source_id of the run.
- Inside the source_id directory, create one directory for each realization, which will each be its own rupture.
- Copy the realizations from the BBP tmpdirs in which the realizations were created, into the corresponding source directory.
- Create a symlink to each SRF with the name <src_id>_<rup_id>_event<rup_id>.srf .
- Create rupture geometries, <src_id>_<rup_id>.txt, for each file. Note that each of these rupture geometries will be slightly different due to the subfaults shifting. To create the rupture geometry files, run RuptureCodes/utils/extract_rup_geom on each rupture.
- Determine the distances from each station to each rupture variation by running calc_distance.py for each variation on a site file in lon, lat, short name format.
- Copy the rupture geometry files and the distance files back locally.
- Copy the random_seeds.txt file from the BBP validation dir/src_files locally.
- Run pop_rvs.py, pointing to the rupture geometry files, to insert the rupture variations into the DB.
- Run pop_site_ruptures.py, pointing to a site list file, the distance files, and rupture geometry files.
- Run pop_seeds.py on the random_seeds.txt file to populate the seeds into the DB.
- Create a new RLS file on shock with the LFNs for both the rupture geometries and the SRFs.
- Run pegasus-rc-client to insert the LFNs into the DB.
Set up the rupture variations remotely.
Insert rupture variations into database.
Perform GoF analysis on results
- Create a working directory.
- Copy the src_files directory from the Broadband validation run directory.
- Copy in the *.stl file from the BBP for this validation event.
- Create site_list.txt, with one CyberShake site per line.
- From another CyberShake validation directory, copy in the following scripts:
- import.lsf
- prep.lsf
- prep_stat.sh
- rename.py
- run_gof.sh
- stage_purple_plot.sh
- run_cs_src0.lsf
- Create a .tgz file on shock with the seismograms for all the stations and all the events.
- Unzip this .tgz file into the working directory.
- Submit prep.lsf. This will copy the seismograms from the station directories into the src directories, and then process them into BBP-friendly format.
- Edit rename.py to point to the BB-to-CS STL file, and then run it to rename the seismograms to include the full BBP site names.
- Edit import.lsf to use the correct Run ID prefix, src_id, directory name, and BBP STL file, then submit it. This script will import the seismograms into BBP working directories, so that the BBP can run GoF on them.
- Run the BBP to create an XML file which includes running the GoF. To do this, run
run_bbp.py --expert -g
When prompted, select- 'validation simulation'
- the validation simulation created in the first part from the list
- 'GP'
- yes custom source
- point to BBP-provided source installed in src_files directory
- yes rupture generator
- custom list
- point to list in the working directory
- yes site response
- yes velocity seismogram plots
- yes acceleration seismogram plots
- no GMPE comparisons
- yes goodness-of-fit
- 'GP'
- no additional metrics
Copy the created XML file into the working directory.
- Edit run_cs_src0.lsf and set SIM_ID_PREFIX, DIR_NAME, and XML_PREFIX to the correct values for this event.
- Edit run_gof.sh to set DIR_NAME and XML_PREFIX to the same values, and modify the paths to the *.src and *.srf files used in the sed commands.
- Run run_gof.sh, which will submit a job to calculate GoF for each of the 64 realizations.
- Create the directory purple_plot/Sims. Inside there, create 4 directories: logs, indata, tmpdata, and outdata.
- Edit stage_purple_plot.sh to use the correct sim id.
- Run stage_purple_plot.sh.
- Create the directory purple_plot_output.
- Run bbp/utils/batch/combine_gof_gen.py -d purple_plot -o purple_plot_output -c gp to produce a purple plot, combining the GoF results from all stations and events.
- Run bbp/utils/batch/compare_bbp_runs.py -d purple_plot -o purple_plot_output/ -c gp to generate the bias values.
Northridge
1D BBP comparisons
We calculated 64 realizations for Northridge for these 38 stations.
Some of the differences between the 1D BBP and 3D CyberShake results can be attributed to differences in site response, which is calculated based on the reference velocity ('vref') and the vs30 of the site. A spreadsheet with vref and Vs30 for both the BBP and CyberShake is available File:BBP v CyberShake Northridge vs30.xls
V1 (2/14/22)
Initially, we used vref=500 for both the high-frequency and low-frequency site response. However, this is incorrect; vref for the low-frequency should vary depending on the properties of the 3D velocity mesh.
V2 (2/25/22)
We continued to use vref=500 for the high-frequency site response. For the low-frequency site response, we are now using the same vref we used in Study 15.12:
vref_LF_eff = Vs30 * [ VsD5H / Vs5H ]
Vs30 = 30 / Sum (1/(Vs sampled from [0.5,29.5] in 1 meter increments))
H = grid spacing
Vs5H = travel time averaged Vs, computed from the CVM in 1 meter increments down to a depth of 5*H
VsD5H = discrete travel time averaged Vs computed from 3D velocity mesh used in the SGT calculation over the upper 5 grid points
So, for H=100m Vs5H would be:
Vs500 = 500 / ( Sum ( 1 / Vs sampled from [0.5,499.5] in 1 meter increments ))
And then VsD5H is given by:
VsD500 = 5/{ 0.5/Vs(Z=0m) + 1/Vs(Z=100m) + 1/Vs(Z=200m) + 1/Vs(Z=300m) + 1/Vs(Z=400m) + 0.5/Vs(Z=500m) }
Below are the ts_process plots for a subset of 10 stations, comparing the 3D CyberShake with 1D BBP results.
Site | TS Process plot |
---|---|
SCE | |
SYL | |
LDM | |
PAC | |
PKC | |
SPV | |
WON | |
KAT | |
RO3 | |
ANA |
V3 (3/2/22)
Next, we recalculated the CyberShake results for 10 sites, using the BBP vs30 values for both the low-frequency and high-frequency elements. Note that the vref low-frequency value for CyberShake is still being derived from the velocity model.
Site | TS Process plot |
---|---|
SCE | |
SYL | |
LDM | |
PAC | |
PKC | |
SPV | |
WON | |
KAT | |
RO3 | |
ANA |
Observational Comparisons
We calculated goodness-of-fit results for both Broadband CyberShake and the BBP against observations for Northridge, using the 64 realizations and 38 stations.
TS process plots comparing the 3 results are available here: v1 ts process plots.
Broadband Platform
Overall GoF | GoF, best realization (#7) | GoF, worse realization (#15) |
---|---|---|
CyberShake v4_8
Overall GoF | GoF, best realization (#53) | GoF, worse realization (#54) |
---|---|---|
Broadband Platform, updated frequency bands
We realized that our GoF comparisons were run using hard-coded frequency bands of [0.05, 50] Hz. This was correct when doing CS-to-BBP comparisons, but not correct when using observations. We updated the frequency bands and reran the GoF.
Overall GoF | GoF, best BBP realization (#47) | GoF, worst BBP realization (#25) | GoF, best CS realization (#40) | GoF, worst CS realization (#37) |
---|---|---|---|---|
CyberShake v4_8, updated frequency bands
As above, reran the GoF.
Overall GoF | GoF, best CS realization (#40) | GoF, worst CS realization (#37) | GoF, best BBP realization (#47) | GoF, worst BBP realization (#25) |
---|---|---|---|---|
CyberShake v4_28, BBP Vs30
We recalculated the CyberShake GoF results, using the BBP Vs30 values.
Overall GoF | GoF, best CS realization (#40) | GoF, worst CS realization (#51) | GoF, best BBP realization (#47) | GoF, worst BBP realization (#25) |
---|---|---|---|---|
BBP, FAS
CyberShake v4_7, FAS
Observational Comparisons, BBP v22.4
We repeated the comparisons, using BBP v22.4 for both the 1D BBP and 3D CyberShake calculations.
BBP
Overall GoF | GoF, best BBP realization (#6) | GoF, worst BBP realization (#51) | GoF, best CS realization (#40) | GoF, worst CS realization (#51) |
---|---|---|---|---|
CyberShake
Overall GoF | GoF, best CS realization (#40) | GoF, worst CS realization (#51) | GoF, best BBP realization (#6) | GoF, worst BBP realization (#51) |
---|---|---|---|---|
CyberShake, updated velocity model
Overall GoF | GoF, best CS realization (#33) | GoF, worst CS realization (#51) |
---|---|---|
Study 22.12 configuration
These are the validation results using the Study 22.12 configuration, with risetime_coef=2.3 and hb_high_v6.1.1.
BBP (12/9/22)
Overall GoF | GoF, best CS realization (#40) | GoF, worst CS realization (#9) | GoF, best BBP realization (#6) | GoF, worst BBP realization (#51) |
---|---|---|---|---|
CyberShake (12/11/22)
Overall GoF | GoF, best CS realization (#40) | GoF, worst CS realization (#9) | GoF, best BBP realization (#6) | GoF, worst BBP realization (#51) |
---|---|---|---|---|
Chino Hills
For our second event, we selected Chino Hills.
Sites
The BBP validation event for Chino Hills has 40 stations. A KML file with the stations is available here.
BBP
Overall GoF | GoF, best realization (#24) | GoF, worst realization (#35) | GoF, worst CS realization (#34) |
---|---|---|---|
CyberShake
Overall GoF | GoF, best realization (#24) | GoF, worst realization (#34) | GoF, worst BBP realization (#35) |
---|---|---|---|
CyberShake, BBP Vs30 values
Overall GoF | GoF, best realization (#24) | GoF, worst realization (#34) | GoF, worst BBP realization (#35) |
---|---|---|---|
CyberShake, BBP v22.4
Overall GoF | GoF, best realization (#24) | GoF, worst realization (#34) | GoF, best BBP realization (#24) | GoF, worst BBP realization (#35) |
---|---|---|---|---|
CyberShake, BBP v22.4, BBP Vs30
Whittier
Our third event is Whittier.
Sites
The BBP validation event for Whittier has 39 stations, one of which is a duplicate. A KML file with the 38 stations used in these tests is available here.
BBP
Overall GoF | GoF, best realization (#15) | GoF, worst realization (#7) | GoF, best CS realization (#62) | GoF, worst CS realization (#3) |
---|---|---|---|---|
CyberShake
These results are incomplete, without 2 of the stations (VER and PMN).
Overall GoF | GoF, best realization (#62) | GoF, worst realization (#3) | GoF, best BBP realization (#15) | GoF, worst BBP realization (#7) |
---|---|---|---|---|
CyberShake, BBP Vs30 values
Overall GoF | GoF, best realization | GoF, worst realization | GoF, best BBP realization | GoF, worst BBP realization |
---|---|---|---|---|
CyberShake, BBP v22.4
Overall GoF | GoF, best realization (#62) | GoF, worst realization (#39) | GoF, best BBP realization (#15) | GoF, worst BBP realization (#7) |
---|---|---|---|---|
CyberShake, BBP v22.4, BBP Vs30 values
Landers
Single fault realization
CS (2/15/22)
Overall GoF | GoF, best CS realization (#6) | GoF, best BBP realization (#13) |
---|---|---|
BBP (12/10/22)
Overall GoF | GoF, best CS realization (#9) | GoF, best BBP realization (#13) |
---|---|---|
CyberShake (Study 22.12 configuration, 12/12/22)
Overall GoF | GoF, best CS realization (#9) | GoF, worst CS realization (#22) | GoF, best BBP realization (#13) | GoF, worst BBP realization (#22) |
---|---|---|---|---|
GMPE
Multi fault realization
BBP
Overall GoF | GoF, best CS realization (#54) | GoF, worst CS realization (#11) | GoF, best BBP realization (#55) | GoF, worst BBP realization (#11) |
---|---|---|---|---|
CyberShake
Overall GoF | GoF, best CS realization (#54) | GoF, worst CS realization (#11) | GoF, best BBP realization (#55) | GoF, worst BBP realization (#11) |
---|---|---|---|---|
GMPE
Fabio notes that the GMPE codes we have on the BBP are not set up for multi-segment events, and probably only use the first segment in the rupture.
Ridgecrest
We looked at both the smaller Ridgecrest (7/4/19, M6.4) 'A' event and the larger (7/6/19, M7.1) multi-fault 'C' event.
Ridgecrest A
BBP (6/19/23)
Overall GoF | GoF, best CS realization (#6) | GoF, worst CS realization (#16) | GoF, best BBP realization (#27) | GoF, worst BBP realization (#16) |
---|---|---|---|---|
CyberShake (9/29/23)
Overall GoF | GoF, best CS realization (#6) | GoF, worst CS realization (#16) | GoF, best BBP realization (#27) | GoF, worst BBP realization (#16) |
---|---|---|---|---|