SCECpedia - User contributions [en]

CyberShake Study 21.12

2021-12-08T18:37:31Z

Kmilner: /* Production Checklist */

CyberShake 21.12 is a computational study to use a new ERF with CyberShake, generated from an RSQSim catalog. We plan to calculate results for 335 sites in Southern California using the RSQSim ERF, a minimum Vs of 500 m/s, and a frequency of 1 Hz. We will use the CVM-S4.26.M01 model, and the GPU implementation of AWP-ODC-SGT enhanced from the BBP verification testing. We will begin by generating all sets of SGTs, on Summit, then post-process them on a combination of Summit and Frontera.

== Status ==

This study is in the pre-production phase. Production is scheduled to begin in mid-December, 2021.

== Data Products ==

== Science Goals ==

The science goals for this study are:

*Calculate a regional CyberShake model using an alternative, RSQSim-derived ERF.
*Compare results from an RSQSim ERF to results using a UCERF2 ERF (Study 15.4).
*Quantify effects of source model non-ergodicity
*Compare spatial distribution of ground motions (including directivity) to empirical and kinematic models

== Technical Goals ==

The technical goals for this study are:

*Perform a study using OLCF Summit as a key compute resource.
*Evaluate the performance of the new workflow submission host, shock-carc.
*Use Globus Online for staging of output data products.

== ERF ==

The ERF was generated from an RSQSim catalog, with the following parameters:
*715kyr catalog (the first 65k years of events were dropped, so that every fault's first event is excluded)
*220,927 earthquakes with M6.5+
*All events have equal probability, 1/715k

Additional details are available on the [http://opensha.usc.edu/ftp/kmilner/markdown/rsqsim-analysis/catalogs/rundir4983_stitched/#bruce-4983-stitched catalog's metadata page], and the catalog and input fault geometry files can be [https://zenodo.org/record/5542222 downloaded from zeonodo]. This is the catalog used in [https://pubs.geoscienceworld.org/ssa/bssa/article/111/2/898/593757/Toward-Physics-Based-Nonergodic-PSHA-A-Prototype Milner et al., 2021], which used 0.5 Hz CyberShake simulations performed in May, 2020.

== Sites ==

We will run a list of 335 sites, taken from the site list that was used in other Southern California studies. The order of execution will be:

*10 sites used in Milner et al. (2021), each with top mesh point Vs at the 500 m/s floor: USC, SMCA, OSI, WSS, SBSM, LAF, s022, STNI, WNGC, PDE
*PAS hard rock site
*20 km site grid
*10 km site grid
*Remaining POIs, select 5km grid sites also used in Study 15.4

{|
| [[File:Study_21.12_site_map.png|thumb|600px]]
|}

*[[Media:Study_21.12_sites.csv|CSV site list]]
*[[Media:Study_21.12_sites_names.kml|KML site list with names]]
*[[Media:Study_21.12_sites_no_names.kml|KML site list without names]]

== Velocity Model ==

We will use CVM-S4.26.M01.

To better represent the near-surface layer, we will populate the velocity parameters for the surface point by querying the velocity model at a depth of (grid spacing)/4. For this study, the grid spacing is 100m, so we will query UCVM at a depth of 25m and use that value to populate the surface grid point. The rationale is that the media parameters at the surface grid point are supposed to represent the material properties for [0, 50m], and this is better represented by using the value at 25m than the value at 0m.

== Verification Tests ==

=== USC Hazard Curves ===

Hazard curve comparisons for site USC, between:

*ERF 58: 0.5 Hz RSQSim ERF used for Milner et al. (2021) calculations, May 2020
*ERF 61: 0.5 Hz RSQSim production ERF with the same catalog
*ERF 62: 1 Hz RSQSim production ERF with the same catalog

The first test run of ERF 61 used the wrong (older) mesh lower depth of 40 km, which is why the top right plot differs slightly from the top left. The middle left plot agrees perfectly with the top left.

{|
|-
| '''ERF 58, Original'''
| '''ERF 61 w/ wrong depth'''
|-
| [[File:USC_curves_3s_ERF58.png|thumb|500px]]
| [[File:USC_curves_3s_ERF61_PREV.png|thumb|500px]]
|-
| '''ERF 61 w/ corrected depth'''
| '''ERF 62 (1 Hz) with old simulation parameters'''
|-
| [[File:USC_curves_3s_ERF61.png|thumb|500px]]
| [[File:USC_curves_3s_ERF62_FIRST.png|thumb|500px]]
|-
| '''ERF 62 (1 Hz) with proposed (new) simulation parameters'''
| '''ERF 62 ccatter comparing new parameters (y) and old parameters (x)'''
|-
| [[File:USC_curves_3s_ASK2014.png|thumb|500px]]
| [[File:Erf_62prev_62_USC_compare.png|thumb|500px]]
|}

The first 1 Hz test run uses (bottom right above) uses the same simulation paramters as used with ERF 58 and 61, changing only the frequency. Some differences are apparent, which also persist for longer period (5s) curves:

{|
|-
| '''ERF 61, 5s Sa, 0.5 Hz, old simulation parameters'''
| '''ERF 62, 5s Sa, 1 Hz, old simulation parameters'''
|-
| [[File:USC_curves_5s_ERF61.png|thumb|500px]]
| [[File:USC_curves_5s_ERF62_FIRST.png|thumb|500px]]
|}

Here are seismograms and RotD's for the largest amplitude rupture for this 0.5 Hz vs 1 Hz test:

{|
| [[File:USC_s1069_r0_1_v_0.5hz.png|thumb|800px]]
|-
| [[File:USC_s1069_r0_1_v_0.5hz_rotd.png|thumb|800px]]
|}

=== 3s Amplitude Scatter plots ===

These plots show that (left) ERF 61 exactly reproduces ERF 58, and (right) that there are indeed differences going from 0.5 Hz to 1 Hz:

{|
| [[File:Erf_58_61_USC_compare.png|thumb|500px]]
| [[File:Erf_58_62_USC_compare.png|thumb|500px]]
|}

=== 1 Hz vs 0.5 Hz comparisons ===

Here is a comparison of 1 Hz runs (y axis) and 0.5 Hz runs (x axis). The top row is a recent USC run with the RSQSim ERF. The bottom row is a previous test with UCERF2-CyberShake and the WNGC site. The columns are 3s, 5s, and 10s SA, respectively, all geometric mean.

{|
| [[File:USC_7236_vs_7237_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|-
| [[File:WNGC_3842_vs_3837_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|}

== Technical and Scientific Updates ==

Since our last study we have made a number of scientific updates to the platform, many as a result of the BBP verification effort.

*Several bugs were found and fixed in the AWP code.
*We have switched from stress insertion to velocity insertion of the impulse when generating SGTs.
*The sponge zone used in the absorbing boundary condition was increased from 50 to 80 points.
*By default, we use a depth of h/4 when querying UCVM to populate the surface grid point.
*The padding between the nearest fault or site and the edge of the volume was increased from 30 to 50 km.
*We fixed a bug in the coordinate conversion between RWG and AWP: previously we were adding 1 to the RWG z-coordinate to produce the AWP z-coordinate, but both codes use z=1 to represent the surface and therefore no increment should be applied.
*When calculating Qs in the SGT header generation code, a default Qs of 25 was always used. This has been changed to Qs=0.05Vs.
*We have turned off the adjustment of mu and lambda.
*FP was increased from 0.5 to 1.0.
*We modified the lambda and mu calculations in AWP to use the original media parameter values from the velocity mesh rather than the FP-modified ones when calculating strains, to be consistent with RWG.

=== Study 18.8 Lessons Learned ===

*Consider separating SGT and PP workflows in auto-submit tool to better manage the number of each, for improved reservation utilization.
*Create a read-only way to look at the CyberShake Run Manager website.
*Consider reducing levels of the workflow hierarchy, thereby reducing load on shock.
*Determine advance plan for SGTs for sites which require fewer GPUs.
*Determine advance plan for SGTs for sites which exceed memory on nodes.
*Create new velocity model ID for composite model, capturing metadata.
We modified the database to enable composite models, but for this study we are just using a single model.
*Verify all Java processes grab a reasonable amount of memory.
*Clear disk space before study begins to avoid disk contention.
*Add stress test before beginning study, for multiple sites at a time, with cleanup.
*In addition to disk space, check local inode usage.
Only 1% of the inodes are used on shock-carc; we will assume /project has sufficient inodes, as we can't check them.
*Establish clear rules and policies about reservation usage.
*If submitting to multiple reservations, make sure enough jobs are eligible to run that no reservation is starved.
We are not planning to run this study with reservations.
*If running primarily SGTs for awhile, make sure they don't get deleted due to quota policies.
We will stage the SGTs to HPSS if there is a delay in post-processing them. Summit has a 90-day purge policy, so we will have some time.

== Output Data Products ==

=== File-based data products ===

We plan to produce the following data products which will be stored at CARC:

<ul>
<li>Seismograms: 2-component seismograms, 8000 timesteps (400 sec) each</li>
<li>PSA: X and Y spectral acceleration at 44 periods (10, 9.5, 9, 8.5, 8, 7.5, 7, 6.5, 6, 5.5, 5, 4.8, 4.6, 4.4, 4.2, 4, 3.8, 3.6, 3.4, 3.2, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.66667, 1.42857, 1.25, 1.11111, 1, .66667, .5, .4, .33333, .285714, .25, .22222, .2, .16667, .142857, .125, .11111, .1 sec)</li>
<li>RotD: PGV, and RotD50, the RotD50 azimuth, and RotD100 at 22 periods (1.0, 1.2, 1.4, 1.5, 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 3.5, 4.0, 4.4, 5.0, 5.5, 6.0, 6.5, 7.5, 8.5, 10.0 sec)</li>
<li>Durations: for X and Y components, energy integral, Arias intensity, cumulative absolute velocity (CAV), and for both velocity and acceleration, 5-75%, 5-95%, and 20-80%.</li>
</ul>

=== Database data products ===

We plan to store the following data products in the database:

<ul>
<li>PSA: none</li>
<li>RotD: RotD50 and RotD100 at 10, 7.5, 5, 4, 3, and 2 sec, and PGV.</li>
<li>Durations: acceleration 5-75% and 5-95% for X and Y components</li>
</ul>

== Computational and Data Estimates ==

=== Computational Estimates ===

We based these estimates by scaling from site USC (the average site has 3.8% more events and a volume 9.7% larger).

{| class="wikitable"
|+SGT calculation
! !! UCVM runtime !! UCVM nodes !! SGT runtime !! SGT nodes !! Other SGT workflow jobs !! Summit Total
|-
! USC
| 372 sec || 80 || 2628 sec || 67 || 1510 node-sec || 106.5 node-hrs
|-
! Average (est)
| 408 sec || 80 || 2883 sec || 67 || 1550 node-sec || 116.8 node-hrs
|}

Adding 10% overrun margin gives us an estimate of 43k node-hours for SGT calculation.

{| class="wikitable"
|+PP calculation
! !! DirectSynth runtime !! DirectSynth nodes !! Summit Total
|-
! USC
| 1081 || 36 || 10.8
|-
! Average (est)
| 1122 || 36 || 11.2
|}

Adding 10% overrun margin gives an estimate of 4.2k node-hours for post-processing.

=== Data Estimates ===

==== Summit ====

{| class='wikitable'
|+Data estimates
! !! Velocity mesh !! SGTs size !! Temp data !! Output data
|-
| USC || 243 GB || 196 GB || 439 GB || 4.5 GB
|-
| Average || 267 GB || 203 GB || 470 GB || 4.7 GB
|-
! Total !! 87 TB !! 66 TB !! 153 TB !! 1.5 TB
|}

This is a total of 307 TB, which we could reach if we calculate all the SGTs first and don't delete anything. The default quota on Summit is 50 TB, so I suggest we request a quota increase to at least 300 TB so we don't need to rely on cleanup.

If we need to keep the SGTs for awhile before performing post-processing, the quota on HPSS is 100 TB, so we could store them there.

==== CARC ====

We estimate 1.5 TB in output data, which will be transferred back to CARC.

==== shock-carc ====

The study should use approximately 200 GB in workflow log space on /home/shock. This drive has approximately 1.7 TB free.

==== moment database ====

The PeakAmplitudes table uses approximately 100 bytes per entry.

100 bytes/entry * 17 entries/event * 76786 events/site * 335 sites = 41 GB. The drive on moment with the mysql database has 919 GB free.

== Lessons Learned ==

== Performance Metrics ==

== Production Checklist ==

Science:
*<s>Confirm that ERF 62 test produces results which closely match ERF 61</s>
*<s>Restore improvements to codes since ERF 58, and rerun USC for ERF 62</s>
*<s>Fix h/4 issue and rerun USC test.</s>
*<s>Create prioritized site list.</s>
*<s>Hold science readiness review.</s>
*<s>Add link to fault geometry on Zenodo, either on the wiki or the fault metadata page.</s>
*<s>Add copy of science readiness review slides to wiki.</s>
*<s>Generate 0.5 Hz v 1 Hz scatterplot from UCERF2 ERF run.</s>
*Go through config updates as a pair to confirm they have been correctly applied.

Technical:
*Approach OLCF for the following requests:
**Quota increase to 400 TB
**8 jobs ready to run
**5 jobs in bin 5.
*To be able to bundle jobs, fix issue with Summit glideins.
*To run post-processing, resolve issues using GO to transfer data back to /project at CARC.
*Tag code
*<s>Modify job sizes and runtimes.</s>
*Test auto-submit script.
*Prepare pending file.
*Create XML file describing study for web monitoring tool.
*Get usage stats for Summit.
*<s>Prepare cronjob on Summit for monitoring jobs.</s>
*Call with OLCF staff
*Activate script for monitoring x509 certificate.
*Modify workflows to not insert or calculate curves for PSA data.
*<s>Modify dax-generator to use h/4 as default for surface point.</s>
*<s>Modify dax-generator to use ERF62 parameter file for generating GMPE comparison curves.</s>
*Hold technical readiness review.
*<s>Modify nt to 8000 (400 sec).</s>
*<s>Add calculation of PGV.</s>
*Test calculation and insertion of PGV.
*Test SGT-only and PP-only create/plan/run scripts.
*Fix issue with moving runs into Verified state

== Presentations, Posters, and Papers ==

Science Readiness Review: [[Media:CyberShake_Study_21.12_Readiness_Review.odp | ODP]], [[Media:CyberShake_Study_21.12_Readiness_Review.pdf | PDF]]

CyberShake Study 21.12

2021-12-08T18:36:52Z

Kmilner: /* Verification Tests */

CyberShake 21.12 is a computational study to use a new ERF with CyberShake, generated from an RSQSim catalog. We plan to calculate results for 335 sites in Southern California using the RSQSim ERF, a minimum Vs of 500 m/s, and a frequency of 1 Hz. We will use the CVM-S4.26.M01 model, and the GPU implementation of AWP-ODC-SGT enhanced from the BBP verification testing. We will begin by generating all sets of SGTs, on Summit, then post-process them on a combination of Summit and Frontera.

== Status ==

This study is in the pre-production phase. Production is scheduled to begin in mid-December, 2021.

== Data Products ==

== Science Goals ==

The science goals for this study are:

*Calculate a regional CyberShake model using an alternative, RSQSim-derived ERF.
*Compare results from an RSQSim ERF to results using a UCERF2 ERF (Study 15.4).
*Quantify effects of source model non-ergodicity
*Compare spatial distribution of ground motions (including directivity) to empirical and kinematic models

== Technical Goals ==

The technical goals for this study are:

*Perform a study using OLCF Summit as a key compute resource.
*Evaluate the performance of the new workflow submission host, shock-carc.
*Use Globus Online for staging of output data products.

== ERF ==

The ERF was generated from an RSQSim catalog, with the following parameters:
*715kyr catalog (the first 65k years of events were dropped, so that every fault's first event is excluded)
*220,927 earthquakes with M6.5+
*All events have equal probability, 1/715k

Additional details are available on the [http://opensha.usc.edu/ftp/kmilner/markdown/rsqsim-analysis/catalogs/rundir4983_stitched/#bruce-4983-stitched catalog's metadata page], and the catalog and input fault geometry files can be [https://zenodo.org/record/5542222 downloaded from zeonodo]. This is the catalog used in [https://pubs.geoscienceworld.org/ssa/bssa/article/111/2/898/593757/Toward-Physics-Based-Nonergodic-PSHA-A-Prototype Milner et al., 2021], which used 0.5 Hz CyberShake simulations performed in May, 2020.

== Sites ==

We will run a list of 335 sites, taken from the site list that was used in other Southern California studies. The order of execution will be:

*10 sites used in Milner et al. (2021), each with top mesh point Vs at the 500 m/s floor: USC, SMCA, OSI, WSS, SBSM, LAF, s022, STNI, WNGC, PDE
*PAS hard rock site
*20 km site grid
*10 km site grid
*Remaining POIs, select 5km grid sites also used in Study 15.4

{|
| [[File:Study_21.12_site_map.png|thumb|600px]]
|}

*[[Media:Study_21.12_sites.csv|CSV site list]]
*[[Media:Study_21.12_sites_names.kml|KML site list with names]]
*[[Media:Study_21.12_sites_no_names.kml|KML site list without names]]

== Velocity Model ==

We will use CVM-S4.26.M01.

To better represent the near-surface layer, we will populate the velocity parameters for the surface point by querying the velocity model at a depth of (grid spacing)/4. For this study, the grid spacing is 100m, so we will query UCVM at a depth of 25m and use that value to populate the surface grid point. The rationale is that the media parameters at the surface grid point are supposed to represent the material properties for [0, 50m], and this is better represented by using the value at 25m than the value at 0m.

== Verification Tests ==

=== USC Hazard Curves ===

Hazard curve comparisons for site USC, between:

*ERF 58: 0.5 Hz RSQSim ERF used for Milner et al. (2021) calculations, May 2020
*ERF 61: 0.5 Hz RSQSim production ERF with the same catalog
*ERF 62: 1 Hz RSQSim production ERF with the same catalog

The first test run of ERF 61 used the wrong (older) mesh lower depth of 40 km, which is why the top right plot differs slightly from the top left. The middle left plot agrees perfectly with the top left.

{|
|-
| '''ERF 58, Original'''
| '''ERF 61 w/ wrong depth'''
|-
| [[File:USC_curves_3s_ERF58.png|thumb|500px]]
| [[File:USC_curves_3s_ERF61_PREV.png|thumb|500px]]
|-
| '''ERF 61 w/ corrected depth'''
| '''ERF 62 (1 Hz) with old simulation parameters'''
|-
| [[File:USC_curves_3s_ERF61.png|thumb|500px]]
| [[File:USC_curves_3s_ERF62_FIRST.png|thumb|500px]]
|-
| '''ERF 62 (1 Hz) with proposed (new) simulation parameters'''
| '''ERF 62 ccatter comparing new parameters (y) and old parameters (x)'''
|-
| [[File:USC_curves_3s_ASK2014.png|thumb|500px]]
| [[File:Erf_62prev_62_USC_compare.png|thumb|500px]]
|}

The first 1 Hz test run uses (bottom right above) uses the same simulation paramters as used with ERF 58 and 61, changing only the frequency. Some differences are apparent, which also persist for longer period (5s) curves:

{|
|-
| '''ERF 61, 5s Sa, 0.5 Hz, old simulation parameters'''
| '''ERF 62, 5s Sa, 1 Hz, old simulation parameters'''
|-
| [[File:USC_curves_5s_ERF61.png|thumb|500px]]
| [[File:USC_curves_5s_ERF62_FIRST.png|thumb|500px]]
|}

Here are seismograms and RotD's for the largest amplitude rupture for this 0.5 Hz vs 1 Hz test:

{|
| [[File:USC_s1069_r0_1_v_0.5hz.png|thumb|800px]]
|-
| [[File:USC_s1069_r0_1_v_0.5hz_rotd.png|thumb|800px]]
|}

=== 3s Amplitude Scatter plots ===

These plots show that (left) ERF 61 exactly reproduces ERF 58, and (right) that there are indeed differences going from 0.5 Hz to 1 Hz:

{|
| [[File:Erf_58_61_USC_compare.png|thumb|500px]]
| [[File:Erf_58_62_USC_compare.png|thumb|500px]]
|}

=== 1 Hz vs 0.5 Hz comparisons ===

Here is a comparison of 1 Hz runs (y axis) and 0.5 Hz runs (x axis). The top row is a recent USC run with the RSQSim ERF. The bottom row is a previous test with UCERF2-CyberShake and the WNGC site. The columns are 3s, 5s, and 10s SA, respectively, all geometric mean.

{|
| [[File:USC_7236_vs_7237_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|-
| [[File:WNGC_3842_vs_3837_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|}

== Technical and Scientific Updates ==

Since our last study we have made a number of scientific updates to the platform, many as a result of the BBP verification effort.

*Several bugs were found and fixed in the AWP code.
*We have switched from stress insertion to velocity insertion of the impulse when generating SGTs.
*The sponge zone used in the absorbing boundary condition was increased from 50 to 80 points.
*By default, we use a depth of h/4 when querying UCVM to populate the surface grid point.
*The padding between the nearest fault or site and the edge of the volume was increased from 30 to 50 km.
*We fixed a bug in the coordinate conversion between RWG and AWP: previously we were adding 1 to the RWG z-coordinate to produce the AWP z-coordinate, but both codes use z=1 to represent the surface and therefore no increment should be applied.
*When calculating Qs in the SGT header generation code, a default Qs of 25 was always used. This has been changed to Qs=0.05Vs.
*We have turned off the adjustment of mu and lambda.
*FP was increased from 0.5 to 1.0.
*We modified the lambda and mu calculations in AWP to use the original media parameter values from the velocity mesh rather than the FP-modified ones when calculating strains, to be consistent with RWG.

=== Study 18.8 Lessons Learned ===

*Consider separating SGT and PP workflows in auto-submit tool to better manage the number of each, for improved reservation utilization.
*Create a read-only way to look at the CyberShake Run Manager website.
*Consider reducing levels of the workflow hierarchy, thereby reducing load on shock.
*Determine advance plan for SGTs for sites which require fewer GPUs.
*Determine advance plan for SGTs for sites which exceed memory on nodes.
*Create new velocity model ID for composite model, capturing metadata.
We modified the database to enable composite models, but for this study we are just using a single model.
*Verify all Java processes grab a reasonable amount of memory.
*Clear disk space before study begins to avoid disk contention.
*Add stress test before beginning study, for multiple sites at a time, with cleanup.
*In addition to disk space, check local inode usage.
Only 1% of the inodes are used on shock-carc; we will assume /project has sufficient inodes, as we can't check them.
*Establish clear rules and policies about reservation usage.
*If submitting to multiple reservations, make sure enough jobs are eligible to run that no reservation is starved.
We are not planning to run this study with reservations.
*If running primarily SGTs for awhile, make sure they don't get deleted due to quota policies.
We will stage the SGTs to HPSS if there is a delay in post-processing them. Summit has a 90-day purge policy, so we will have some time.

== Output Data Products ==

=== File-based data products ===

We plan to produce the following data products which will be stored at CARC:

<ul>
<li>Seismograms: 2-component seismograms, 8000 timesteps (400 sec) each</li>
<li>PSA: X and Y spectral acceleration at 44 periods (10, 9.5, 9, 8.5, 8, 7.5, 7, 6.5, 6, 5.5, 5, 4.8, 4.6, 4.4, 4.2, 4, 3.8, 3.6, 3.4, 3.2, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.66667, 1.42857, 1.25, 1.11111, 1, .66667, .5, .4, .33333, .285714, .25, .22222, .2, .16667, .142857, .125, .11111, .1 sec)</li>
<li>RotD: PGV, and RotD50, the RotD50 azimuth, and RotD100 at 22 periods (1.0, 1.2, 1.4, 1.5, 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 3.5, 4.0, 4.4, 5.0, 5.5, 6.0, 6.5, 7.5, 8.5, 10.0 sec)</li>
<li>Durations: for X and Y components, energy integral, Arias intensity, cumulative absolute velocity (CAV), and for both velocity and acceleration, 5-75%, 5-95%, and 20-80%.</li>
</ul>

=== Database data products ===

We plan to store the following data products in the database:

<ul>
<li>PSA: none</li>
<li>RotD: RotD50 and RotD100 at 10, 7.5, 5, 4, 3, and 2 sec, and PGV.</li>
<li>Durations: acceleration 5-75% and 5-95% for X and Y components</li>
</ul>

== Computational and Data Estimates ==

=== Computational Estimates ===

We based these estimates by scaling from site USC (the average site has 3.8% more events and a volume 9.7% larger).

{| class="wikitable"
|+SGT calculation
! !! UCVM runtime !! UCVM nodes !! SGT runtime !! SGT nodes !! Other SGT workflow jobs !! Summit Total
|-
! USC
| 372 sec || 80 || 2628 sec || 67 || 1510 node-sec || 106.5 node-hrs
|-
! Average (est)
| 408 sec || 80 || 2883 sec || 67 || 1550 node-sec || 116.8 node-hrs
|}

Adding 10% overrun margin gives us an estimate of 43k node-hours for SGT calculation.

{| class="wikitable"
|+PP calculation
! !! DirectSynth runtime !! DirectSynth nodes !! Summit Total
|-
! USC
| 1081 || 36 || 10.8
|-
! Average (est)
| 1122 || 36 || 11.2
|}

Adding 10% overrun margin gives an estimate of 4.2k node-hours for post-processing.

=== Data Estimates ===

==== Summit ====

{| class='wikitable'
|+Data estimates
! !! Velocity mesh !! SGTs size !! Temp data !! Output data
|-
| USC || 243 GB || 196 GB || 439 GB || 4.5 GB
|-
| Average || 267 GB || 203 GB || 470 GB || 4.7 GB
|-
! Total !! 87 TB !! 66 TB !! 153 TB !! 1.5 TB
|}

This is a total of 307 TB, which we could reach if we calculate all the SGTs first and don't delete anything. The default quota on Summit is 50 TB, so I suggest we request a quota increase to at least 300 TB so we don't need to rely on cleanup.

If we need to keep the SGTs for awhile before performing post-processing, the quota on HPSS is 100 TB, so we could store them there.

==== CARC ====

We estimate 1.5 TB in output data, which will be transferred back to CARC.

==== shock-carc ====

The study should use approximately 200 GB in workflow log space on /home/shock. This drive has approximately 1.7 TB free.

==== moment database ====

The PeakAmplitudes table uses approximately 100 bytes per entry.

100 bytes/entry * 17 entries/event * 76786 events/site * 335 sites = 41 GB. The drive on moment with the mysql database has 919 GB free.

== Lessons Learned ==

== Performance Metrics ==

== Production Checklist ==

Science:
*<s>Confirm that ERF 62 test produces results which closely match ERF 61</s>
*<s>Restore improvements to codes since ERF 58, and rerun USC for ERF 62</s>
*Fix h/4 issue and rerun USC test.
*<s>Create prioritized site list.</s>
*<s>Hold science readiness review.</s>
*<s>Add link to fault geometry on Zenodo, either on the wiki or the fault metadata page.</s>
*<s>Add copy of science readiness review slides to wiki.</s>
*<s>Generate 0.5 Hz v 1 Hz scatterplot from UCERF2 ERF run.</s>
*Go through config updates as a pair to confirm they have been correctly applied.

Technical:
*Approach OLCF for the following requests:
**Quota increase to 400 TB
**8 jobs ready to run
**5 jobs in bin 5.
*To be able to bundle jobs, fix issue with Summit glideins.
*To run post-processing, resolve issues using GO to transfer data back to /project at CARC.
*Tag code
*<s>Modify job sizes and runtimes.</s>
*Test auto-submit script.
*Prepare pending file.
*Create XML file describing study for web monitoring tool.
*Get usage stats for Summit.
*<s>Prepare cronjob on Summit for monitoring jobs.</s>
*Call with OLCF staff
*Activate script for monitoring x509 certificate.
*Modify workflows to not insert or calculate curves for PSA data.
*<s>Modify dax-generator to use h/4 as default for surface point.</s>
*<s>Modify dax-generator to use ERF62 parameter file for generating GMPE comparison curves.</s>
*Hold technical readiness review.
*<s>Modify nt to 8000 (400 sec).</s>
*<s>Add calculation of PGV.</s>
*Test calculation and insertion of PGV.
*Test SGT-only and PP-only create/plan/run scripts.
*Fix issue with moving runs into Verified state

== Presentations, Posters, and Papers ==

Science Readiness Review: [[Media:CyberShake_Study_21.12_Readiness_Review.odp | ODP]], [[Media:CyberShake_Study_21.12_Readiness_Review.pdf | PDF]]

File:USC curves 3s ASK2014.png

2021-12-08T18:34:09Z

Kmilner:

File:Erf 62prev 62 USC compare.png

2021-12-08T18:33:29Z

Kmilner:

CyberShake Study 21.12

2021-12-07T01:39:19Z

Kmilner: /* 1 Hz vs 0.5 Hz comparisons */

CyberShake 21.12 is a computational study to use a new ERF with CyberShake, generated from an RSQSim catalog. We plan to calculate results for 335 sites in Southern California using the RSQSim ERF, a minimum Vs of 500 m/s, and a frequency of 1 Hz. We will use the CVM-S4.26.M01 model, and the GPU implementation of AWP-ODC-SGT enhanced from the BBP verification testing. We will begin by generating all sets of SGTs, on Summit, then post-process them on a combination of Summit and Frontera.

== Status ==

This study is in the pre-production phase. Production is scheduled to begin in mid-December, 2021.

== Data Products ==

== Science Goals ==

The science goals for this study are:

*Calculate a regional CyberShake model using an alternative, RSQSim-derived ERF.
*Compare results from an RSQSim ERF to results using a UCERF2 ERF (Study 15.4).
*Quantify effects of source model non-ergodicity
*Compare spatial distribution of ground motions (including directivity) to empirical and kinematic models

== Technical Goals ==

The technical goals for this study are:

*Perform a study using OLCF Summit as a key compute resource.
*Evaluate the performance of the new workflow submission host, shock-carc.
*Use Globus Online for staging of output data products.

== ERF ==

The ERF was generated from an RSQSim catalog, with the following parameters:
*715kyr catalog (the first 65k years of events were dropped, so that every fault's first event is excluded)
*220,927 earthquakes with M6.5+
*All events have equal probability, 1/715k

Additional details are available on the [http://opensha.usc.edu/ftp/kmilner/markdown/rsqsim-analysis/catalogs/rundir4983_stitched/#bruce-4983-stitched catalog's metadata page], and the catalog and input fault geometry files can be [https://zenodo.org/record/5542222 downloaded from zeonodo]. This is the catalog used in [https://pubs.geoscienceworld.org/ssa/bssa/article/111/2/898/593757/Toward-Physics-Based-Nonergodic-PSHA-A-Prototype Milner et al., 2021], which used 0.5 Hz CyberShake simulations performed in May, 2020.

== Sites ==

We will run a list of 335 sites, taken from the site list that was used in other Southern California studies. The order of execution will be:

*10 sites used in Milner et al. (2021), each with top mesh point Vs at the 500 m/s floor: USC, SMCA, OSI, WSS, SBSM, LAF, s022, STNI, WNGC, PDE
*PAS hard rock site
*20 km site grid
*10 km site grid
*Remaining POIs, select 5km grid sites also used in Study 15.4

{|
| [[File:Study_21.12_site_map.png|thumb|600px]]
|}

*[[Media:Study_21.12_sites.csv|CSV site list]]
*[[Media:Study_21.12_sites_names.kml|KML site list with names]]
*[[Media:Study_21.12_sites_no_names.kml|KML site list without names]]

== Velocity Model ==

We will use CVM-S4.26.M01.

To better represent the near-surface layer, we will populate the velocity parameters for the surface point by querying the velocity model at a depth of (grid spacing)/4. For this study, the grid spacing is 100m, so we will query UCVM at a depth of 25m and use that value to populate the surface grid point. The rationale is that the media parameters at the surface grid point are supposed to represent the material properties for [0, 50m], and this is better represented by using the value at 25m than the value at 0m.

== Verification Tests ==

=== USC Hazard Curves ===

Hazard curve comparisons for site USC, between:

*ERF 58: 0.5 Hz RSQSim ERF used for Milner et al. (2021) calculations, May 2020
*ERF 61: 0.5 Hz RSQSim production ERF with the same catalog
*ERF 62: 1 Hz RSQSim production ERF with the same catalog

The first test run of ERF 61 used the wrong (older) mesh lower depth of 40 km, which is why the top right plot differs slightly from the top left. The bottom left plot agrees perfectly with the tol left.

{|
|-
| '''ERF 58, Original'''
| '''ERF 61 w/ wrong depth'''
|-
| [[File:USC_curves_3s_ERF58.png|thumb|500px]]
| [[File:USC_curves_3s_ERF61_PREV.png|thumb|500px]]
|-
| '''ERF 61 w/ corrected depth'''
| '''ERF 62 (1 Hz) with old simulation parameters'''
|-
| [[File:USC_curves_3s_ERF61.png|thumb|500px]]
| [[File:USC_curves_3s_ERF62_FIRST.png|thumb|500px]]
|}

The first 1 Hz test run uses (bottom right above) uses the same simulation paramters as used with ERF 58 and 61, changing only the frequency. Some differences are apparent, which also persist for longer period (5s) curves:

{|
|-
| '''ERF 61, 5s Sa, 0.5 Hz, old simulation parameters'''
| '''ERF 62, 5s Sa, 1 Hz, old simulation parameters'''
|-
| [[File:USC_curves_5s_ERF61.png|thumb|500px]]
| [[File:USC_curves_5s_ERF62_FIRST.png|thumb|500px]]
|}

Here are seismograms and RotD's for the largest amplitude rupture for this 0.5 Hz vs 1 Hz test:

{|
| [[File:USC_s1069_r0_1_v_0.5hz.png|thumb|800px]]
|-
| [[File:USC_s1069_r0_1_v_0.5hz_rotd.png|thumb|800px]]
|}

=== 3s Amplitude Scatter plots ===

These plots show that (left) ERF 61 exactly reproduces ERF 58, and (right) that there are indeed differences going from 0.5 Hz to 1 Hz:

{|
| [[File:Erf_58_61_USC_compare.png|thumb|500px]]
| [[File:Erf_58_62_USC_compare.png|thumb|500px]]
|}

=== 1 Hz vs 0.5 Hz comparisons ===

Here is a comparison of 1 Hz runs (y axis) and 0.5 Hz runs (x axis). The top row is a recent USC run with the RSQSim ERF. The bottom row is a previous test with UCERF2-CyberShake and the WNGC site. The columns are 3s, 5s, and 10s SA, respectively, all geometric mean.

{|
| [[File:USC_7236_vs_7237_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|-
| [[File:WNGC_3842_vs_3837_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|}

== Technical and Scientific Updates ==

Since our last study we have made a number of scientific updates to the platform, many as a result of the BBP verification effort.

*Several bugs were found and fixed in the AWP code.
*We have switched from stress insertion to velocity insertion of the impulse when generating SGTs.
*The sponge zone used in the absorbing boundary condition was increased from 50 to 80 points.
*By default, we use a depth of h/4 when querying UCVM to populate the surface grid point.
*The padding between the nearest fault or site and the edge of the volume was increased from 30 to 50 km.
*We fixed a bug in the coordinate conversion between RWG and AWP: previously we were adding 1 to the RWG z-coordinate to produce the AWP z-coordinate, but both codes use z=1 to represent the surface and therefore no increment should be applied.
*When calculating Qs in the SGT header generation code, a default Qs of 25 was always used. This has been changed to Qs=0.05Vs.
*We have turned off the adjustment of mu and lambda.
*FP was increased from 0.5 to 1.0.
*We modified the lambda and mu calculations in AWP to use the original media parameter values from the velocity mesh rather than the FP-modified ones when calculating strains, to be consistent with RWG.

=== Study 18.8 Lessons Learned ===

*Consider separating SGT and PP workflows in auto-submit tool to better manage the number of each, for improved reservation utilization.
*Create a read-only way to look at the CyberShake Run Manager website.
*Consider reducing levels of the workflow hierarchy, thereby reducing load on shock.
*Determine advance plan for SGTs for sites which require fewer GPUs.
*Determine advance plan for SGTs for sites which exceed memory on nodes.
*Create new velocity model ID for composite model, capturing metadata.
We modified the database to enable composite models, but for this study we are just using a single model.
*Verify all Java processes grab a reasonable amount of memory.
*Clear disk space before study begins to avoid disk contention.
*Add stress test before beginning study, for multiple sites at a time, with cleanup.
*In addition to disk space, check local inode usage.
Only 1% of the inodes are used on shock-carc; we will assume /project has sufficient inodes, as we can't check them.
*Establish clear rules and policies about reservation usage.
*If submitting to multiple reservations, make sure enough jobs are eligible to run that no reservation is starved.
We are not planning to run this study with reservations.
*If running primarily SGTs for awhile, make sure they don't get deleted due to quota policies.
We will stage the SGTs to HPSS if there is a delay in post-processing them. Summit has a 90-day purge policy, so we will have some time.

== Output Data Products ==

=== File-based data products ===

We plan to produce the following data products which will be stored at CARC:

<ul>
<li>Seismograms: 2-component seismograms, 8000 timesteps (400 sec) each</li>
<li>PSA: X and Y spectral acceleration at 44 periods (10, 9.5, 9, 8.5, 8, 7.5, 7, 6.5, 6, 5.5, 5, 4.8, 4.6, 4.4, 4.2, 4, 3.8, 3.6, 3.4, 3.2, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.66667, 1.42857, 1.25, 1.11111, 1, .66667, .5, .4, .33333, .285714, .25, .22222, .2, .16667, .142857, .125, .11111, .1 sec)</li>
<li>RotD: RotD50, the RotD50 azimuth, and RotD100 at 22 periods (1.0, 1.2, 1.4, 1.5, 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 3.5, 4.0, 4.4, 5.0, 5.5, 6.0, 6.5, 7.5, 8.5, 10.0 sec)</li>
<li>Durations: for X and Y components, energy integral, Arias intensity, cumulative absolute velocity (CAV), and for both velocity and acceleration, 5-75%, 5-95%, and 20-80%.</li>
</ul>

=== Database data products ===

We plan to store the following data products in the database:

<ul>
<li>PSA: none</li>
<li>RotD: RotD50 and RotD100 at 10, 7.5, 5, 4, 3, and 2 sec.</li>
<li>Durations: acceleration 5-75% and 5-95% for X and Y components</li>
</ul>

== Computational and Data Estimates ==

=== Computational Estimates ===

We based these estimates by scaling from site USC (the average site has 3.8% more events and a volume 9.7% larger).

{| class="wikitable"
|+SGT calculation
! !! UCVM runtime !! UCVM nodes !! SGT runtime !! SGT nodes !! Other SGT workflow jobs !! Summit Total
|-
! USC
| 372 sec || 80 || 2628 sec || 67 || 1510 node-sec || 106.5 node-hrs
|-
! Average (est)
| 408 sec || 80 || 2883 sec || 67 || 1550 node-sec || 116.8 node-hrs
|}

Adding 10% overrun margin gives us an estimate of 43k node-hours for SGT calculation.

{| class="wikitable"
|+PP calculation
! !! DirectSynth runtime !! DirectSynth nodes !! Summit Total
|-
! USC
| 1081 || 36 || 10.8
|-
! Average (est)
| 1122 || 36 || 11.2
|}

Adding 10% overrun margin gives an estimate of 4.2k node-hours for post-processing.

=== Data Estimates ===

==== Summit ====

{| class='wikitable'
|+Data estimates
! !! Velocity mesh !! SGTs size !! Temp data !! Output data
|-
| USC || 243 GB || 196 GB || 439 GB || 4.5 GB
|-
| Average || 267 GB || 203 GB || 470 GB || 4.7 GB
|-
! Total !! 87 TB !! 66 TB !! 153 TB !! 1.5 TB
|}

This is a total of 307 TB, which we could reach if we calculate all the SGTs first and don't delete anything. The default quota on Summit is 50 TB, so I suggest we request a quota increase to at least 300 TB so we don't need to rely on cleanup.

If we need to keep the SGTs for awhile before performing post-processing, the quota on HPSS is 100 TB, so we could store them there.

==== CARC ====

We estimate 1.5 TB in output data, which will be transferred back to CARC.

==== shock-carc ====

The study should use approximately 200 GB in workflow log space on /home/shock. This drive has approximately 1.7 TB free.

==== moment database ====

The PeakAmplitudes table uses approximately 100 bytes per entry.

100 bytes/entry * 16 entries/event * 76786 events/site * 335 sites = 38 GB. The drive on moment with the mysql database has 919 GB free.

== Lessons Learned ==

== Performance Metrics ==

== Production Checklist ==

Science:
*<s>Confirm that ERF 62 test produces results which closely match ERF 61</s>
*<s>Restore improvements to codes since ERF 58, and rerun USC for ERF 62</s>
*Fix h/4 issue and rerun USC test.
*<s>Create prioritized site list.</s>
*<s>Hold science readiness review.</s>
*<s>Add link to fault geometry on Zenodo, either on the wiki or the fault metadata page.</s>
*<s>Add copy of science readiness review slides to wiki.</s> [[Media:CyberShake_Study_21.12_Readiness_Review.pdf]] [[Media:CyberShake_Study_21.12_Readiness_Review.odp]]
*<s>Generate 0.5 Hz v 1 Hz scatterplot from UCERF2 ERF run.</s>
*Go through config updates as a pair to confirm they have been correctly applied.

Technical:
*Approach OLCF for the following requests:
**Quota increase to 400 TB
**8 jobs ready to run
**5 jobs in bin 5.
*To be able to bundle jobs, fix issue with Summit glideins.
*To run post-processing, resolve issues using GO to transfer data back to /project at CARC.
*Tag code
*<s>Modify job sizes and runtimes.</s>
*Test auto-submit script.
*Prepare pending file.
*Create XML file describing study for web monitoring tool.
*Get usage stats for Summit.
*<s>Prepare cronjob on Summit for monitoring jobs.</s>
*Call with OLCF staff
*Activate script for monitoring x509 certificate.
*Modify workflows to not insert or calculate curves for PSA data.
*<s>Modify dax-generator to use h/4 as default for surface point.</s>
*<s>Modify dax-generator to use ERF62 parameter file for generating GMPE comparison curves.</s>
*Hold technical readiness review.
*Modify nt to 8000 (400 sec).
*Add calculation of PGV.
*Test SGT-only and PP-only create/plan/run scripts.

== Presentations, Posters, and Papers ==

CyberShake Study 21.12

2021-12-07T01:35:36Z

Kmilner: /* Production Checklist */

CyberShake 21.12 is a computational study to use a new ERF with CyberShake, generated from an RSQSim catalog. We plan to calculate results for 335 sites in Southern California using the RSQSim ERF, a minimum Vs of 500 m/s, and a frequency of 1 Hz. We will use the CVM-S4.26.M01 model, and the GPU implementation of AWP-ODC-SGT enhanced from the BBP verification testing. We will begin by generating all sets of SGTs, on Summit, then post-process them on a combination of Summit and Frontera.

== Status ==

This study is in the pre-production phase. Production is scheduled to begin in mid-December, 2021.

== Data Products ==

== Science Goals ==

The science goals for this study are:

*Calculate a regional CyberShake model using an alternative, RSQSim-derived ERF.
*Compare results from an RSQSim ERF to results using a UCERF2 ERF (Study 15.4).
*Quantify effects of source model non-ergodicity
*Compare spatial distribution of ground motions (including directivity) to empirical and kinematic models

== Technical Goals ==

The technical goals for this study are:

*Perform a study using OLCF Summit as a key compute resource.
*Evaluate the performance of the new workflow submission host, shock-carc.
*Use Globus Online for staging of output data products.

== ERF ==

The ERF was generated from an RSQSim catalog, with the following parameters:
*715kyr catalog (the first 65k years of events were dropped, so that every fault's first event is excluded)
*220,927 earthquakes with M6.5+
*All events have equal probability, 1/715k

Additional details are available on the [http://opensha.usc.edu/ftp/kmilner/markdown/rsqsim-analysis/catalogs/rundir4983_stitched/#bruce-4983-stitched catalog's metadata page], and the catalog and input fault geometry files can be [https://zenodo.org/record/5542222 downloaded from zeonodo]. This is the catalog used in [https://pubs.geoscienceworld.org/ssa/bssa/article/111/2/898/593757/Toward-Physics-Based-Nonergodic-PSHA-A-Prototype Milner et al., 2021], which used 0.5 Hz CyberShake simulations performed in May, 2020.

== Sites ==

We will run a list of 335 sites, taken from the site list that was used in other Southern California studies. The order of execution will be:

*10 sites used in Milner et al. (2021), each with top mesh point Vs at the 500 m/s floor: USC, SMCA, OSI, WSS, SBSM, LAF, s022, STNI, WNGC, PDE
*PAS hard rock site
*20 km site grid
*10 km site grid
*Remaining POIs, select 5km grid sites also used in Study 15.4

{|
| [[File:Study_21.12_site_map.png|thumb|600px]]
|}

*[[Media:Study_21.12_sites.csv|CSV site list]]
*[[Media:Study_21.12_sites_names.kml|KML site list with names]]
*[[Media:Study_21.12_sites_no_names.kml|KML site list without names]]

== Velocity Model ==

We will use CVM-S4.26.M01.

To better represent the near-surface layer, we will populate the velocity parameters for the surface point by querying the velocity model at a depth of (grid spacing)/4. For this study, the grid spacing is 100m, so we will query UCVM at a depth of 25m and use that value to populate the surface grid point. The rationale is that the media parameters at the surface grid point are supposed to represent the material properties for [0, 50m], and this is better represented by using the value at 25m than the value at 0m.

== Verification Tests ==

=== USC Hazard Curves ===

Hazard curve comparisons for site USC, between:

*ERF 58: 0.5 Hz RSQSim ERF used for Milner et al. (2021) calculations, May 2020
*ERF 61: 0.5 Hz RSQSim production ERF with the same catalog
*ERF 62: 1 Hz RSQSim production ERF with the same catalog

The first test run of ERF 61 used the wrong (older) mesh lower depth of 40 km, which is why the top right plot differs slightly from the top left. The bottom left plot agrees perfectly with the tol left.

{|
|-
| '''ERF 58, Original'''
| '''ERF 61 w/ wrong depth'''
|-
| [[File:USC_curves_3s_ERF58.png|thumb|500px]]
| [[File:USC_curves_3s_ERF61_PREV.png|thumb|500px]]
|-
| '''ERF 61 w/ corrected depth'''
| '''ERF 62 (1 Hz) with old simulation parameters'''
|-
| [[File:USC_curves_3s_ERF61.png|thumb|500px]]
| [[File:USC_curves_3s_ERF62_FIRST.png|thumb|500px]]
|}

The first 1 Hz test run uses (bottom right above) uses the same simulation paramters as used with ERF 58 and 61, changing only the frequency. Some differences are apparent, which also persist for longer period (5s) curves:

{|
|-
| '''ERF 61, 5s Sa, 0.5 Hz, old simulation parameters'''
| '''ERF 62, 5s Sa, 1 Hz, old simulation parameters'''
|-
| [[File:USC_curves_5s_ERF61.png|thumb|500px]]
| [[File:USC_curves_5s_ERF62_FIRST.png|thumb|500px]]
|}

Here are seismograms and RotD's for the largest amplitude rupture for this 0.5 Hz vs 1 Hz test:

{|
| [[File:USC_s1069_r0_1_v_0.5hz.png|thumb|800px]]
|-
| [[File:USC_s1069_r0_1_v_0.5hz_rotd.png|thumb|800px]]
|}

=== 3s Amplitude Scatter plots ===

These plots show that (left) ERF 61 exactly reproduces ERF 58, and (right) that there are indeed differences going from 0.5 Hz to 1 Hz:

{|
| [[File:Erf_58_61_USC_compare.png|thumb|500px]]
| [[File:Erf_58_62_USC_compare.png|thumb|500px]]
|}

=== 1 Hz vs 0.5 Hz comparisons ===

Here is a comparison of 1 Hz runs (y axis) and 0.5 Hz runs (x axis). The top row is a recent USC run, with the RSQSim ERF. The bottom row is a previous test with UCERF2-CyberShake, and the WNGC site.

{|
| [[File:USC_7236_vs_7237_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:USC_7236_vs_7237_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|-
| [[File:WNGC_3842_vs_3837_scatter_3.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_5.0s_GEOM_compare.png|thumb|400px]]
| [[File:WNGC_3842_vs_3837_scatter_10.0s_GEOM_compare.png|thumb|400px]]
|}

== Technical and Scientific Updates ==

Since our last study we have made a number of scientific updates to the platform, many as a result of the BBP verification effort.

*Several bugs were found and fixed in the AWP code.
*We have switched from stress insertion to velocity insertion of the impulse when generating SGTs.
*The sponge zone used in the absorbing boundary condition was increased from 50 to 80 points.
*By default, we use a depth of h/4 when querying UCVM to populate the surface grid point.
*The padding between the nearest fault or site and the edge of the volume was increased from 30 to 50 km.
*We fixed a bug in the coordinate conversion between RWG and AWP: previously we were adding 1 to the RWG z-coordinate to produce the AWP z-coordinate, but both codes use z=1 to represent the surface and therefore no increment should be applied.
*When calculating Qs in the SGT header generation code, a default Qs of 25 was always used. This has been changed to Qs=0.05Vs.
*We have turned off the adjustment of mu and lambda.
*FP was increased from 0.5 to 1.0.
*We modified the lambda and mu calculations in AWP to use the original media parameter values from the velocity mesh rather than the FP-modified ones when calculating strains, to be consistent with RWG.

=== Study 18.8 Lessons Learned ===

*Consider separating SGT and PP workflows in auto-submit tool to better manage the number of each, for improved reservation utilization.
*Create a read-only way to look at the CyberShake Run Manager website.
*Consider reducing levels of the workflow hierarchy, thereby reducing load on shock.
*Determine advance plan for SGTs for sites which require fewer GPUs.
*Determine advance plan for SGTs for sites which exceed memory on nodes.
*Create new velocity model ID for composite model, capturing metadata.
We modified the database to enable composite models, but for this study we are just using a single model.
*Verify all Java processes grab a reasonable amount of memory.
*Clear disk space before study begins to avoid disk contention.
*Add stress test before beginning study, for multiple sites at a time, with cleanup.
*In addition to disk space, check local inode usage.
Only 1% of the inodes are used on shock-carc; we will assume /project has sufficient inodes, as we can't check them.
*Establish clear rules and policies about reservation usage.
*If submitting to multiple reservations, make sure enough jobs are eligible to run that no reservation is starved.
We are not planning to run this study with reservations.
*If running primarily SGTs for awhile, make sure they don't get deleted due to quota policies.
We will stage the SGTs to HPSS if there is a delay in post-processing them. Summit has a 90-day purge policy, so we will have some time.

== Output Data Products ==

=== File-based data products ===

We plan to produce the following data products which will be stored at CARC:

<ul>
<li>Seismograms: 2-component seismograms, 8000 timesteps (400 sec) each</li>
<li>PSA: X and Y spectral acceleration at 44 periods (10, 9.5, 9, 8.5, 8, 7.5, 7, 6.5, 6, 5.5, 5, 4.8, 4.6, 4.4, 4.2, 4, 3.8, 3.6, 3.4, 3.2, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.66667, 1.42857, 1.25, 1.11111, 1, .66667, .5, .4, .33333, .285714, .25, .22222, .2, .16667, .142857, .125, .11111, .1 sec)</li>
<li>RotD: RotD50, the RotD50 azimuth, and RotD100 at 22 periods (1.0, 1.2, 1.4, 1.5, 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 3.5, 4.0, 4.4, 5.0, 5.5, 6.0, 6.5, 7.5, 8.5, 10.0 sec)</li>
<li>Durations: for X and Y components, energy integral, Arias intensity, cumulative absolute velocity (CAV), and for both velocity and acceleration, 5-75%, 5-95%, and 20-80%.</li>
</ul>

=== Database data products ===

We plan to store the following data products in the database:

<ul>
<li>PSA: none</li>
<li>RotD: RotD50 and RotD100 at 10, 7.5, 5, 4, 3, and 2 sec.</li>
<li>Durations: acceleration 5-75% and 5-95% for X and Y components</li>
</ul>

== Computational and Data Estimates ==

=== Computational Estimates ===

We based these estimates by scaling from site USC (the average site has 3.8% more events and a volume 9.7% larger).

{| class="wikitable"
|+SGT calculation
! !! UCVM runtime !! UCVM nodes !! SGT runtime !! SGT nodes !! Other SGT workflow jobs !! Summit Total
|-
! USC
| 372 sec || 80 || 2628 sec || 67 || 1510 node-sec || 106.5 node-hrs
|-
! Average (est)
| 408 sec || 80 || 2883 sec || 67 || 1550 node-sec || 116.8 node-hrs
|}

Adding 10% overrun margin gives us an estimate of 43k node-hours for SGT calculation.

{| class="wikitable"
|+PP calculation
! !! DirectSynth runtime !! DirectSynth nodes !! Summit Total
|-
! USC
| 1081 || 36 || 10.8
|-
! Average (est)
| 1122 || 36 || 11.2
|}

Adding 10% overrun margin gives an estimate of 4.2k node-hours for post-processing.

=== Data Estimates ===

==== Summit ====

{| class='wikitable'
|+Data estimates
! !! Velocity mesh !! SGTs size !! Temp data !! Output data
|-
| USC || 243 GB || 196 GB || 439 GB || 4.5 GB
|-
| Average || 267 GB || 203 GB || 470 GB || 4.7 GB
|-
! Total !! 87 TB !! 66 TB !! 153 TB !! 1.5 TB
|}

This is a total of 307 TB, which we could reach if we calculate all the SGTs first and don't delete anything. The default quota on Summit is 50 TB, so I suggest we request a quota increase to at least 300 TB so we don't need to rely on cleanup.

If we need to keep the SGTs for awhile before performing post-processing, the quota on HPSS is 100 TB, so we could store them there.

==== CARC ====

We estimate 1.5 TB in output data, which will be transferred back to CARC.

==== shock-carc ====

The study should use approximately 200 GB in workflow log space on /home/shock. This drive has approximately 1.7 TB free.

==== moment database ====

The PeakAmplitudes table uses approximately 100 bytes per entry.

100 bytes/entry * 16 entries/event * 76786 events/site * 335 sites = 38 GB. The drive on moment with the mysql database has 919 GB free.

== Lessons Learned ==

== Performance Metrics ==

== Production Checklist ==

Science:
*<s>Confirm that ERF 62 test produces results which closely match ERF 61</s>
*<s>Restore improvements to codes since ERF 58, and rerun USC for ERF 62</s>
*Fix h/4 issue and rerun USC test.
*<s>Create prioritized site list.</s>
*<s>Hold science readiness review.</s>
*<s>Add link to fault geometry on Zenodo, either on the wiki or the fault metadata page.</s>
*<s>Add copy of science readiness review slides to wiki.</s> [[Media:CyberShake_Study_21.12_Readiness_Review.pdf]] [[Media:CyberShake_Study_21.12_Readiness_Review.odp]]
*<s>Generate 0.5 Hz v 1 Hz scatterplot from UCERF2 ERF run.</s>
*Go through config updates as a pair to confirm they have been correctly applied.

Technical:
*Approach OLCF for the following requests:
**Quota increase to 400 TB
**8 jobs ready to run
**5 jobs in bin 5.
*To be able to bundle jobs, fix issue with Summit glideins.
*To run post-processing, resolve issues using GO to transfer data back to /project at CARC.
*Tag code
*<s>Modify job sizes and runtimes.</s>
*Test auto-submit script.
*Prepare pending file.
*Create XML file describing study for web monitoring tool.
*Get usage stats for Summit.
*<s>Prepare cronjob on Summit for monitoring jobs.</s>
*Call with OLCF staff
*Activate script for monitoring x509 certificate.
*Modify workflows to not insert or calculate curves for PSA data.
*<s>Modify dax-generator to use h/4 as default for surface point.</s>
*<s>Modify dax-generator to use ERF62 parameter file for generating GMPE comparison curves.</s>
*Hold technical readiness review.
*Modify nt to 8000 (400 sec).
*Add calculation of PGV.
*Test SGT-only and PP-only create/plan/run scripts.

== Presentations, Posters, and Papers ==

File:CyberShake Study 21.12 Readiness Review.odp

2021-12-07T01:33:45Z

Kmilner:

File:CyberShake Study 21.12 Readiness Review.pdf

2021-12-07T01:33:13Z

Kmilner:

CyberShake Study 21.12

2021-12-07T01:31:37Z

CyberShake Study 21.12

2021-12-03T19:26:30Z

2021-12-03T19:18:30Z

2018-01-19T19:33:14Z

Kmilner:

'''Goal: determine the importance of background seismicity in different parts of CA to either validate it's exclusion in CyberShake or motivate an inclusion strategy.'''

== UCERF2 ==

Calculation parameters:
* GMPE: NGA-West2 GMPE (average of 4 models)
* ERF: UCERF2, with and without background seismicity
* Site Effects: Vs30 is from Wills (2015), no basin depth terms included
* IMT: 2 second SA.

{|
! Model
! 10% in 50yr
! 2% in 50yr
! 1% in 50yr
|-
! Excluding Background
| [[File:ngaw2_exclude_2sec_10p_in_50.png|400px|thumb|left]]
| [[File:ngaw2_exclude_2sec_2p_in_50.png|400px|thumb|left]]
| [[File:ngaw2_exclude_2sec_1p_in_50.png|400px|thumb|left]]
|-
! Including Background
| [[File:ngaw2_include_2sec_10p_in_50.png|400px|thumb|left]]
| [[File:ngaw2_include_2sec_2p_in_50.png|400px|thumb|left]]
| [[File:ngaw2_include_2sec_1p_in_50.png|400px|thumb|left]]
|-
! Ratio Exclude/Include
| [[File:ratio_ngaw2_exclude_ngaw2_include_10p_in_50.png|400px|thumb|left]]
| [[File:ratio_ngaw2_exclude_ngaw2_include_2p_in_50.png|400px|thumb|left]]
| [[File:ratio_ngaw2_exclude_ngaw2_include_1p_in_50.png|400px|thumb|left]]
|}

== UCERF3 ==

Calculation parameters:
* GMPE: NGA-West2 GMPE (average of 4 models)
* ERF: UCERF3, with the following configurations
** Supra-seismogenic ruptures only (only supra-seismogenic ruptures have finite fault surfaces in UCERF3, all others treated as gridded)
** Supra and Sub-seismogenic ruptures only (this adds gridded ruptures that correspond to sub-seimogenic ruptures on known faults)
** Supra, Sub-seismogenic, and off fault ruptures (complete model)
* Site Effects: Vs30 is from Wills (2015), no basin depth terms included
* IMT: 2 second SA.

{|
! Model
! 10% in 50yr
! 2% in 50yr
! 1% in 50yr
|-
! Supra-Seismogenic On-Fault Only
| [[File:ucerf3_supra_only_10p_in_50.png|400px|thumb|left]]
| [[File:ucerf3_supra_only_2p_in_50.png|400px|thumb|left]]
| [[File:ucerf3_supra_only_1p_in_50.png|400px|thumb|left]]
|-
! Sub+Supra-Seismogenic On-Fault Only
| [[File:ucerf3_sub+supra_10p_in_50.png|400px|thumb|left]]
| [[File:ucerf3_sub+supra_2p_in_50.png|400px|thumb|left]]
| [[File:ucerf3_sub+supra_1p_in_50.png|400px|thumb|left]]
|-
! Complete Model (Supra+Sub+Off)
| [[File:ucerf3_sub+supra+off_10p_in_50.png|400px|thumb|left]]
| [[File:ucerf3_sub+supra+off_2p_in_50.png|400px|thumb|left]]
| [[File:ucerf3_sub+supra+off_1p_in_50.png|400px|thumb|left]]
|-
! Ratio Supra-Only/Complete
| [[File:ratio_ucerf3_supra_only_vs_sub+supra+off_10p_in_50.png|400px|thumb|left]]
| [[File:ratio_ucerf3_supra_only_vs_sub+supra+off_2p_in_50.png|400px|thumb|left]]
| [[File:ratio_ucerf3_supra_only_vs_sub+supra+off_1p_in_50.png|400px|thumb|left]]
|-
! Ratio Supra+Sub/Complete
| [[File:ratio_ucerf3_sub+supra_vs_sub+supra+off_10p_in_50.png|400px|thumb|left]]
| [[File:ratio_ucerf3_sub+supra_vs_sub+supra+off_2p_in_50.png|400px|thumb|left]]
| [[File:ratio_ucerf3_sub+supra_vs_sub+supra+off_1p_in_50.png|400px|thumb|left]]
|}

File:Ucerf3 supra only 10p in 50.png

2018-01-19T19:25:28Z

Kmilner: