CyberShake Study 17.3

CyberShake Study 16.8 is a computational study to calculate 2 CyberShake hazard models - one with a 1D velocity model, one with a 3D - at 1 Hz in a new region, CyberShake Central California. We will use the GPU implementation of AWP-ODC-SGT, the Graves & Pitarka (2014) rupture variations with 200m spacing and uniform hypocenters, and the UCERF2 ERF. The SGT and post-processing calculations will both be run on both NCSA Blue Waters and OLCF Titan.

Status

Currently we are in the planning stages and hope to begin the study by the end of August, 2016.

Science Goals

The science goals for Study 16.8 are:

• Expand CyberShake to include Central California sites.
• Create CyberShake models using both a Central California 1D velocity model and a 3D model (CCA-06).
• Calculate hazard curves for PG&E pumping stations.

Technical Goals

The technical goals for Study 16.8 are:

• Run end-to-end CyberShake workflows on Titan, including post-processing.
• Show that the database migration improved database performance.

Sites

We will run a total of 503 sites as part of Study 16.8. A KML file of these sites, along with the Central and Southern California boxes, is available here (with names) or here (without names).

We created a Central California CyberShake box, defined here.

We have identified a list of 472 sites which fall within the box and outside of the CyberShake Southern California box. These include:

• 302 sites on a 10 km grid
• 112 CISN stations, decimated so they are at least 5 km apart, and no closer than 2 km from another station
• 28 cities used by the USGS in locating earthquakes
• 4 PG&E pumping stations; one is just outside the box but will be included.
• 6 historic Spanish missions

In addition, we will include 31 sites which overlap with the Southern California box, enabling direct comparison of results.

Velocity Models

We are planning to use 2 velocity models in Study 16.8:

1. CCA-06, a 3D model created via tomographic inversion by En-Jui Lee. This model has a minimum Vs of 900 m/s and no GTL.
2. CCA-1D, a 1D model created by averaging CCA-06 throughout the Central California region.

We will first run the model using the 1D model, then move to the 3D.

Verification

As part of our verification work, we plan to generate hazard curves for 4 sites:

• Bakersfield (-119.018711,35.373292), Wald Vs30 = 206
• Santa Barbara (-119.698189,34.420831), Wald Vs30 = 332
• Parkfield (-120.432800,35.899700), Wald Vs30 = 438.4

Computational and Data Estimates

Computational Time

Since we are using a min Vs=900 m/s, we will use a grid spacing of 175 m, and dt=0.00875 in the SGT simulation (and 0.0875 in the seismogram synthesis).

For computing these estimates, we are using a volume of 420 km x 1160 km x 60 km, or about 5.5 billion grid points, approximately half the size of the Study 15.4 typical volume.

Titan

SGTs (GPU): 750 node-hrs per site x 502 sites = 376,500 node-hours.

Post-processing (CPU): 1400 node-hrs per site x 502 sites = 702,800 node-hours.

Production Checklist

• Complete the database migration outlined in 2016_CyberShake_database_migration.
• Install 1D model in UCVM 15.10.0 on Blue Waters and Titan.
• Decide on 3D velocity model to use.
• Upgrade Condor on shock to v8.4.8.
• Get the Pegasus Dashboard up and running.
• Generate test hazard curves for 4 sites, including an overlapping site between Blue Waters and Titan.
• Confirm test results with science group.
• Determine CyberShake volume for corner points in Central CA region, and if we need to modify the 200 km cutoff.
• Modify submit job on shock to distribute end-to-end workflows between Blue Waters and Titan.
• Add new velocity models into CyberShake database.
• Create XML file describing study for web monitoring tool