CyberShake BBP Verification

From SCECpedia
(Redirected from CyberShake BBP Validation)
Jump to navigationJump to search

This page documents our initial CyberShake Broadband Platform verification work, running the BBP validation events through CyberShake.

Events

The BBP has support for the following events. We plan to run the ones in bold:

  • Alum Rock Mw 5.45
  • Chino Hills Mw 5.39
  • Hector Mine Mw 7.13
  • Landers Mw Mw 7.22*
  • Loma Prieta Mw 6.94
  • North Palm Springs Mw 6.12
  • Northridge Mw 6.73
  • Parkfield Mw 6.0
  • Ridgecrest A Mw 6.47
  • Ridgecrest B Mw 5.4
  • Ridgecrest C Mw 7.08*
  • San Simeon Mw 6.6*
  • Whittier Narrows Mw 5.89

Verification and Validation

Planned process will involve verification and validation stages.

  • Verification: Compare CyberShake broadband seismograms to Broadband Platform seismograms
  • Validation: Compare CyberShake broadband seismograms to observed seismograms

Verification Process

Validation Process

For each of the 5 BBP validation events listed above, go through the following process:

  • Run the BBP validation event with the Graves & Pitarka modules to produce BBP results for comparison.
  • Review the BBP validation data for the event, including station list and seismogram files and validation results from GP method for this event.
  • Identify about 10 stations from the BBP station list that have observed seismograms to run in CyberShake.
  • Modify station list low-pass filter to 50 Hz.
  • Use the Vs30 values from the BBP station list for both the low-frequency and high-frequency site response in CyberShake, for consistency.
  • We will use CVM-S4.26.M01 as the velocity model for CyberShake.
  • Run one realization using the SRF file which is generated by the BBP when run with one realization.
  • Create an ERF in the CyberShake database containing all the validation events, with a unique source and rupture ID for each one.
  • Save the SGTs in case we want to run additional realizations later.
  • To interface with the BBP tools for generating data products, convert the CyberShake seismograms into BBP format seismograms

Data Products

The CyberShake method calculates two component seismograms. Broadband platform supports three component seismograms. We will convert CyberShake 2 components to 3 components by duplicating the N/S component in the vertical component data.

We will produce the following data products to compare CyberShake results with BBP results:

  • ts_process plot, comparing traces and response spectra.
  • RotD50 comparison plots
  • Goodness-of-fit bias plots
  • Map of GoF, by station

Northridge

A KML of the Northridge stations is available here. Selected stations are in orange. The BBP station list file for these stations is available here.

We made the following modifications to the default BBP platform to support better comparison to the BB CS results:

  • Changed the BBP LF and HF seismogram length to 500 sec / 50,000 timesteps
  • Modified the filtering so the merge frequency is 1.0 Hz and phase=0

3D comparisons

Below are comparisons using the CVM-S4.26.M01 model in CyberShake.

Stage Seismogram comparison
Filtered high-frequency acceleration
with site response
SCE bbp v cs hf site filtered cropped.png
Low-frequency velocity
SCE bbp v cs lf cropped.png
Broadband velocity
SCE bbp v cs bb tsprocess cropped.png

BBP LA Basin 1D comparisons

Below are comparisons using the BBP 1D LA Basin model in CyberShake with no interpolation. We also:

  • Modified the CyberShake filtering scheme to match the BBP filtering scheme, which uses a high-pass filter for the HF at around 0.89 Hz and a low-pass filter for the LF at about 1.11 Hz.
  • Modified the CyberShake low-frequency site response to use the high-frequency PGA values, consistent with the BBP approach.

We ran SCE with both the original 1D model, and a modified version (File:Modified nr02-vs500.txt)

Stage BBP vs CS, original 1D BBP vs CS, modified 1D
Low-frequency (velocity)
SCE 1D bbp v cs lf.png
SCE modified1D bbp v cs lf.png
Low-frequency
with site response (velocity)
SCE 1D bbp v cs lf site.png
SCE modified1D bbp v cs lf site.png
Low-frequency
with site response and filtering (velocity)
SCE 1D bbp v cs lf site filtered.png
SCE modified1D bbp v cs lf site filtered.png
High-frequency
with site response and filtering (velocity)
SCE 1D bbp v cs hf.png
SCE modified1D bbp v cs hf.png
Broadband results (velocity)
SCE 1D bbp v cs.png
SCE modified1D bbp v cs.png

Comparable dt

We reran our comparisons with the same dt (0.1) for SCE in CyberShake and the BBP:

Stage BBP vs CS, original 1D
Low-frequency
with site response and filtering (velocity)
SCE 1D dt bbp v cs lf site filtered.png
High-frequency
with site response and filtering (velocity)
SCE 1D dt bbp v cs hf.png
Broadband results (velocity)
SCE 1D dt bbp v cs.png

Other sites

RO3 (5/170 interchange) PEL (Santa Monica/Highland)
Low-frequency (velocity)
RO3 1D bbp v cs lf.png
PEL 1D bbp v cs lf.png
Low-frequency
with site response (velocity)
RO3 1D bbp v cs lf site.png
PEL 1D bbp v cs lf site.png
Low-frequency
with site response and filtering (velocity)
RO3 1D bbp v cs lf site filtered.png
PEL 1D bbp v cs lf site filtered.png
High-frequency
with site response and filtering (velocity)
RO3 1D bbp v cs hf.png
PEL 1D bbp v cs hf.png
Broadband results (velocity)
RO3 1D bbp v cs.png
PEL 1D bbp v cs.png

Halfspace comparisons

Seismograms

Point source. Red = AWP, Black = RWG.

Unfiltered
SCE hspace ptsrc rwg v awp uncorrected.png
Filtered
SCE hspace ptsrc rwg v awp uncorrected filtered.png

Northridge BBP validation event. Red = AWP, Black = RWG.

Unfiltered
SCE hspace nr rwg v awp uncorrected.png
Filtered

SGTs

X-component Y-component
Adjusted mu in AWP
SCE hspace sgts rwg v awp.png
SCE hspace sgts rwg v awp y.png
Unadjusted mu in AWP
SCE hspace sgts rwg v awp uncorrmu.png
File:SCE hspace sgts rwg v awp uncorrmu y.png
600px

Uncorrected mu

Modified 1D velocity model

RWG and AWP SGTs, using SGT headers with uncorrected mu values

Filtered seismograms (red = AWP, black = RWG)

Point source
SCE 1D ptsrc rwg v awp uncorrected filt.png
Northridge
SCE 1D nr rwg v awp uncorrected filt.png

SGTs (red = AWP, black = RWG)

X Y
SCE 1D sgt rwg v awp.png
SCE 1D sgt rwg v awp y.png

1Hz source plots

To reduce ringing effects, we started using an impulse low-pass filtered at 1 Hz.

All the plots below are of SGTs.

Slow halfspace

The plots below were calculated using a halfspace model with Vp=1500, Vs=750, rho=2200.

Qs=37.5, Qp=75.0

X Y
SCE slow halfspace loQ sgt rwg v awp x.png
SCE slow halfspace loQ sgt rwg v awp y.png

Qs=10000, Qp=20000

X Y
SCE slow halfspace hiQ sgt rwg v awp x.png
SCE slow halfspace hiQ sgt rwg v awp y.png

4-layered model

Next we ran with a 4-layered model: 

Thickness(km)   Vp      Vs      rho     
0.2500          2.4000  1.0000  2.2500  
0.7000          3.7000  1.8000  2.4500  
1.5000          4.7500  2.4000  2.6000 
2.5000          5.4000  3.0000  2.6500 
999.00          6.2000  3.5000  2.7200

Qs=10000, Qp=20000

X Y
SCE 4lay hiQ sgt rwg v awp x.png
SCE 4lay hiQ sgt rwg v awp y.png

Qs=0.05Vs, Qp=2Qs

X Y
SCE 4lay normalQ sgt rwg v awp x.png
SCE 4lay normalQ sgt rwg v awp y.png

Qs=0.05Vs, Qp=2Qs

Unadjusted mu and lambda used for stress->strain conversion. This was accomplished by removing adjustment factors in the SGT kernel.

Depth X Y
Shallow (5 km)
SCE 4lay normalQ unadj sgt rwg v awp x.png
SCE 4lay normalQ unadj sgt rwg v awp y.png
Deep (15 km)
SCE 4lay normalQ unadj sgt rwg v awp x deep.png
SCE 4lay normalQ unadj sgt rwg v awp y deep.png

12-layered model

   Vp      Vs      rho         Depth to layer bottom (km)
  2.4000   1.0000   2.2500     0.250001
  3.1000   1.4000   2.3000     0.450001
  3.3495   1.5560   2.3750     0.550001
  3.7000   1.8000   2.4500     0.950001
  3.9190   1.9285   2.5125     1.050001
  4.2000   2.1000   2.5750     1.450001
  4.4483   2.2343   2.5875     1.550001
  4.7500   2.4000   2.6000     2.450001
  4.9150   2.5565   2.6100     2.550001
  5.1000   2.7500   2.6200     3.450001
  5.2426   2.8661   2.6350     3.550001
  5.4000   3.0000   2.6500     4.950001
  6.2000   3.5000   2.7200     9999.999999

Qs=0.05Vs, Qp=2Qs

Unadjusted mu and lambda used for stress->strain conversion.

Depth X Y
Shallow (5 km)
SCE 12lay sgt rwg v awp shallow x.png
SCE 12lay sgt rwg v awp shallow y.png
Deep (15 km)
SCE 12lay sgt rwg v awp x deep.png
SCE 12lay sgt rwg v awp y deep.png

14-layered model

   Vp      Vs      rho      Depth to layer bottom (km)
  1.6447   0.5378   2.0600     0.050001
  1.8893   0.7115   2.1500     0.150001
  2.3653   0.9766   2.2500     0.250001
  3.1000   1.4000   2.3000     0.450001
  3.3495   1.5560   2.3750     0.550001
  3.7000   1.8000   2.4500     0.950001
  3.9190   1.9285   2.5125     1.050001
  4.2000   2.1000   2.5750     1.450001
  4.4483   2.2343   2.5875     1.550001
  4.7500   2.4000   2.6000     2.450001
  4.9150   2.5565   2.6100     2.550001
  5.1000   2.7500   2.6200     3.450001
  5.2426   2.8661   2.6350     3.550001
  5.4000   3.0000   2.6500     4.950001
  6.2000   3.5000   2.7200  9999.999999


Qs=0.05Vs, Qp=2Qs

Unadjusted mu and lambda used for stress->strain conversion.

Depth X Y
Shallow (5 km)
SCE 14lay sgt rwg v awp shallow x.png
SCE 14lay sgt rwg v awp shallow y.png
Deep (15 km)
SCE 14lay sgt rwg v awp x deep.png
SCE 14lay sgt rwg v awp y deep.png

Forward/Reverse Tests

To narrow down potential issues further, we performed a series of forward/reverse tests. In these, we place a force at one location and observe the velocities at another, then place a force at the 2nd location and observe the velocities at the first. As long as the correct components are being compared, the resulting seismograms should be very similar.

Results from Rob's 4th-order code (14-layer model), using the points at 5 km and the surface, and 15 km depth and the surface.

Depth Results
5 km
SCE shal-14lay rwg recp.png
15 km
SCE deep-14lay rwg recp.png

Rob repeated the experiment with 2nd-order operators and got agreement to floating-point error.

Using the AWP-CS code, we got the following results. We averaged the Z=1 and Z=2 layers to correctly get the free surface velocities. For some reason, one of the seismograms needs to be inverted in order to match correctly.

Model Results for 5 km/source pair
14 layer
SCE fwd shallow 14lay avg invert.png
12 layer
SCE fwd shallow 12lay avg invert.png
|
4 layer
SCE fwd shallow 4lay avg invert.png
halfspace
SCE fwd shallow halfspace avg invert.png

Velocity source insertion

We determined that the offset is due to the way the AWP staggered grid is set up. The normal stresses are in the center of the grid cell, but the X velocity is in the center of the side, in the +x direction. So inserting a source at location A and recording velocities at location B is actually recording velocities at B+.5, and doing the reverse tests means velocities are measured at A+.5. So in the first test, the distance traveled is (B+.5)-A = (B-A)+.5, and in the second it's B-(A+.5) = (B-A)-.5, a difference of 1 grid point. If the force is inserted by modifying velocities,

To illustrate this, the plots below show the same test with a stress insertion and a velocity insertion. Both points have the same Y and Z indices (depth=1km), only the X is varied.

Stress insertion
Awp halfspace X stress insertion.png
Velocity insertion
Awp halfspace X velocity insertion.png

Here's a plot of the relative differences:

Awp halfspace X velocity insertion diff.png

We also tried increasing Q to 1e10:

Awp halfspace X velocity insertion hugeQ diff.png


Since the above results looked good, we tried moving one of the points to the surface:

Awp halfspace X velocity insertion surf.png
Awp halfspace X velocity insertion surf diff.png

The differences in the late arrivals are ~3%.

We repeated these velocity insertion tests for the 4, 12, and 14-layer models, with one point at the surface and one point at depth.

Model X results Y results
4-layer
Awp 4layer X velocity insertion diff.png
Awp 4layer Y velocity insertion diff.png
12-layer
Awp 12layer X velocity insertion diff.png
Awp 12layer Y velocity insertion diff.png
14-layer
Awp 14layer X velocity insertion diff.png
Awp 14layer Y velocity insertion diff.png

I ran a model at 33m resolution. Here are the difference plots, and comparison to the 100m model, for a halfspace.

Awp halfspace X 33m diff.png
Awp 14layer X 33m v 100m.png


Kim made some updates to the symmetries in the kernel. I reran the 14-layer tests with the new kernel.

Awp 14layer X newkernel diff.png

We recomputed results using 100m and 33m spacing, and compared them to the FK results for the 14-layer model. We chose the AWP points (432, 365, 1) and (317, 453, 52) in the 100m run and (1294, 1093, 1) and (948, 1362, 154) in the 33m run.

100m self-reciprocity, X component
Awp 14layer X 100m diff.png
33m self-reciprocity, X component
Awp 14layer X 33m diff.png
100m vs 33m
Awp 14layer X 100m v 33m.png
33m vs FK
Awp 14layer X 33m v FK.png

SGT Tests

Using the 14-layer model, we ran tests to generate SGTs at 100m and 33.333m grid spacing, using both velocity and stress insertion

AWP 100m vs RWG 100m, X component
Awp v rwg 14lay 100m X velocity sgts.png
AWP 33m vs AWP 100m, X component
Awp 14lay 33m v 100m X velocity sgts.png
AWP 33m vs RWG 100m, X component
Awp 33m v rwg 100m 14lay X velocity sgts.png
AWP 100m velocity insertion vs AWP 100m stress insertion, X component
Awp velocity v stress 100m X sgts.png

BBP Comparisons

For these comparisons, we used the 14-layer model in the BBP, and a 24-layer pre-processed version of this model better for finite difference codes.

Site SCE

BBP vs AWP-CS
SCE 1D bbp v awpcs lf.png
BBP vs AWP-OOR (modified source insertion)
SCE 1D bbp v awpoor lf.png
BBP vs RWG reciprocity
SCE 1D bbp v rwg lf.png

Here's a comparison of a 33m run with AWP-CS, a 100m, and the BBP.

SCE 1D awpcs33m awpcs100m bbp lf.png

Other sites, low-frequency

RO3 (LF only)
RO3 1D awp-cs awp-oor bbp lf.png
PEL (LF only)
PEL 1D awp-cs awp-oor bbp lf.png


Broadband comparisons

Site Broadband seismogram comparisons
SCE
SCE 1D awpcs bbp bb.png
JGB
JGB 1D awpcs bbp bb.png
LDM
LDM 1D awpcs bbp bb.png
SPV
SPV 1D awpcs bbp bb.png
NYA
NYA 1D awpcs bbp bb.png
LBR (~45 km)
LBR 1D awpcs bbp bb.png
FEA (~80 km)
FEA 1D awpcs bbp bb.png
SBG (~105 km)
SBG 1D awpcs bbp bb.png

Goodness of fit plots

Results for 10 stations, 1 realization

Gof Northridge 1real 10stat rotd50.png
Gof dist-log Northridge 1real 10stat rotd50.png
Gof map Northridge 1real 10stat rotd50.png

Results for modified interpolation in jbsim3d

Gof modified jbsim3d Northridge 1real 10stat rotd50.png
Gof modified jbsim3d dist-log Northridge 1real 10stat rotd50.png
Gof modified jbsim3d map Northridge 1real 10stat rotd50.png

GoF for multiple realizations

For clarity, here are the steps involved in producing the below plots.

  1. Generate the 10 realizations by using a random number generator to generate 9 numbers, and create copies of the src file which is used in the BBP validation event, changing only the seed.
  2. Produce BBP results for each of the 10 sites x 10 realizations. Note that all must be run individually so reproducibility with the random seed isn't an issue.
    1. Run the BBP workflow generator for 1 site and 1 realization to generate an XML file. This workflow includes SRF generation and site response.
    2. Using the XML file, replace the path to the station name and the path to the src file.
    3. Create batch submission scripts which call all 100 BBP workflows.
  3. Create validation events for each realization.
    1. Copy over the Northridge validation event data that ships with the BBP.
    2. Update the data. This includes changing the event name, updating the src file, updating the stl file to filter at the correct frequencies, and stage in the output acceleration data from the BBP runs for the validation event to use as an 'observation'.
  4. Produce CyberShake results for each of the 10 sites.
    1. Create a new source in the ERF, in which each realization is a separate rupture of that source.
    2. Extract the rupture geometry from one of the realizations, to obtain the list of points the SGTs should be saved for.
    3. Generate a CyberShake volume oriented N/S/E/W which contains the Northridge rupture surface, the site of interest, is 64 km deep, and uses 30 km of padding.
    4. Generate a velocity mesh using the 24-layer model provided by Rob, which is a post-processed version of the LA Basin 14-layer model included with the BBP.
    5. Run the original version of the AWP code ('AWP-CS'), using velocity insertion of a source low-pass filtered at 2 Hz, for 30 seconds (unless it's a distance site, in which case we run longer)
    6. Run the post-processing code for the 10 realizations.
    7. Run the broadband steps (high-frequency synthesis, low-frequency site response, and merge) for each of the 10 realizations.
  5. Perform goodness-of-fit, using CyberShake as the simulation and BBP as the reference.
    1. For each realization:
      1. Create a temporary directory for the CyberShake results.
      2. Stage in the CyberShake results for all 10 sites
      3. Use BBP helper scripts to create a simulation ID and stage the CyberShake results to that directory.
      4. Run the BBP to generate an XML file which will perform GoF (details at New_BBP_Validation_Package)
      5. Run using the XML file to generate the GoF comparisons and plots.
Realization GoF plot Ratio by distance
1
Gof modified jbsim3d Northridge 1real 10stat rotd50.png
Gof modified jbsim3d dist-log Northridge 1real 10stat rotd50.png
2
Gof Northridge 2real 10stat rotd50.png
Gof dist-log Northridge 2real 10stat rotd50.png
3
Gof Northridge 3real 10stat rotd50.png
Gof dist-log Northridge 3real 10stat rotd50.png
4
Gof Northridge 4real 10stat rotd50.png
Gof dist-log Northridge 4real 10stat rotd50.png
5
Gof Northridge 5real 10stat rotd50.png
Gof dist-log Northridge 5real 10stat rotd50.png
6
Gof Northridge 6real 10stat rotd50.png
Gof dist-log Northridge 6real 10stat rotd50.png
7
Gof Northridge 7real 10stat rotd50.png
Gof dist-log Northridge 7real 10stat rotd50.png
8
Gof Northridge 8real 10stat rotd50.png
Gof dist-log Northridge 8real 10stat rotd50.png
9
Gof Northridge 9real 10stat rotd50.png
Gof dist-log Northridge 9real 10stat rotd50.png
10
Gof Northridge 10real 10stat rotd50.png
Gof dist-log Northridge 10real 10stat rotd50.png

Combined goodness-of-fit

Gof Northridge aggregate 10real 10stat rotd50.png

33m tests

We modified the grid spacing from 100m to 33.333m to see what impact it has on the BBP/CS comparisons. To simplify the comparisons, results here are for 1 Northridge realization, realization 6.

Site 100m 33m
SCE
SCE 1D nrR6 100m bbp v cs.png
SCE 1D nrR6 33m bbp v cs.png
SBG
SBG 1D nrR6 100m bbp v cs.png
SBG 1D nrR6 33m bbp v cs.png

Expanded buffer layer tests

When calculating the size of the CyberShake volume, we construct a volume large enough to contain the site and all the fault surfaces, then pad it on all sides by 30 km. We modified the buffer zone to be larger, 50 km.

Site 30 km buffer zone 50 km buffer zone
SCE
SCE 1D nrR6 30km bbp v cs.png
SCE 1D nrR6 50km bbp v cs.png
SBG
SBG 1D nrR6 30km bbp v cs.png
SBG 1D nrR6 50km bbp v cs.png

We then performed additional tests, deepening the volume to 80 km, increasing the padding to 60 km, and also increasing the number of grid points in the boundary condition sponge zone from 50 to 80. Below are comparisons with the original results, including GoF for site SBG.

All plots are for Northridge realization 6.

Site (spacing) 64 km deep, 30 km padding, 50 point sponge zone GoF for original configuration 80 km deep, 60 km padding, 80 point sponge zone GoF for modified configuration 80 km deep, 60 km padding, 240 point sponge zone
SCE (100m)
SCE 1D nrR6 bbp v 100m cs.png
SCE 1D nrR6 bbp v 100m sponge cs.png
SCE 1D nrR6 bbp v 33m cs 240sponge.png
SCE (33m)
SCE 1D nrR6 bbp v 33m cs.png
SCE 1D nrR6 bbp v 33m sponge cs.png
SBG (100m)
SBG 1D nrR6 bbp v 100m cs.png
Gof SBG 1D nrR6 100m cs.png
SBG 1D nrR6 bbp v 100m cs sponge.png
Gof SBG 1D nrR6 100m cs sponge.png
SBG (33m)
SBG 1D nrR6 bbp v 33m cs.png
Gof SBG 1D nrR6 33m cs.png
SBG 1D nrR6 bbp v 33m cs sponge.png
Gof SBG 1D nrR6 33m cs sponge.png

We tried SBG for both 100m and 33m, saving the SGTs at the same dt as the BBP (0.1s).

SBG (100m), sim dt=0.005, SGT dt=0.1
SBG 1D nrR6 bbp v 33m dt0.1.png
SBG (33m), sim dt=0.001, SGT dt=0.1
SBG 1D nrR6 bbp v 100m dt0.1.png

Northridge, 38 stations

We ran CyberShake and the BBP for 10 realizations for the 38 distinct stations listed here. Note that 2006-PAC and 2007-PUL are colocated, so we only ran one. Below are goodness-of-fit and scatterplot GOF results.

Realization GoF plot GoF scatterplot
2
Gof Northridge 2real 38stat rotd50.png
Gof Northridge 2real 38stat scatter.png
3
Gof Northridge 3real 38stat rotd50.png
Gof Northridge 3real 38stat scatter.png
4
Gof Northridge 4real 38stat rotd50.png
Gof Northridge 4real 38stat scatter.png
5
Gof Northridge 5real 38stat rotd50.png
Gof Northridge 5real 38stat scatter.png
6
Gof Northridge 6real 38stat rotd50.png
Gof Northridge 6real 38stat scatter.png
7
Gof Northridge 7real 38stat rotd50.png
Gof Northridge 7real 38stat scatter.png
8
Gof Northridge 8real 38stat rotd50.png
Gof Northridge 8real 38stat scatter.png
9
Gof Northridge 9real 38stat rotd50.png
Gof Northridge 9real 38stat scatter.png
10
Gof Northridge 10real 38stat rotd50.png
Gof Northridge 10real 38stat scatter.png

Related Entries

Retrieved from "https://strike.scec.org/scecwiki/index.php?title=CyberShake_BBP_Verification&oldid=27927"