CyberShake SGT reference test

This page details a small-scale (single GPU) test of the AWP-ODC-SGT code, with inputs and reference solutions.

Code

The code used for this test is available at https://github.com/SCECcode/cybershake-core/tree/main/AWP-GPU-SGT .

In AWP, the value BLOCK_SIZE_Z, set in a #define at the top of src/pmcl3d_cons.h, must be set to a factor of the number of grid points in the Z dimension. Since this test is 200 grid points deep, set BLOCK_SIZE_Z to 200:

#define BLOCK_SIZE_Z 200

Then compile the code by running 'make' in the src directory. This will generate the executable, pmcl3d, and copy it to the bin directory. You may need to edit the Makefile to use the correct alias for the MPI C compiler on your system.

Input Files

AWP requires three different input files:

• CyberShake_Code_Base#AWP_cordfile AWP cordfile, which contains a list of the grid points for which SGTs are saved.
• CyberShake_Code_Base#AWP_format AWP velocity mesh, which contains the material properties for the region. For this test, the material properties are homogeneous (vp=1500 m/s, vs=750 m/s, rho=2200 kg/m3).
• CyberShake_Code_Base#AWP_source source, which contains the impulse placed at the site of interest. This is a point source.

These input files are available on Summit at /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test . The cordfile is 'awp.SMALL.cordfile', the source is 'SMALL_fx_src', and the velocity mesh is 'awp.SMALL.smoothed.media'.

AWP expects a certain directory structure, and the input files to be staged in a certain way. To create the directory structure, run the 'make_dirs.sh' script in /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test . Then, either copy or create symlinks to the 3 input files in comp_x/input .

Execution

A sample LSF batch script is available at /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test/run_awp.lsf . Edit the 'EXEC_PATH' line to point to the location of your pmcl3d install. When I tested this, it took about 15 minutes to run on a single Summit GPU.

Comparisons

To compare your results to the reference results, I recommend using AWP-GPU-SGT/utils/compare_sgts . cd into the utils directory and run 'make compare_sgts'. The usage for this code is:

./compare_sgts <reference SGT file> <test SGT file> <number of SGT points> <number of timesteps>

So for this test, you'll run:

$>./compare_sgts <> <> 31042 2000

As a reference point, when I ran a larger test on Perlmutter, which has A100 GPUs, I saw an average absolute percent difference of about 0.02%.