CyberShake SGT reference test

This page details a small-scale (single GPU) test of the AWP-ODC-SGT GPU code used in CyberShake, with inputs and reference solutions.

Clone CyberShake GitHub Repository Code

The code used for this test is available at https://github.com/SCECcode/cybershake-core

You can clone the repository this way.

 git clone https://github.com/SCECcode/cybershake-core

After the checkout, the parallel, wave propagation, CUDA-based, GPU code of interest is located in

 <yourdirectory>/cybershake-core/AWP-GPU-SGT/src

Setup Modules on Summit

To compile the code, you'll want to use the gnu and NVIDIA CUDA compilers. On Summit, for standard environments, you can load the required modules with these command:

  module swap xl gcc
  module load cuda 

These will add the required compilers to your environment. Since your module environment may be different, here is a module list that shows a module environment on Summit that builds the AWP-GPU-SGT code correctly. All of these may not be required, but this shows a working module environment for the code.

[login2.summit src]$ module list
Currently Loaded Modules:
  1) lsf-tools/2.0                4) xalt/1.2.1   7) spectrum-mpi/10.4.0.3-20210112  10) cuda/11.0.3
  2) hsi/5.0.2.p5                 5) DefApps      8) nsight-compute/2021.2.1
  3) darshan-runtime/3.3.0-lite   6) gcc/9.1.0    9) nsight-systems/2021.3.1.54

Configure AWP-ODC-SGT

Move into the AWP-GPU-SGT source directory, which is located at:

cd <yourdirectory>/cybershake-core/AWP-GPU-SGT/src

You will need to edit a file in this directory to configure for the small-sized test problem we have posted.

In AWP, the value BLOCK_SIZE_Z, set in a #define at the top of src/pmcl3d_cons.h, must be set to a factor of the number of grid points in the Z dimension. Since this test is 200 grid points deep, set BLOCK_SIZE_Z to 200:

#define BLOCK_SIZE_Z 200

Create a ../bin directory

The make file expects a ../bin directory to exist, and it will try to install the executable there once it is built.

mkdir <yourdirectory>/cybershake-core/AWP-GPU-SGT/bin

Compile the executable - pmcl3d

Then compile the code by running 'make' in the src directory. This will generate the executable, pmcl3d, and copy it to the bin directory.

The make may generate some warnings, but should build the executable called pmcl3d.

You may need to edit the Makefile to use the correct alias for the MPI C compiler on your system.

This make file also defines a "clean" target, so you can run "make clean" and re-build the executable if needed.

Input Files

AWP requires three different input files. These links to a SCEC wiki give detailed descriptions of each of these files:

• AWP cordfile, which contains a list of the grid points for which SGTs are saved.
• AWP velocity mesh, which contains the material properties for the region. For this test, the material properties are homogeneous (vp=1500 m/s, vs=750 m/s, rho=2200 kg/m3).
• Impulse source, which contains the impulse placed at the site of interest. This is a point source.

These input files are available on Summit at

 /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test

• cordfile is 'awp.SMALL.cordfile'
• source is 'SMALL_fx_src'
• velocity mesh is 'awp.SMALL.smoothed.media'

Creating the Runtime Directory Structure

AWP expects a certain directory structure, and the input files to be staged in a certain way. To create the directory structure, run the 'make_dirs.sh' script in /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test . Then, either copy or create symlinks to the 3 input files in comp_x/input .

Execution

A sample LSF batch script is available at /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test/run_awp.lsf . Edit the 'EXEC_PATH' line to point to the location of your pmcl3d install. When I tested this, it took about 15 minutes to run on a single Summit GPU.

When run successfully, you should see no error messages in stderr, and the job should create the file comp_x/output_sgt/awp-strain-SMALL-fx . It should be 1490016000 bytes (~1.4 GB) in size.

Reference Results

Reference results are available on Summit at /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test/awp-strain-SMALL-fx-reference . Details about the file format are available at CyberShake_Code_Base#AWP_SGT.

Comparisons

To compare your results to the reference results, I recommend using AWP-GPU-SGT/utils/compare_sgts . cd into the utils directory and run 'make compare_sgts'. The usage for this code is:

./compare_sgts <reference SGT file> <test SGT file> <number of SGT points> <number of timesteps>

So for this test, you'll run:

$>./compare_sgts awp-strain-SMALL-fx-reference comp_x/output_sgt/awp-strain-SMALL-fx 31042 2000 

It's hard to say what constitutes 'good enough', but moving between systems I usually see average absolute percent differences of a few hundredths of a percent.