CyberShake SGT reference test
This page details a small-scale (single GPU) test of the AWP-ODC-SGT code, with inputs and reference solutions.
Code
The code used for this test is available at https://github.com/SCECcode/cybershake-core/tree/main/AWP-GPU-SGT .
In AWP, the value BLOCK_SIZE_Z, set in a #define at the top of src/pmcl3d_cons.h, must be set to a factor of the number of grid points in the Z dimension. Since this test is 200 grid points deep, set BLOCK_SIZE_Z to 200:
#define BLOCK_SIZE_Z 200
Then compile the code by running 'make' in the src directory. This will generate the executable, pmcl3d, and copy it to the bin directory. You may need to edit the Makefile to use the correct alias for the MPI C compiler on your system.
Input Files
AWP requires three different input files:
- AWP cordfile, which contains a list of the grid points for which SGTs are saved.
- AWP velocity mesh, which contains the material properties for the region. For this test, the material properties are homogeneous (vp=1500 m/s, vs=750 m/s, rho=2200 kg/m3).
- Impulse source, which contains the impulse placed at the site of interest. This is a point source.
These input files are available on Summit at /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test . The cordfile is 'awp.SMALL.cordfile', the source is 'SMALL_fx_src', and the velocity mesh is 'awp.SMALL.smoothed.media'.
AWP expects a certain directory structure, and the input files to be staged in a certain way. To create the directory structure, run the 'make_dirs.sh' script in /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test . Then, either copy or create symlinks to the 3 input files in comp_x/input .
Execution
A sample LSF batch script is available at /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test/run_awp.lsf . Edit the 'EXEC_PATH' line to point to the location of your pmcl3d install. When I tested this, it took about 15 minutes to run on a single Summit GPU.
When run successfully, you should see no error messages in stderr, and the job should create the file comp_x/output_sgt/awp-strain-SMALL-fx . It should be 1490016000 bytes in size.
Reference Results
Reference results are available on Summit at /gpfs/alpine/scratch/callag/geo112/SGT_SMALL_reference_test/awp-strain-SMALL-fx-reference . Details about the file format are available at CyberShake_Code_Base#AWP_SGT.
Comparisons
To compare your results to the reference results, I recommend using AWP-GPU-SGT/utils/compare_sgts . cd into the utils directory and run 'make compare_sgts'. The usage for this code is:
./compare_sgts <reference SGT file> <test SGT file> <number of SGT points> <number of timesteps>
So for this test, you'll run:
$>./compare_sgts comp_x/output_sgt/awp-strain-SMALL-fx 31042 2000
It's hard to say what constitutes 'good enough', but moving between systems I usually see average absolute percent differences of a few hundredths of a percent.
