CyberShake SGT reference test
This page details a small-scale (single GPU) test of the AWP-ODC-SGT GPU code used in CyberShake, with inputs and reference solutions.
Contents
Clone CyberShake GitHub Repository Code
The code used for this test is available at https://github.com/SCECcode/cybershake-core
You can clone the repository this way.
git clone https://github.com/SCECcode/cybershake-core
This clones the full CyberShake codebase, which includes several applications, one of which is the GPU code that we want to work with at this time. After the checkout, the parallel, wave propagation, CUDA-based, GPU code of interest is located in
<yourdirectory>/cybershake-core/AWP-GPU-SGT/src
Setup Modules on Summit
To compile the code, you'll want to use the gnu and NVIDIA CUDA compilers. On Summit, for standard environments, you can load the required modules with these command:
module swap xl gcc module load cuda
These will add the required compilers to your environment. Since your module environment may be different, here is a module list that shows a module environment on Summit that builds the AWP-GPU-SGT code correctly. All of these may not be required, but this shows a working module environment for the code.
[login2.summit src]$ module list Currently Loaded Modules: 1) lsf-tools/2.0 4) xalt/1.2.1 7) spectrum-mpi/10.4.0.3-20210112 10) cuda/11.0.3 2) hsi/5.0.2.p5 5) DefApps 8) nsight-compute/2021.2.1 3) darshan-runtime/3.3.0-lite 6) gcc/9.1.0 9) nsight-systems/2021.3.1.54
Configure AWP-ODC-SGT
Move into the AWP-GPU-SGT source directory, which is located at:
cd <yourdirectory>/cybershake-core/AWP-GPU-SGT/src
You will need to edit a file in this directory to configure for the small-sized test problem we have posted.
In AWP, the value BLOCK_SIZE_Z, set in a #define at the top of src/pmcl3d_cons.h, must be set to a factor of the number of grid points in the Z dimension. Since this test is 200 grid points deep, set BLOCK_SIZE_Z to 200:
#define BLOCK_SIZE_Z 200
Compile the executable - pmcl3d
Then compile the code by running 'make -f Makefile-summit' in the src directory. This will generate the executable, pmcl3d, and copy it to the bin directory.
The make may generate some warnings, but should build the executable called pmcl3d.
If not running on Summit, you may need to edit the Makefile to use the correct alias MPI C compiler and point to the NVIDIA CUDA compiler on your system.
This make file also defines a "clean" target, so you can run "make clean" and re-build the executable if needed.
Create Run Directory
The input and output files for AWP-ODC-GPU can be quite large, so we recommend, on Summit, that users create a runtime directory on a directory that is visible from the compute nodes and that has a large disk quota. For this testing, we recommend creating a directory on proj-shared file system, and copying the required inputs files and outputs there.
mkdir /gpfs/alpine/geo112/proj-shared/scallag/testawp
Copy Test Files to runtime
We has posted the required input files and example results and other scripts into a directory on Summit. Copy them to your runtime directory. These input files are available on Summit at
/gpfs/alpine/geo112/world-shared/callag/SGT_SMALL_reference_test
cd /gpfs/alpine/geo112/proj-shared/scallag/testawp
cp /gpfs/alpine/geo112/world-shared/callag/SGT_SMALL_reference_test/* .
This will copy the following files into your runtime directory:
752K SMALL_fx_src 1.4G awp-strain-SMALL-fx-reference 363K awp.SMALL.cordfile 770M awp.SMALL.smoothed.media 669 awp_x.lsf 90 make_dirs.sh
Input Files
AWP requires three different input files. The links below to a SCEC wiki give detailed descriptions of each of these files:
- AWP cordfile, which contains a list of the grid points for which SGTs are saved.
- AWP velocity mesh, which contains the material properties for the region. For this test, the material properties are homogeneous (vp=1500 m/s, vs=750 m/s, rho=2200 kg/m3).
- Impulse source, which contains the impulse placed at the site of interest. This is a point source.
- cordfile is 'awp.SMALL.cordfile'
- source is 'SMALL_fx_src'
- velocity mesh is 'awp.SMALL.smoothed.media
Creating the Runtime Directory Structure
AWP expects a certain directory structure, and the input files to be staged in a certain way. To create the directory structure, run the 'make_dirs.sh' script that you copied into your runtime directory.
./make_dirs.sh
Then, either copy or create symlinks to the 3 input files in comp_x/input. The command below will create symlinks in the correct directory for the test problem.
ln -s ../../awp.SMALL.cordfile comp_x/input/awp.SMALL.cordfile ln -s ../../SMALL_fx_src comp_x/input/SMALL_fx_src ln -s ../../awp.SMALL.smoothed.media comp_x/input/awp.SMALL.smoothed.media
Execution
A sample LSF batch script is available in the input file collection, called awp.lsf
Edit the 'EXEC_PATH' line to point to the location of your pmcl3d install.
EXEC_PATH=/ccs/home/scallag/testcs/cybershake-core/AWP-GPU-SGT/bin
And Edit the project allocation to an active allocation
#BSUB -P GEO112
Submit the job to the debug queue:
bsub awp_x.lsf
Check the job status:
bjobs
When I tested this, it took about 15 minutes to run on a single Summit GPU.
When run successfully, you should see no error messages in stderr, and the job should create the file comp_x/output_sgt/awp-strain-SMALL-fx . It should be 1490016000 bytes (~1.4 GB) in size.
Reference Results
Reference results are available in the testcfg files you copied over. The results are called:
/awp-strain-SMALL-fx-reference .
Details about the file format are available at
Comparisons
To compare your results to the reference results, I recommend using AWP-GPU-SGT/utils/compare_sgts . cd into the utils directory and run 'make compare_sgts'. The usage for this code is:
./compare_sgts <reference SGT file> <test SGT file> <number of SGT points> <number of timesteps>
So for this test, you'll run:
$>./compare_sgts awp-strain-SMALL-fx-reference comp_x/output_sgt/awp-strain-SMALL-fx 31042 2000
It's hard to say what constitutes 'good enough', but moving between systems I usually see average absolute percent differences of a few hundredths of a percent.
