NorCal CyberShake SW4 Mesh

• Create CyberShake Mesh for NorCal in 3 files (80m)
  • Tile with background

Pre-study

goal: to validate the sw4 plot of sfcvm native hdf5 datafile via its gmg reader shows topography features at surface and yet

     accessing UCVM's sfcvm is showing a chopped plot at Z=0

Frontier setup

modules :

module load cray-python module unload PrgEnv-cray module load PrgEnv-gnu gcc module load libtool/2.4.6module load Core/24.07 module load openblas/0.3.26

building UCVM

#!/bin/bash
#
hn=`hostname -d`
ppwd=`pwd`

export MY_TOP=$ppwd/scratch
export TOP_UCVM_TARGET=$MY_TOP/TARGET_UCVM_SFCVM

export UCVM_SRC_PATH=$TOP_UCVM_TARGET/UCVM
export UCVM_INSTALL_PATH=$TOP_UCVM_TARGET/ucvm_install

rm -rf $TOP_UCVM_TARGET
mkdir $TOP_UCVM_TARGET

cd $TOP_UCVM_TARGET
git clone https://github.com/SCECcode/ucvm.git -b withSFCVM UCVM

cd $UCVM_SRC_PATH/largefiles
./get_largefiles.py -m sfcvm,cca,cvmsi

cd $UCVM_SRC_PATH/largefiles; ./stage_largefiles.py

cd $UCVM_SRC_PATH
./ucvm_setup.py -d -a -p $UCVM_INSTALL_PATH &> ucvm_setup_install.log

echo "..EXITING.."
exit

SPECIAL NOTE 1:

on Frontier, the first time ucvm_setup.py call seems to fail when configure is called via script. If rerun by hand, It actually goes through.

         1) extract the configure command from config.log and run it by hand
         2) at UCVM top level, call 

make clean; make; make install

         3) make sure ucvm_env.sh is setup properly

./ucvm_setup.py -a -d -r -p $UCVM_INSTALL_PATH

         4) source $UCVM_INSTALL_PATH/conf/uvcm_env.sh

SPECIAL NOTE 2:

for sw4, hdf5 needs to be enabled with parallel option,

           1) cd $UCVM_SRC_PATH/work/lib/hdf5
           2) extract the configure command from config.log, and rerun it with --enable-parallel 
           3) and then call 

make clean; make; make install;

building sw4

git clone https://github.com/geodynamics/sw4.git sw4-clone

git clone https://github.com/geodynamics/sw4.git -b ucvm_cmd sw4-ucvm_cmd

building sw4-clone,

add this file, config.inc, under configs directory

FC = gfortran
CXX = mpicxx

proj=yes
hdf5=yes
SW4ROOT = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/proj
SQL3ROOT = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/sqlite
HDF5ROOT = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/hdf5

EXTRA_LINK_FLAGS = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/openssl/lib/libcrypto.so.3 /usr/lib64/libopenblas.so.0  -L/opt/cray/pe/gcc/12.2.0/snos/lib64 -lgfortran 

top level Makefile needs to be tweaked alittle..

diff --git a/Makefile b/Makefile
index 171d1c5a..ca1ef313 100644
--- a/Makefile
+++ b/Makefile
@@ -78,6 +78,8 @@ SW4LIB64  = $(SW4ROOT)/lib64
 #Default, override with configs/make.name. Preferably, FFTW is installed under SW4ROOT
 FFTWHOME  = $(SW4ROOT)
 
+SQL3INC    = $(SQL3ROOT)/include
+SQL3LIB    = $(SQL3ROOT)/lib
 
 emptystring := ""
 foundincfile := $(emptystring)
@@ -174,8 +176,8 @@ ifdef EXTRA_FORT_FLAGS
 endif
 
 ifeq ($(proj),yes)
-   CXXFLAGS += -DENABLE_PROJ -I$(SW4INC)
-   linklibs += -L$(SW4LIB) -L$(SW4LIB64) -lproj -lsqlite3 -Wl,-rpath,$(SW4LIB) -Wl,-rpath,$(SW4LIB64)
+   CXXFLAGS += -DENABLE_PROJ -I$(SW4INC) -I$(SQL3INC)
+   linklibs += -L$(SW4LIB) -L$(SW4LIB64) -lproj -L$(SQL3LIB) -lsqlite3 -Wl,-rpath,$(SW4LIB) -Wl,-rpath,$(SW4LIB64)
 endif
 

compile sw4 by

make

making sfcvm mesh in sfile format using sw4

building sw4-ucvm_clone is just the same as sw4-stock but it is alittle ad-hoc and so just assume it is done and the sfcvm_mesh.sfile is here..

ccs/home/mei/proj-geo156/ucvm/sw4/sfcvm.output/sfcvm_mesh.sfile

the sfcvm.txt used for generate this sfcvm_mesh.sfile,

# SW4 SCRIPT

fileio path=./sfcvm.output pfs=1 verbose=1 printcycle=10 nwriters=16

time steps=10

# GRID SIZE AND REFINEMENT
grid x=280000 y=115000 z=30000 h=100.0 lat=39.111563 lon=-122.569504 az=143.638 proj=tmerc datum=NAD83 lon_p=-123.0 lat_p=35.0 scale=0.
9996

supergrid gp=30
time t=0.1

globalmaterial vsmin=400 vpmin=800
attenuation phasefreq=1.0 nmech=3 maxfreq=10.0

ucvm filename=sfcvm.h100

# POINT SOURCE
source depth=8320 lat=36.86733 lon=-121.6105 m0=2.515e15 dip=80 strike=138 rake=-177 t0=1.0 type=Dirac

# Station
rec lat=37.52886 lon=-122.25522 depth=0 file=1002_37.52886_-122.25522 variables=velocity nsew=1 usgsformat=1 sacformat=0

sfileoutput file=sfcvm.mesh

and the slurm script used is,

#!/bin/bash

#SBATCH -J sfcvm_sw4          # Job name
#SBATCH -o sfcvm_sw4.oo       # Name of stdout output file
#SBATCH -e sfcvm_sw4.ee       # Name of stderr error file
#SBATCH -N 5             # Total # of nodes 
#SBATCH -n 100
#SBATCH -t 2:00:00           # Run time (hh:mm:ss)
#SBATCH --mail-user=mei@usc.edu
#SBATCH --mail-type=all
#SBATCH -p batch
#SBATCH -A geo156
#SBATCH --open-mode=truncate

source /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/conf/ucvm_env.sh

srun -N5 -n100 /ccs/home/mei/scratch/sw4-ucvm_cmd/optimize_mp/sw4 sfcvm.txt

making image slice from sfcvm_mesh.sfile

the input file ,sw4_sfcvm.sw4input, needs to update the path links,

# SW4 SCRIPT  --  to process sfcvm.mesh.sfile

fileio path=output pfs=1 verbose=1 printcycle=10 nwriters=16

# GRID SIZE AND REFINEMENT
grid x=280000 y=115000 z=30000 h=100.0 lat=39.111563 lon=-122.569504 az=143.638 proj=tmerc datum=NAD83 lon_p=-123.0 lat_p=35.0 scale=0.
9996

#grid x=20000 y=10000 z=25000 h=200 lat=38.070951 lon=-122.253136 az=143.638 proj=tmerc datum=NAD83 lon_p=-123.0 lat_p=35.0 scale=0.999
6

refinement zmax=3600
#refinement zmax=1500
#refinement zmax=400

supergrid gp=1
time steps=1 

globalmaterial vsmin=250 vpmin=500
attenuation phasefreq=1.0 nmech=3 maxfreq=10.0

sfile filename=sfcvm.mesh.sfile directory=/ccs/home/mei/scratch/sw4-ucvm_cmd/ucvm/sfcvm.output

#prefilter fc2=2.5 type=lowpass passes=2 order=2

# POINT SOURCE
source depth=12300 lat=37.85517 lon=-122.25683 m0=4.563e15 dip=80 strike=145 rake=180 t0=1.0 type=Dirac

## generate image at z depth

image mode=s z=0 file=image cycle=0
#image mode=s z=100 file=image cycle=0
#image mode=s z=500 file=image cycle=0

# Stations
rec lat=37.59669 lon=-121.88094 depth=0 file=1688_37.59669_-121.88094 variables=velocity nsew=1 usgsformat=1 sacformat=0

and the slurm script to run this

#!/bin/bash
#SBATCH -J run_sw4_sfcvm          # Job name
#SBATCH -o run_sw4_sfcvm.oo       # Name of stdout output file
#SBATCH -e run_sw4_sfcvm.ee       # Name of stderr error file
#SBATCH -N 5             # Total # of nodes 
#SBATCH -n 100
#SBATCH --mem 0 # Set to unlimited memory
#SBATCH -t 2:00:00           # Run time (hh:mm:ss)
#SBATCH --mail-user=mei@usc.edu
#SBATCH --mail-type=all
#SBATCH -p batch
#SBATCH -A geo156
#SBATCH --open-mode=truncate
#SBATCH --export=NONE 

source /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/conf/ucvm_env.sh

srun -N 5 -n 100 /ccs/home/mei/scratch/sw4-stock/optimize_mp/sw4 ./sw4_sfcvm.sw4input

2 images files are created in 'output' directory:

$ ls output
1688_37.59669_-121.88094.txt  image.cycle=00.z=0.s.sw4img  image.cycle=0.z=0.s.sw4img

using sw4img_printer.py to print,

NOTE : the pySW4 is here, https://github.com/shaharkadmiel/pySW4

Discovery Setup

Currently Loaded Modules:

 1) gcc/11.3.0      3) pmix/3.2.3       5) perl/5.36.0   7) openblas/0.3.21
 2) openmpi/4.1.4   4) libxml2/2.9.13   6) git/2.36.1

UCVM (withSFCVM branch) at /project/scec_608/ucvm_main_sfcvm

sw4 (ucvm_cmd branch) at /project/scec_608/mei/sw4-ucvm_cmd

using config/make.inc

FC = gfortran
CXX = /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/openmpi-4.1.4-4w23jca/bin/mpiCC

proj=yes
hdf5=yes
SW4ROOT = /project/scec_608/mei/ucvm_main_sfcvm/install/lib/proj
SQL3ROOT = /project/scec_608/mei/ucvm_main_sfcvm/install/lib/sqlite
HDF5ROOT = /project/scec_608/mei/ucvm_main_sfcvm/install/lib/hdf5

#EXTRA_LINK_FLAGS = -L/usr/lib64 -llapack -lblas -lgfortran 
EXTRA_LINK_FLAGS = -L/usr/lib64 -lgfortran -L/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/openbla
s-0.3.21-afcoykn/lib -lopenblas

sw4 (main branch with v3.0 tag)

mei@discovery1:/project/scec_608/mei/sw4-stock$ git checkout v3.0
Note: switching to 'v3.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 13e6d431 Merge pull request #195 from geodynamics/developer

Related Entries

• UCVM