NorCal CyberShake SW4 Mesh

• Create CyberShake Mesh for NorCal in 3 files (80m)
  • Tile with background

Pre-study

goal: to validate the sw4 plot of sfcvm native hdf5 datafile via its gmg reader shows topography features at surface and yet

     accessing UCVM's sfcvm is showing a chopped plot at Z=0

Frontier setup

modules :

module load cray-python module unload PrgEnv-cray module load PrgEnv-gnu gcc module load libtool/2.4.6module load Core/24.07 module load openblas/0.3.26

build UCVM

#!/bin/bash
#
hn=`hostname -d`
ppwd=`pwd`

export MY_TOP=$ppwd/scratch
export TOP_UCVM_TARGET=$MY_TOP/TARGET_UCVM_SFCVM

export UCVM_SRC_PATH=$TOP_UCVM_TARGET/UCVM
export UCVM_INSTALL_PATH=$TOP_UCVM_TARGET/ucvm_install

rm -rf $TOP_UCVM_TARGET
mkdir $TOP_UCVM_TARGET

cd $TOP_UCVM_TARGET
git clone https://github.com/SCECcode/ucvm.git -b withSFCVM UCVM

cd $UCVM_SRC_PATH/largefiles
./get_largefiles.py -m sfcvm,cca,cvmsi

cd $UCVM_SRC_PATH/largefiles; ./stage_largefiles.py

cd $UCVM_SRC_PATH
./ucvm_setup.py -d -a -p $UCVM_INSTALL_PATH &> ucvm_setup_install.log

echo "..EXITING.."
exit

SPECIAL NOTE 1:

on Frontier, the first time ucvm_setup.py call seems to fail when configure is called via script. If rerun by hand, It actually goes through.

         1) extract the configure command from config.log and run it by hand
         2) at UCVM top level, call 

make clean; make; make install

         3) make sure ucvm_env.sh is setup properly

./ucvm_setup.py -a -d -r -p $UCVM_INSTALL_PATH

         4) source $UCVM_INSTALL_PATH/conf/uvcm_env.sh

SPECIAL NOTE 2:

for sw4, hdf5 needs to be enabled with parallel option,

           1) cd $UCVM_SRC_PATH/work/lib/hdf5
           2) extract the configure command from config.log, and rerun it with --enable-parallel 
           3) and then call 

make clean; make; make install;

build sw4

git clone https://github.com/geodynamics/sw4.git sw4-clone

git clone https://github.com/geodynamics/sw4.git -b ucvm_cmd sw4-ucvm_cmd

building sw4-clone,

add this file, config.inc, under configs directory

FC = gfortran
CXX = mpicxx

proj=yes
hdf5=yes
SW4ROOT = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/proj
SQL3ROOT = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/sqlite
HDF5ROOT = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/hdf5

EXTRA_LINK_FLAGS = /ccs/home/mei/scratch/TARGET_UCVM_SFCVM/ucvm_install/lib/openssl/lib/libcrypto.so.3 /usr/lib64/libopenblas.so.0  -L/opt/cray/pe/gcc/12.2.0/snos/lib64 -lgfortran 

top level Makefile has to be tweaked alittle..

diff --git a/Makefile b/Makefile
index 171d1c5a..ca1ef313 100644
--- a/Makefile
+++ b/Makefile
@@ -78,6 +78,8 @@ SW4LIB64  = $(SW4ROOT)/lib64
 #Default, override with configs/make.name. Preferably, FFTW is installed under SW4ROOT
 FFTWHOME  = $(SW4ROOT)
 
+SQL3INC    = $(SQL3ROOT)/include
+SQL3LIB    = $(SQL3ROOT)/lib
 
 emptystring := ""
 foundincfile := $(emptystring)
@@ -174,8 +176,8 @@ ifdef EXTRA_FORT_FLAGS
 endif
 
 ifeq ($(proj),yes)
-   CXXFLAGS += -DENABLE_PROJ -I$(SW4INC)
-   linklibs += -L$(SW4LIB) -L$(SW4LIB64) -lproj -lsqlite3 -Wl,-rpath,$(SW4LIB) -Wl,-rpath,$(SW4LIB64)
+   CXXFLAGS += -DENABLE_PROJ -I$(SW4INC) -I$(SQL3INC)
+   linklibs += -L$(SW4LIB) -L$(SW4LIB64) -lproj -L$(SQL3LIB) -lsqlite3 -Wl,-rpath,$(SW4LIB) -Wl,-rpath,$(SW4LIB64)
 endif
 
 

Discovery Setup

Currently Loaded Modules:

 1) gcc/11.3.0      3) pmix/3.2.3       5) perl/5.36.0   7) openblas/0.3.21
 2) openmpi/4.1.4   4) libxml2/2.9.13   6) git/2.36.1

UCVM (withSFCVM branch) at /project/scec_608/ucvm_main_sfcvm

sw4 (ucvm_cmd branch) at /project/scec_608/mei/sw4-ucvm_cmd

using config/make.inc

FC = gfortran
CXX = /spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/openmpi-4.1.4-4w23jca/bin/mpiCC

proj=yes
hdf5=yes
SW4ROOT = /project/scec_608/mei/ucvm_main_sfcvm/install/lib/proj
SQL3ROOT = /project/scec_608/mei/ucvm_main_sfcvm/install/lib/sqlite
HDF5ROOT = /project/scec_608/mei/ucvm_main_sfcvm/install/lib/hdf5

#EXTRA_LINK_FLAGS = -L/usr/lib64 -llapack -lblas -lgfortran 
EXTRA_LINK_FLAGS = -L/usr/lib64 -lgfortran -L/spack/2206/apps/linux-centos7-x86_64_v3/gcc-11.3.0/openbla
s-0.3.21-afcoykn/lib -lopenblas

sw4 (main branch with v3.0 tag)

mei@discovery1:/project/scec_608/mei/sw4-stock$ git checkout v3.0
Note: switching to 'v3.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 13e6d431 Merge pull request #195 from geodynamics/developer

Related Entries

• UCVM