UCVM with post-Geomodelgrid software stack

Frontera environment

This is for newer UCVM software stack with geomodelgrids support.

Modules to load,

module load gcc/13.2.0

module load phdf5/1.14.4
module load sqlite/3.43.0
module load proj/9.3.1

#conflict with gcc
#module load python3/3.9.2

module load automake
module load autoconf

module load mkl

Because SFCVM/Geomodelgrids needs newer gcc but Frontera's various python3 module do not support that (gcc9.1.0) at this moment, so need to install anaconda for the python3 and some additional python pkgs.

Currently Loaded Modules:
  1) git/2.24.1      4) hwloc/1.11.12   7) gcc/13.2.0    10) sqlite/3.43.0    13) autoconf/2.72
  2) autotools/1.2   5) xalt/2.10.34    8) impi/21.9.0   11) proj/9.3.1       14) mkl/23.1.0
  3) cmake/3.24.2    6) TACC            9) phdf5/1.14.4  12) automake/1.16.5

Inactive Modules:
  1) python3

Script for setting up the UCVM/SW4 environment (setup-env)

Note: should change setting for MY_TOP to match your directory setup

#!/bin/bash

export MY_TOP=/home1/07508/mei/scratch1

export TOP_UCVM_TARGET=$MY_TOP/TARGET_UCVM_FRONTERA
export UCVM_SRC_PATH=$TOP_UCVM_TARGET/src
export UCVM_INSTALL_PATH=$TOP_UCVM_TARGET/ucvm_install

export ANACONDA3_TOP_DIR=$MY_TOP/anaconda3

# for python3.12
match="python3.12"
if [[ $PATH != *"$match"* ]]; then    
   export PATH=$ANACONDA3_TOP_DIR/bin:$PATH
   export PYTHONPATH=$ANACONDA3_TOP_DIR/lib/$match/site-packages
   echo "adding for "$match
   echo $PYTHONPATH
fi

export SW4_TOP_DIR=$MY_TOP/sw4

Script to build up anaconda

This should be called once only.

#!/bin/bash
#
# install only if conda has not install yet
# 

source ~/setup_env

if [ -d $ANACONDA3_TOP_DIR ]; then
  echo "FOUND anaconda top"
  exit
fi


wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O anaconda.sh 
bash anaconda.sh -b -f -p $ANACONDA3_TOP_DIR
rm -f anaconda.sh
conda init bash

/bin/python -m pip install scipy h5py numpy pandas pybind11

Script to build up UCVM

#!/bin/bash

source ~/setup-env

rm -rf $TOP_UCVM_TARGET 
mkdir $TOP_UCVM_TARGET

cd $TOP_UCVM_TARGET
git clone https://github.com/SCECcode/ucvm.git -b testFrontera src


cd $UCVM_SRC_PATH/largefiles
./get_largefiles.py -m cvmsi,sfcvm

cd $UCVM_SRC_PATH/largefiles; ./check_largefiles_md5.py
cd $UCVM_SRC_PATH/largefiles; ./stage_largefiles.py

cd $UCVM_SRC_PATH
export PYTHONUNBUFFERED=TRUE
./ucvm_setup.py -d -a -p $UCVM_INSTALL_PATH &> ucvm_setup_install.log

cd $UCVM_SRC_PATH; make check

echo "..EXITING.."
exit

Script to build up sw4

source ~/setup-env

rm -rf $SW4_TOP_DIR
mkdir $SW4_TOP_DIR

cd $SW4_TOP_DIR
git clone https://github.com/meihuisu/sw4.git -b ucvm_cmd src 

cd $SW4_TOP_DIR/src
cd configs
ln -s make.ucvm_cmd.frontera  make.inc
cd $SW4_TOP_DIR/src
make

echo "..EXITING.."
exit

make.ucvm_cmd.frontera

## FOR TACC Frontera

FC = gfortran
CXX = mpicxx

proj=yes
hdf5=yes

SW4ROOT = ${TACC_PROJ_DIR}
SQL3ROOT = ${TACC_SQLITE_DIR} 
HDF5ROOT = ${TACC_PHDF5_DIR}

EXTRA_CXX_FLAGS  = -fopenmp -I$MKLROOT/include
EXTRA_FORT_FLAGS =
EXTRA_LINK_FLAGS = -L${UCVM_INSTALL_PATH}/lib/openssl/lib -lcrypto -lssl -Wl,-L${MKLROOT}/lib/intel64 -lmkl_i
ntel_lp64 -lmkl_core -lmkl_gnu_thread -lpthread -lm -ldl -L${GCC_LIB} -lgfortran    

##MKLROOT /opt/intel/oneapi/mkl/2023.1.0

Vista environment

module load tacc_tips

module load gcc/13.2.0

module load cuda/12.6 
module load openmpi/5.0.5

module load phdf5/1.14.4
module load sqlite/3.46.1
module load proj/9.3.1

module load python3/3.11.8
module load automake
module load autoconf
module load blis
module load nvpl

Currently Loaded Modules:
  1) ucc/1.3.0      5) TACC                9) openmpi/5.0.5  13) python3/3.11.8   17) nvpl/24.7
  2) ucx/1.17.0     6) tacc_tips/1.5      10) phdf5/1.14.4   14) automake/1.16.5
  3) cmake/3.31.5   7) gcc/13.2.0         11) sqlite/3.46.1  15) autoconf/2.72
  4) xalt/3.1       8) cuda/12.6     (g)  12) proj/9.3.1     16) blis/1.0

  Where:
   g:  built for GPU

Script for setting up the UCVM/SW4 environment (setup-env)

Note: change setting for MY_TOP to match your directory setup

#!/bin/bash

export MY_TOP=/home1/07508/mei/scratch

export TOP_UCVM_TARGET=$MY_TOP/TARGET_UCVM_VISTA
export UCVM_SRC_PATH=$TOP_UCVM_TARGET/src
export UCVM_INSTALL_PATH=$TOP_UCVM_TARGET/install

##export CVM_LARGEDATA_DIR=/home1/07508/mei/work/cvm-largedata-dir

export SW4_TOP_DIR=$MY_TOP/sw4

Python3

Vista's python3 is usable and so just need to install some python packages

pip3 install scipy h5py numpy pandas pybind11

Script to build up UCVM

#!/bin/bash

source ~/setup-env

rm -rf $TOP_UCVM_TARGET 
mkdir $TOP_UCVM_TARGET

cd $TOP_UCVM_TARGET
git clone https://github.com/SCECcode/ucvm.git -b testFrontera src

cd $UCVM_SRC_PATH/largefiles
./get_largefiles.py -m sfcvm,cvmsi

cd $UCVM_SRC_PATH/largefiles; ./check_largefiles_md5.py
cd $UCVM_SRC_PATH/largefiles; ./stage_largefiles.py

cd $UCVM_SRC_PATH
export PYTHONUNBUFFERED=TRUE
./ucvm_setup.py -d -a -p $UCVM_INSTALL_PATH &> ucvm_setup_install.log

cd $UCVM_SRC_PATH; make check

echo "..EXITING.."
exit

Script to build up sw4

#!/bin/bash

source ~/setup-env

rm -rf $SW4_TOP_DIR
mkdir $SW4_TOP_DIR

cd $SW4_TOP_DIR
git clone https://github.com/meihuisu/sw4.git -b ucvm_cmd src 

cd $SW4_TOP_DIR/src
cd configs
ln -s make.ucvm_cmd.vista  make.inc
cd $SW4_TOP_DIR/src
make

echo "..EXITING.."
exit

make.ucvm_cmd.vista

## TACC vista

FC = gfortran
CXX = mpicxx

proj=yes
hdf5=yes

SW4ROOT=${UCVM_INSTALL_PATH}/lib/proj
##SW4ROOT=${TACC_PROJ_DIR}
SQL3ROOT=${TACC_SQLITE_DIR}
HDF5ROOT=${TACC_PHDF5_DIR}

EXTRA_LINK_FLAGS = -L${TACC_NVPL_DIR}/lib -lnvpl_blas_lp64_gomp  -lnvpl_lapack_lp64_gomp -lnvpl_blacs_lp64_open
mpi5 -lnvpl_scalapack_lp64 -L${GCC_LIB} -lgfortran

UCVM with pre-Geomodelgrid software stack

Frontera environment

Does not need any additional environment setup

Default Modules :

Currently Loaded Modules:

 1) intel/19.1.1   3) git/2.24.1      5) python3/3.7.0   7) pmix/3.1.4      9) xalt/2.10.34
 2) impi/19.0.9    4) autotools/1.2   6) cmake/3.24.2    8) hwloc/1.11.12  10) TACC

Script to build UCVM

Copy the script below to your home directory.

Replace <<YOUR_BUILD_LOCATION>> in the script to point to your big scratch location(either scratch1 or scratch2)

#!/bin/bash

export MY_TOP=<<YOUR_BUILD_LOCATION>>

export TOP_UCVM_TARGET=$MY_TOP/TARGET_UCVM
export UCVM_SRC_PATH=$TOP_UCVM_TARGET/ucvm
export UCVM_INSTALL_PATH=$TOP_UCVM_TARGET/ucvm_install

rm -rf $TOP_UCVM_TARGET
mkdir $TOP_UCVM_TARGET

cd $TOP_UCVM_TARGET
git clone https://github.com/SCECcode/ucvm.git -b withSCPBR


# all available velocity models in this UCVM branch
#
#cvmh,cencal,cca,cs173,cs173h,albacore,cvlsu,ivlsu,wfcvm
#cvms,cvms5,cvmsi
#cvmhibbn,cvmhlabn,cvmhrbn,cvmhsbbn,cvmhsbcbn,cvmhsgbn,cvmhsmbn,cvmhstbn,cvmhvbn
#sjcz
#

cd $UCVM_SRC_PATH/largefiles
# select the models to build/install
./get_largefiles.py -m cvmsi,cvms5,cvmh,cvms,cencal

cd $UCVM_SRC_PATH/largefiles; ./check_largefiles_md5.py
cd $UCVM_SRC_PATH/largefiles; ./stage_largefiles.py

cd $UCVM_SRC_PATH
./ucvm_setup.py -d -a -p $UCVM_INSTALL_PATH &> ucvm_setup_install.log

cd $UCVM_SRC_PATH; make check

echo "..EXITING.."
exit

Run the script to build your UCVM installation,

chmod +x script
source ./script

Test your UCVM installation

setup your UCVM enabled environment,

source <<YOUR_BUILD_LOCATION>>/TARGET_UCVM/ucvm_install/conf/ucvm_env.sh

Your $UCVM_INSTAlL_PATH is now set to "<<YOUR_BUILD_LOCATION>>/TARGET_UCVM/ucvm_install"
and your $UCVM_SRC_PATH is set to "<<YOUR_BUILD_LOCATION>>/TARGET_UCVM/ucvm"

To check if your ucvm environment is setup properly,

which ucvm_query

Following command will show what is available in this UCVM installation,

ucvm_query -H

and it should return,

Usage: ucvm_query [-m models<:ifunc>] [-p user_map] [-c coordtype] [-f config] [-z zmin,zmax] [-b] < file.in

Flags:
	-h This help message.
	-H Detail help message.
	-m Comma delimited list of crustal/GTL models to query in order
	   of preference. GTL models may optionally be suffixed with ':ifunc'
	   to specify interpolation function.
	-c Z coordinate mode: geo-depth (gd, default), geo-elev (ge).
	-f Configuration file. Default is ./ucvm.conf.
	-p User-defined map to use for elevation and vs30 data.
	-v Display model version information only.
	-z Optional depth range for gtl/crust interpolation.

	-b Optional output in json format

	-l Optional input lat,lon,Z(depth/elevation)

	-L Optional interpolation floor limit vs,vp,density(meter in depth mode)

Input format is:
	lon lat Z

Output format is:
	lon lat Z surf vs30 crustal cr_vp cr_vs cr_rho gtl gtl_vp gtl_vs gtl_rho cmb_algo cmb_vp cmb_vs cmb_rho

Notes:
	- If running interactively, type Cntl-D to end input coord list.

Version: 22.7.0

Installed Resources:
          1d : crustal model
       bbp1d : crustal model
    cmuetree : crustal model
       1dgtl : gtl
      elygtl : gtl
        cvmh : crustal model
      cencal : crustal model
       cvms5 : crustal model
       cvmsi : crustal model
        cvms : crustal model
         ely : ifunc
       taper : ifunc
        ucvm : map
        yong : map
 model_etree : model i/f
 model_patch : model i/f
   map_etree : map i/fand it should return,

Using a test file from UCVM regression suite,

-118.0 34.0 0.0
-118.0 34.0 50.0
-118.0 34.0 100.0
-118.0 34.0 500.0
-118.0 34.0 1000.0

Calling ucvm_query to query cvmsi model and interpolate it with the taper algorithm,

ucvm_query -m cvmsi,elygtl:taper -f $UCVM_INSTALL_PATH/conf/ucvm.conf < $UCVM_INSTALL_PATH/tests/inputs/test_latlons.txt

should return,

Using Geo Depth coordinates as default mode.
 -118.0000    34.0000      0.000    284.191    580.043      cvmsi   3035.433   1719.851   2402.684     elygtl      0.000    580.043      0.000      taper   1700.000    500.000   1974.976
 -118.0000    34.0000     50.000    284.191    580.043      cvmsi   3035.433   1719.851   2402.684     elygtl      0.000    580.043      0.000      taper   1700.000    548.000   2128.620
 -118.0000    34.0000    100.000    284.191    580.043      cvmsi   3035.433   1719.851   2402.684     elygtl      0.000    580.043      0.000      taper   1700.000    603.470   2130.773
 -118.0000    34.0000    500.000    284.191    580.043      cvmsi   3035.433   1719.851   2402.684     elygtl      0.000    580.043      0.000      taper   2701.194   1475.595   2354.105
 -118.0000    34.0000   1000.000    284.191    580.043      cvmsi   3330.891   1945.586   2443.042       none      0.000      0.000      0.000      crust   3330.891   1945.586   2443.042

UCVM install on Frontera

Contents

UCVM with post-Geomodelgrid software stack

Frontera environment

Script for setting up the UCVM/SW4 environment (setup-env)

Script to build up anaconda

Script to build up UCVM

Script to build up sw4

Vista environment

Script for setting up the UCVM/SW4 environment (setup-env)

Python3

Script to build up UCVM

Script to build up sw4

UCVM with pre-Geomodelgrid software stack

Frontera environment

Script to build UCVM

Test your UCVM installation

Related Links

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

Tools