OLCF Summit

Configuration for large mesh extraction using UCVM

Scott module list

Currently Loaded Modules:
  1) lsf-tools/2.0   5) nsight-compute/2021.2.1      9) imagemagick/7.0.8-7             13) fftw/3.3.10
  2) hsi/5.0.2.p5    6) nsight-systems/2021.3.1.54  10) python/3.6.10-anaconda3
  3) xalt/1.2.1      7) cuda/11.0.3                 11) gcc/9.1.0
  4) DefApps         8) subversion/1.14.1           12) spectrum-mpi/10.4.0.3-20210112

50 mesh configuration

[pmaech@login5.summit ucvm_for_yifeng]$ more *.conf

# List of CVMs to query
ucvmlist=cvmsi,elygtl:ely
# UCVM conf file
ucvmconf=/gpfs/alpine/proj-shared/geo112/CyberShake/software/UCVM/ucvm-22.7.0/conf/ucvm.conf
# Gridding cell centered or vertex
gridtype=CENTER
# Spacing of cells
spacing=50.0
# Projection
proj=+proj=utm +datum=NAD27 +zone=11
rot=-27.5082989431
x0=-119.261301
y0=33.834327
z0=0.0
# Number of cells along each dim
nx=6000
ny=2744
nz=1600
# Partitioning of grid among processors
px=40
py=7
pz=4
# Vs/Vp minimum
vp_min=710.
vs_min=500.
# Mesh and grid files, format
meshfile=cvmsi50+elygtl.media
gridfile=cvmsi50+elygtl.grid
meshtype=IJK-12
# Location of scratch dir
scratch=/gpfs/alpine/scratch/callag/geo112/ucvm_for_yifeng

job submit script

#!/bin/bash
#BSUB -P geo112
#BSUB -W 2:00
#BSUB -q debug
#BSUB -nnodes 28
#BSUB -J cvmsi50Mesh
#BSUB -oo cvmsi50Mesh.o
#BSUB -oe cvmsi50Mesh.e

#source ./ucvm_env.sh

cd $LS_SUBCWD
jsrun -n 1120 -r 40 -a 1 -c 1 /gpfs/alpine/proj-shared/geo112/CyberShake/software/UCVM/ucvm-22.7.0/bin/ucvm2mesh_mpi -f cvmsi50.conf
date

jsrun configuration tools

• https://code.ornl.gov/t4p/Hello_jsrun
• https://jobstepviewer.olcf.ornl.gov/