UCVMC La Habra mesh generation

Problem

A general solution for creating a large mesh is to partition a target 3D mesh into a regular grid with smaller segments. The segments are then processed in a divide-and-conquer manner and the results are gathered into a single file.

UCVMC17.1's ucvm2mesh-mpi maps the target 3D mesh into a regular grid based on the parameters supplied in the mesh configuration file. The parameters: px, py and pz are the number of partitions in x,y,z orientation. A unit of process or rank is defined as an unit within the grid. Therefore, there are exactly px * py * pz ranks within the 3D mesh.

ucvm2mesh-mpi starts up multiple mpi processes equal in number to ranks required for the target mesh. All processes are ran concurrently and write their results to the same output file based on their 'ranking' position.

This solution is simple but does not scale well when the target mesh is extremely large.

La Habra at blue waters (NCSA)

The resulting mesh is about 2.2T in size. The resulting directory should be striped to improve the performance.

cd /path/to/working/directory
lfs setstripe -c 4 ./

bw_la_habra_mesh.conf

#List of CVMs to query
ucvmlist=cvmsi

# UCVM conf file
ucvmconf=/u/sciteam/meisu/scratch/UCVMC_TEST/ucvm.conf

# Gridding cell centered or vertex
gridtype=CENTER

# Spacing of cells
spacing=20.0

# Projection
proj=+proj=utm +datum=WGS84 +zone=11
rot=-39.9
x0=-119.288842
y0=34.120549
z0=0.0

# Number of cells along each dim
nx=9000
ny=6750
nz=3072

# Partitioning of grid among processors
px=18
py=18
pz=64

# Vs/Vp minimum
vp_min=0
vs_min=0

# Mesh and grid files, format
meshfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_mesh.media
gridfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_mesh.grid
meshtype=IJK-12

# Location of scratch dir
scratch=/u/sciteam/meisu/scratch

bw_la_habra_mesh.pbs

#!/bin/bash
#PBS -l walltime=24:00:00,nodes=81:ppn=8:xe
#PBS -A baln
#PBS -e /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_mesh.err
#PBS -o /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_mesh.out

## prepend velocity model's lib
export TEST_UCVMC_TARGET=/projects/sciteam/baln/meisu/TARGET_UCVMC
export UCVM_SRC_PATH=$TEST_UCVMC_TARGET/UCVMC
export UCVM_INSTALL_PATH=$TEST_UCVMC_TARGET/install

if [ $LD_LIBRARY_PATH ] ; then
  export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib:$LD_LIBRARY_PATH
  else
    export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib
fi
#echo $LD_LIBRARY_PATH

export TEST_TOP_PATH=/u/sciteam/meisu/scratch/UCVMC_TEST
export TEST_SRC_PATH=${TEST_TOP_PATH}/ncsa/ucvm2mesh

###############################################################################


cd $PBS_O_WORKDIR
cp ${UCVM_INSTALL_PATH}/bin/ucvm2mesh_mpi_layer .
cp ${TEST_SRC_PATH}/bw_la_habra_mesh.conf .


#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_mesh.conf -l 1  -c 16 
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_mesh.conf -l 17 -c 16
aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_mesh.conf -l 33 -c 16
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_mesh.conf -l 49 -c 16 


echo "Jobs done"
exit 0

Related Entries

• Ucvm2mesh-mpi