UCVMC La Habra mesh generation
Problem
A general solution for creating a large mesh is to partition a target 3D mesh into a regular grid with smaller segments. The segments are then processed in a divide-and-conquer manner and the results are gathered into a single file.
UCVMC17.1's ucvm2mesh-mpi maps the target 3D mesh into a regular grid based on the parameters supplied in the mesh configuration file. The parameters: px, py and pz are the number of partitions in x,y,z orientation. A unit of process or rank is defined as an unit within the grid. Therefore, there are exactly px * py * pz ranks within the 3D mesh.
ucvm2mesh-mpi starts up multiple mpi processes equal in number to ranks required for the target mesh. All processes are ran concurrently and write their results to the same output file based on their 'ranking' position.
This solution is simple but does not scale well when the target mesh is extremely large.
La Habra
#List of CVMs to query ucvmlist=cvmsi # UCVM conf file ucvmconf=/u/sciteam/meisu/scratch/UCVMC_TEST/ucvm.conf # Gridding cell centered or vertex gridtype=CENTER # Spacing of cells spacing=20.0 # Projection proj=+proj=utm +datum=WGS84 +zone=11 rot=-39.9 x0=-119.288842 y0=34.120549 z0=0.0 # Number of cells along each dim nx=9000 ny=6750 nz=3072 # Partitioning of grid among processors px=18 py=18 pz=64 # Vs/Vp minimum vp_min=0 vs_min=0 # Mesh and grid files, format meshfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.media gridfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.grid meshtype=IJK-12 # Location of scratch dir scratch=/u/sciteam/meisu/scratch
#!/bin/bash
#PBS -l walltime=24:00:00,nodes=81:ppn=8:xe
#PBS -A baln
#PBS -e /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.err
#PBS -o /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.out
## prepend velocity model's lib
export TEST_UCVMC_TARGET=/projects/sciteam/baln/meisu/TARGET_UCVMC
export UCVM_SRC_PATH=$TEST_UCVMC_TARGET/UCVMC
export UCVM_INSTALL_PATH=$TEST_UCVMC_TARGET/install
if [ $LD_LIBRARY_PATH ] ; then
export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib:$LD_LIBRARY_PATH
else
export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib
fi
#echo $LD_LIBRARY_PATH
export TEST_TOP_PATH=/u/sciteam/meisu/scratch/UCVMC_TEST
export TEST_SRC_PATH=${TEST_TOP_PATH}/ncsa/ucvm2mesh
###############################################################################
cd $PBS_O_WORKDIR
cp ${UCVM_INSTALL_PATH}/bin/ucvm2mesh_mpi_layer .
cp ${TEST_SRC_PATH}/bw_la_habra_l_mesh.conf .
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 49 -c 16
aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 33 -c 16
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 17 -c 16
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 1 -c 16
echo "Jobs done"
exit 0
