Difference between revisions of "UCVMC La Habra mesh generation"

From SCECpedia
Jump to navigationJump to search
Line 16: Line 16:
  
 
This solution is simple but does not scale well when the target mesh is extremely large.
 
This solution is simple but does not scale well when the target mesh is extremely large.
 +
 +
== La Habra ==
 +
 +
<pre>
 +
#List of CVMs to query
 +
ucvmlist=cvmsi
 +
 +
# UCVM conf file
 +
ucvmconf=/u/sciteam/meisu/scratch/UCVMC_TEST/ucvm.conf
 +
 +
# Gridding cell centered or vertex
 +
gridtype=CENTER
 +
 +
# Spacing of cells
 +
spacing=20.0
 +
 +
# Projection
 +
proj=+proj=utm +datum=WGS84 +zone=11
 +
rot=-39.9
 +
x0=-119.288842
 +
y0=34.120549
 +
z0=0.0
 +
 +
# Number of cells along each dim
 +
nx=9000
 +
ny=6750
 +
nz=3072
 +
 +
# Partitioning of grid among processors
 +
px=18
 +
py=18
 +
pz=64
 +
 +
# Vs/Vp minimum
 +
vp_min=0
 +
vs_min=0
 +
 +
# Mesh and grid files, format
 +
meshfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.media
 +
gridfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.grid
 +
meshtype=IJK-12
 +
 +
# Location of scratch dir
 +
scratch=/u/sciteam/meisu/scratch
 +
 +
</pre>
 +
 +
<pre>
 +
#!/bin/bash
 +
#PBS -l walltime=24:00:00,nodes=81:ppn=8:xe
 +
#PBS -A baln
 +
#PBS -e /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.err
 +
#PBS -o /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.out
 +
 +
## prepend velocity model's lib
 +
export TEST_UCVMC_TARGET=/projects/sciteam/baln/meisu/TARGET_UCVMC
 +
export UCVM_SRC_PATH=$TEST_UCVMC_TARGET/UCVMC
 +
export UCVM_INSTALL_PATH=$TEST_UCVMC_TARGET/install
 +
 +
if [ $LD_LIBRARY_PATH ] ; then
 +
  export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib:$LD_LIBRARY_PATH
 +
  else
 +
    export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib
 +
fi
 +
#echo $LD_LIBRARY_PATH
 +
 +
export TEST_TOP_PATH=/u/sciteam/meisu/scratch/UCVMC_TEST
 +
export TEST_SRC_PATH=${TEST_TOP_PATH}/ncsa/ucvm2mesh
 +
 +
###############################################################################
 +
 +
 +
cd $PBS_O_WORKDIR
 +
cp ${UCVM_INSTALL_PATH}/bin/ucvm2mesh_mpi_layer .
 +
cp ${TEST_SRC_PATH}/bw_la_habra_l_mesh.conf .
 +
 +
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 49 -c 16
 +
aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 33 -c 16
 +
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 17 -c 16
 +
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 1  -c 16
 +
 +
echo "Jobs done"
 +
exit 0
 +
 +
</pre>
  
 
== Related Entries ==
 
== Related Entries ==
 
*[[Ucvm2mesh-mpi]]
 
*[[Ucvm2mesh-mpi]]

Revision as of 21:28, 4 June 2018

Problem

A general solution for creating a large mesh is to partition a target 3D mesh into a regular grid with smaller segments. The segments are then processed in a divide-and-conquer manner and the results are gathered into a single file.

UCVMC17.1's ucvm2mesh-mpi maps the target 3D mesh into a regular grid based on the parameters supplied in the mesh configuration file. The parameters: px, py and pz are the number of partitions in x,y,z orientation. A unit of process or rank is defined as an unit within the grid. Therefore, there are exactly px * py * pz ranks within the 3D mesh.

ucvm2mesh-mpi starts up multiple mpi processes equal in number to ranks required for the target mesh. All processes are ran concurrently and write their results to the same output file based on their 'ranking' position.

This solution is simple but does not scale well when the target mesh is extremely large.

La Habra

#List of CVMs to query
ucvmlist=cvmsi

# UCVM conf file
ucvmconf=/u/sciteam/meisu/scratch/UCVMC_TEST/ucvm.conf

# Gridding cell centered or vertex
gridtype=CENTER

# Spacing of cells
spacing=20.0

# Projection
proj=+proj=utm +datum=WGS84 +zone=11
rot=-39.9
x0=-119.288842
y0=34.120549
z0=0.0

# Number of cells along each dim
nx=9000
ny=6750
nz=3072

# Partitioning of grid among processors
px=18
py=18
pz=64

# Vs/Vp minimum
vp_min=0
vs_min=0

# Mesh and grid files, format
meshfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.media
gridfile=/u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.grid
meshtype=IJK-12

# Location of scratch dir
scratch=/u/sciteam/meisu/scratch


#!/bin/bash
#PBS -l walltime=24:00:00,nodes=81:ppn=8:xe
#PBS -A baln
#PBS -e /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.err
#PBS -o /u/sciteam/meisu/scratch/UCVM_REVIEW/RESULT/bw_la_habra_l_mesh.out

## prepend velocity model's lib
export TEST_UCVMC_TARGET=/projects/sciteam/baln/meisu/TARGET_UCVMC
export UCVM_SRC_PATH=$TEST_UCVMC_TARGET/UCVMC
export UCVM_INSTALL_PATH=$TEST_UCVMC_TARGET/install

if [ $LD_LIBRARY_PATH ] ; then
  export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib:$LD_LIBRARY_PATH
  else
    export LD_LIBRARY_PATH=$UCVM_INSTALL_PATH/lib/euclid3/lib:$UCVM_INSTALL_PATH/lib/proj-4/lib:$UCVM_INSTALL_PATH/model/cvms426/lib:$UCVM_INSTALL_PATH/model/cencal/lib
fi
#echo $LD_LIBRARY_PATH

export TEST_TOP_PATH=/u/sciteam/meisu/scratch/UCVMC_TEST
export TEST_SRC_PATH=${TEST_TOP_PATH}/ncsa/ucvm2mesh

###############################################################################


cd $PBS_O_WORKDIR
cp ${UCVM_INSTALL_PATH}/bin/ucvm2mesh_mpi_layer .
cp ${TEST_SRC_PATH}/bw_la_habra_l_mesh.conf .

#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 49 -c 16 
aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 33 -c 16 
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 17 -c 16 
#aprun -n 648 ./ucvm2mesh_mpi_layer -f bw_la_habra_l_mesh.conf -l 1  -c 16 

echo "Jobs done"
exit 0

Related Entries

Retrieved from "https://strike.scec.org/scecwiki/index.php?title=UCVMC_La_Habra_mesh_generation&oldid=20935"