CVM-T User Guide

CVM-T

Overview

The SCEC CVM-Toolkit (CVM-T) is a software system that provides one or more community velocity models (CVMs), a mesh extraction tool for generated 3D velocity meshes from these CVMs in a variety of formats, a testing framework for evaluating the performance of CVMs, and visualization tools for plotting slices from either a CVM or a mesh. Each of these components is largely independent of the others (an exception being cvm2mesh which has dependencies on CVM-H), so the user is able to choose which components he wishes to use. The following sections of this user guide describe how to install and operate each tool.

Requirements/Dependencies

The CVM-T system has many dependencies on third-party software packages. Each CVM-T component has its own specific requirements which are listed in that component's user guide. This document does not describe in detail how to install these dependencies, as many are complex enough to warrant their own guide. Instead, this document refers the reader to the URL where the software distribution and documentation can be found. In general, the CVM-T system relies on these packages:

• SCEC CVM-H https://source.usc.edu/svn/cvmh
• SCEC CVM-4 http://earth.usc.edu/~gely/coseis/www/index.html
• SCEC Broadband Platform https://source.usc.edu/svn/broadband
• AWP-ODC Finite Difference Codes http://hpgeoc.sdsc.edu/AWPODC/
• Python 2.6+ with numpy, matplotlib, basemap, and pyproj modules on local/remote hosts
• USGS DEMs [Optional] from the USGS National Map Seamless Server http://seamless.usgs.gov/index.php
• CruiseControl
• Globus Toolkit (gsissh, GridFTP)

CVM-H

Requirements

None.

Installation

• The package is built by executing the following make command from the root directory:

% make clean;make all

Utilities

The following executable utilities are provided:

vx

Original interface to Harvard's model. Accepts lon/lat/elev points from stdin and prints material properties to stdout. The original user guide can be found here .

vx_lite

Updated interface to Harvard's model that supports optional 1D background, Vs30 derived GTL, and querying by depth. Accepts lon/lat/z points from stdin and prints material properties to stdout.

vx_slice

Visualization utility for extracting a 2D horizontal slice of Vp, Vs, or Rho (density) from the model, suitable for plotting with a graphics package.

Querying for SoCal Material Properties on the Command Line

Two utilities are provided for querying the community velocity model, vx and vx_lite. The vx utility is the original interfance to Harvard's model. It is described in the PDF linked in the previous section. The vx_lite utility is a new SCEC-developed interface which supports a number of enhancements, including query by depth, extension of the coverage region with a SCEC 1D model, and replacement of the original GTL with a Vs30-derived GTL.

The command line format for vx_lite is as follows:

% vx_lite [-s] [-d] [-v] [< input_coords]

where the options:

-s

Instructs use of SCEC 1D background model

-d

Interpret z coordinate as depth from free surface

-v

Interpret z coordinate as elevation

input_coords

3D point specified as either (lon, lat, z (meters)) or (easting, northing, z (meters)), space delimited. Input coordinates can be specified interactively or redirected to stdin from a file.

Note: Options -d and -v are mutually exclusive. Specifying neither option instructs vx_lite to interpret the z coordinate as elevation offset from free surface.

The query data is printed to stdout, and this may be redirected to a file.

CVM-H defines a set of sample test points (given in ./bin/test_data/test.dat):

-125 35 -7777
-118.56 32.55 -2450
360061 3750229 -1400
-118.52 34.12 -1400
-116.40 32.34 -1000
376592 3773379 -1770
376592 3773379 -17700
408669 3766189 -3000

These may be submitted to vx_lite with this command:

% ./vx_lite -s -v < ./test_data/test.dat

The material properties at the sample points will be printed to stdout. You can expect to see the following output, annotated here with column headers. The three important fields are the last three, containing vp, vs, and rho.

     X               Y             Z       utmX        utmY      elevX       elevY      topo      mtop      base      moho  hr/lr/cm cellX     cellY     cellZ       tag        vp        vs       rho
  -125.000000       35.000000  -7777.00 -230844.88  3902223.73  -99999.00   -99999.00      0.00      0.00 -99999.00 -99999.00 bk  -99999.00   -99999.00 -99999.00     14.00   6300.00   3637.31   2859.77
  -118.560000       32.550000  -2450.00  353525.18  3602285.14  353625.00  3602375.00  -1114.91  -1150.00  -1327.54 -21571.67 lr  354000.00  3602000.00  -2400.00      2.00   5575.15   3132.10   2631.81
360061.000000  3750229.000000  -1400.00  360061.00  3750229.00  360125.00  3750125.00    -56.93    -50.00  -1404.07 -24868.83 lr  360000.00  3750000.00  -1400.00      2.00   4554.52   2313.56   2469.78
  -118.520000       34.120000  -1400.00  359819.67  3776309.78  359875.00  3776375.00    491.46    450.00     38.42 -28061.40 lr  360000.00  3776000.00  -1400.00      2.00   5066.61   2916.30   2545.10
  -116.400000       32.340000  -1000.00  556464.74  3578092.46  556375.00  3578125.00    780.43    750.00    616.39 -31413.62 lr  556000.00  3578000.00  -1000.00      2.00   5372.79   3024.30   2595.55
376592.000000  3773379.000000  -1770.00  376592.00  3773379.00  376625.00  3773375.00     99.38    100.00  -2374.53 -28165.35 hr  376552.25  3773500.00  -1800.00      3.00   4181.37   2432.22   2418.45
376592.000000  3773379.000000 -17700.00  376592.00  3773379.00  376625.00  3773375.00     99.38    100.00  -2374.53 -28165.35 cm  380000.00  3770000.00 -18000.00      2.00   6533.31   3776.40   2841.47
408669.000000  3766189.000000  -3000.00  408669.00  3766189.00  408625.00  3766125.00     93.89     50.00  -2820.45 -29799.86 hr  408552.25  3766250.00  -3000.00      2.00   4997.06   2889.03   2534.30

Querying for SoCal Material Properties in a C Program

SCEC has developed a C API to the Harvard model. The header containing the datatype and function definitions is located in ./src/vx_sub.h. By including this header in your source, and the libvxapi.a library at link time, your program can directly query CVM-H. See the cvm2mesh mesh extraction tool below for an example implementation that uses this API.

Known Issues

None.

cvm2mesh

Requirements

• SCEC CVM-H https://source.usc.edu/svn/cvmh
• SCEC CVM-4 [Optional] http://earth.usc.edu/~gely/coseis/www/index.html Set ibig var in newin.h to 9,000,000+ and recompile using the following Makefile.

integer, parameter :: ibig = 10000000

# CVM-4 Makefile
all: cvm4_txt cvm4_bin cvm4_mpi
cvm4_txt: cvm4.f iotxt.f
       gfortran -Wall -O3 cvm4.f iotxt.f -o cvm4_txt
cvm4_bin: cvm4.f iobin.f
       gfortran -Wall -O3 cvm4.f iobin.f -o cvm4_bin
cvm4_mpi: cvm4.f iompi.f
       gfortran -Wall -O3 cvm4.f iobin.f -o cvm4_mpi
clean:
       rm *.o *~ cvm4_txt cvm4_bin cvm4_mpi

Installation

• If you wish to extract meshes from CVM-4, it must be installed (see requirements).
• Update ./src/Makefile with paths to the CVM-H library libvxapi.a and system compilers. Execute 'make' to build all meshing executables.

Generating a Mesh

Prepatory steps

• Determine meshing region, projection to use (CMU or UTM), and desired dimensions.
• Determine if CVM can be pre-staged before the job executes. One copy of each CVM (model, extraction binary) is required per core in the MPI extraction job. Some computing resources (jaguar, kraken, ranger) have a permanent scratch space where these CVMs may be staged before the job executes, while others (USC HPCC) have sratch space that exists only while the job is in progress.
  • If CVM pre-staging is possible and desired, copy N copies of the CVM to your scratch space. Example pbs scripts which perform this task can be found in ./pbs.

Create extraction configuration file

Create extraction configuration file that defines the following properties:

cvmtype

CVM4 or CVMH

cvmdir

Root directory containing pre-staged CVM if using that option, or CVM install directory that mesh-create-MPI will use to stage the data itself

spacing

grid spacing in meters

proj

UTM or CMU

rot

rotation angle in degrees, only used in UTM mode

lon

longitude of SW corner of mesh box

lat

latitude of SW corner of mesh box

dep

starting depth in km from free surface

x-size

length along x-axis in km

y-size

length along y-axis in km

z-size

length along z-axis in km. NOTE: Ensure (z-size/spacing) is equally divisible by your anticipated number of cores

vp_min

vp to assign when model vs < vs_min in m/s

vs_min

vs to assign when model vs < vs_min in m/s

outputfile

Name of final mesh file

format

Output format, IJK-12, IJK-20, IJK-32, SORD

The following is an example configuration file:

# Example cvm2mesh config file for IJK12 CVM4 extraction
cvmtype=CVM4
cvmdir=/lustre/widow1/scratch/spatrick/tmp/cvm4
spacing=80
proj=UTM
rot=-45.42589966
lon=-122.500798753
lat=35.796092162
dep=0
x_size=810
y_size=400
z_size=84.8
vp_min=0
vs_min=0
outputfile=/lustre/widow1/proj/geo008/yfcui/w2w1hz40m_SN_cvmH/input/mesh_m8_cvm4_ijk12_1hz
meshtype=IJK-12

Create PBS job submission script

There are many examples in the ./pbs directory. The general command-line format for mesh-create-MPI is as follows:

./mesh-create-MPI [-i input_cvm_dir] [-o output_mesh_file] -f mesh_params.conf

where the options -i and -o are used for optional staging in of the CVM and stage out of the final mesh:

-i

Option instructing mesh-create-MPI to stage in NUM_CORES copies of the CVM from the directory specified in the option argument into the scratch directory specified by the cvmdir keyword in the configuration file. Each copy will be placed in cvmdir/# where # is the rank number (0 to NUM_CORES-1). Useful on systems with temporary scratch space.

-o

Option instructing mesh-create-MPI to stage out the mesh file from scratch space to a permanent directory. The scratch space mesh file name is specified by the outputfile keyword in the configuration file, an the final stage out mesh file is specified by the option argument. Useful on systems that employ temporary scratch space.

Submit Extraction Job to Job Scheduler

System dependent, but generally accomplished with the qsub command:

% qsub example_extract_mesh.pbs

Check Mesh for Correctness

• Check for correct size of the mesh file (for IJK-12, x-size/spacing * y-size/spacing * z-size/spacing * 4 bytes/float * 3 floats)
• Perform octal dump on file to sanity check values:

% od -f mesh_file | more

• Visualize slices with viz-cvm

Known Issues

None.

cvmtest

Requirements

• SCEC Broadband Platform https://source.usc.edu/svn/broadband on remote host
• AWP-ODC Finite Difference Codes http://hpgeoc.sdsc.edu/AWPODC/ on remote host
• Python 2.6+ with numpy, matplotlib, basemap, and pyproj modules on local/remote hosts
• USGS DEMs [Optional] from the USGS National Map Seamless Server http://seamless.usgs.gov/index.php on remote host
• CruiseControl [Optional] on local host
• Globus Toolkit (gsissh, GridFTP) on local/remote hosts
• CVM-T cvm2mesh from https://source.usc.edu/svn/cvm2mesh on remote host
• CVM-T viz-cvm from https://source.usc.edu/svn/viz-cvm on remote host

Installation

Installation of the automated test framework has two components, an install on a local machine (from which you will launch the GoF acceptance tests) and an install on a remote grid resource (such as NICS Kraken, where the tests will execute). If you wish to have the tests managed by CruiseControl, your local machine must have the CC package installed.

Local Install

• Check out cvmtest distribution from https://source.usc.edu/svn/cvmtest/trunk or unpack the pre-packaged tar file into ${CVMTEST_HOME}, a directory of your choosing.
• Check out a local copy of CVM-H from https://source.usc.edu/svn/cvmtest into ${CVMTEST_HOME}/cvmh
• If cvmtest will be run under the auspices of a CruiseControl installation:
  • Configure CC to execute the update, build, and unit test targets in the ${CVMTEST_HOME}/cvmh/trunk/build.xml ANT file according to your software configuration management policies.
  • Configure CC to invoke the GoF acceptance tests, ${CVMTEST_HOME}/remote/AcceptTest.py, at the desired date and time.
  • Ensure the owner of the CC processes and the owner of the cvmtest/cvmh checkouts are the same userid.

Remote Install on Grid Resource

Currently, cvmtest is configured to run GoF acceptance tests on NICS Kraken. The following steps outline the setup required on Kraken.

• Check out cvmtest distribution from https://source.usc.edu/svn/cvmtest/trunk or unpack the pre-packaged tar file into ${CVMTEST_HOME}, a directory of your choosing on the /lustre scratch filesystem.

• Customize ${CVMTEST_HOME}/config/python/ATFConfig.py for system specific paths. The installation base directory needs to be configured as well as the location of the SCEC Broadband GoF codes:

INSTALL_BASE_PATH='/lustre/scratch/patricks/build'
GOF_PATH = '/lustre/scratch/patricks/broadband/src/urs/bin'

• Copy AWP-ODC executable pmcl3d in all ${CVMTEST_HOME}/awp/${EVENT}/${CVM} directories, where EVENT and CVM are all event and CVM combinations defined in ${CVMTEST_HOME}/config/python/ATFConfig.py

Running the Existing Chino Hills scenario with CVM-H

Running from CruiseControl

• Create a valid Teragrid x.509 proxy certificate for the owner of cvmtest/cvmh/CC. Note that the owner of the grid user certificate does not need to be the same as the owner of cvmtest/cvmh/CC. In other words, cvmtest can piggy-back on a different userid's certificate.
• Configure CC to execute ${CVMTEST_HOME}/remote/AcceptTest.py at the desired date and time

The test results and plots in ${CVMTEST_HOME}/web_results.tar may then be transferred to a webhost and unpacked for public release.

Running manually from Local Host

• Create a valid Teragrid x.509 proxy certificate for the owner of cvmtest/cvmh/CC. Note that the owner of the grid user certificate does not need to be the same as the owner of cvmtest/cvmh/CC. In other words, cvmtest can piggy-back on a different userid's certificate.
• Execute the following script from the shell prompt:

% cd ${CVMTEST_HOME}/remote
% ./AcceptTest.py

The test results and plots in ${CVMTEST_HOME}/web_results.tar may then be transferred to a webhost and unpacked for public release.

Running manually on NICS Kraken

Since the workflow management provided in AcceptTest.py is not being used, the user must login to Kraken and execute these steps sequentially while monitor the job progress himself.

• Build CVM-H

% qsub ${CVMTEST_HOME}/pbs/kraken/build_cvmh.pbs

• Build cvm2mesh linking in the previously build CVM-H

% qsub ${CVMTEST_HOME}/pbs/kraken/build_cvm2mesh.pbs

• Extract Chino Hills 100m mesh

% qsub ${CVMTEST_HOME}/pbs/kraken/cvm2mesh_chino_uf_cvmh.pbs

• Execute AWP-ODC simulation for event=chino_uf and cvm=cvmh

% qsub ${CVMTEST_HOME}./pbs/kraken/awp_chino_uf_cvmh.pbs

• Extract synthetic seimsograms from AWP and perform GoF calculation

% qsub ${CVMTEST_HOME}/pbs/kraken/gof_chino_uf_cvmh.pbs

• Plot results

% cd ${CVMTEST_HOME}/tools/plotting
% ./DrawPlots.py chino_uf cvmh

• Package results

% cd ${CVMTEST_HOME}/tools/presentation
% ./PackageWork.py ${CVMTEST_HOME}/work ../../web_results.tar

The test results and plots in web_results.tar may then be transferred to a webhost and unpacked for public release.

Installing a New Earthquake Scenario

• Edit ${CVMTEST_HOME}/config/python/ATFConfig.py to add your new event, event tag, event description, and mesh dimensions.
• Create the following new directories:

${CVMTEST_HOME}/awp/${NEW_EVENT_TAG}/${CVM} (one for each CVM defined in ATFConfig.py)
${CVMTEST_HOME}/config/topo/${NEW_EVENT_TAG}
${CVMTEST_HOME}/config/cvm2mesh/${NEW_EVENT_TAG}
${CVMTEST_HOME}/config/stations/${NEW_EVENT_TAG}

where ${NEW_EVENT_TAG} is the new event's tag

• Download and unpack DEMs covering the entire meshing region for the new scenario into ${CVMTEST_HOME}/dem. Download 1/3 arcsec in GridFloat format. The USGS seamless server places limits on how much elevation data you may request, therefore limit requests to boxes of 1 deg lat/lon.
• Add the new DEM directories to the list of DEMs searched by ${CVMTEST_HOME}/topo/ExtractDEM.py. For example, add appropriate lines such as the following in main():

demlist.append('../../dem/NED_28288029/ned_28288029')

• Generate a topography file for the event with ${CVMTEST_HOME}/tools/topo/ExtractDEM.py:

% cd ${CVMTEST_HOME}/tools/topo
% ./ExtractDEM.py ${NEW_EVENT_TAG}

• Retrieve event hypocenter from SCEC DC and save in ${CVMTEST_HOME}/config/stations/${NEW_EVENT_TAG}/${NEW_EVENT_TAG}_event.txt
• Retrieve complete CISN station list from SCEC DC http://www.data.scec.org/stationinfo.html (or copy existing list from ${CVMTEST_HOME}/config/stations/chino_uf/cisn_stations.txt) to ${CVMTEST_HOME}/config/stations/${NEW_EVENT_TAG}/cisn_stations.txt
• Define list of cities within the meshing region and save to ${CVMTEST_HOME}/config/station/${NEW_EVENT_TAG}/socal_cities.txt
• Generate a new event station list for ${NEW_EVENT_TAG} with the CISN2XY.py script:

% cd ${CVMTEST_HOME}/tools/stations
% ./CISN2XY.py ${NEW_EVENT_TAG}

Installing a New CVM

Installing a new CVM into the test framework is somewhat involved. The general steps needed to perform this are:

• Install copy of CVM on the local host. If using CruiseControl, update CC to build the CVM and run the unit tests
• Edit ${CVMTEST_HOME}/remote/AcceptTest.py on the local host to change the default CVM used in the testing
• Edit cvm2mesh to support the new CVM
• Edit ${CVMTEST_HOME}/config/python/ATFConfig.py on the remote host to add the new CVM tag and properties
• Create PBS job submit scripts in ${CVMTEST_HOME}/pbs/kraken for building the CVM, cvm2mesh, mesh extraction, AWP sim, GoF calculation, and plotting

Known Issues

None.

viz-cvm

The viz-cvm package contains scripts for plotting slices and profiles from either CVM-H, CVM-4, or 3D meshes.

Requirements

• Python 2.6+, with numpy, matplotlib, and basemap modules
• SCEC CVM-4 [Optional] http://earth.usc.edu/~gely/coseis/www/index.html Set ibig var in newin.h to 9,000,000+ and recompile using the following Makefile.

integer, parameter :: ibig = 10000000

# CVM-4 Makefile
all: cvm4_txt cvm4_bin cvm4_mpi
cvm4_txt: cvm4.f iotxt.f
       gfortran -Wall -O3 cvm4.f iotxt.f -o cvm4_txt
cvm4_bin: cvm4.f iobin.f
       gfortran -Wall -O3 cvm4.f iobin.f -o cvm4_bin
cvm4_mpi: cvm4.f iompi.f
       gfortran -Wall -O3 cvm4.f iobin.f -o cvm4_mpi
clean:
       rm *.o *~ cvm4_txt cvm4_bin cvm4_mpi

• SCEC CVM-H [Optional] https://source.usc.edu/svn/cvmh

Installation

• If you wish to plot slices from CVM-H/CVM-4, the CVMs must be installed (see requirements).
• Within your CVM-4 installation directory, install the following shell script, run_cvm4_gely.sh:

#!/bin/bash
# CVM-4 helper script that abstracts stdin, stdout redirection
IN_FILE=$1
OUT_FILE=$2
CVMBIN=cvm_txt
./${CVMBIN} < ${IN_FILE} > ${OUT_FILE}
if [ $? -ne 0 ]; then
    exit 1
fi
exit 0

An equivalent script for CVM-H is already present in the official CVM-H distribution.

• Check out the viz-cvm distribution from https://source.usc.edu/svn/viz-cvm or unpack the pre-packaged tar file.
• Update CVM-H/CVM-4 installation paths in Slice.py:

# CVM Constants
cvmh_dir = '/home/rcf-104/patrices/cvmh/trunk/bin'
cvmh_bin = './run_vx_lite.sh'
cvm4_dir = '/home/rcf-104/patrices/utils/cvm4_gely'
cvm4_bin = 'run_cvm4_gely.sh'

Plotting from a CVM

Supported plot types include horizontal slices from either CVM, horizontal difference (CVM-H - CVM-4) slice, and profile slices from either CVM or both.

./Slice.py <map_type> <outfile> <map_params>

where:

map_type

hor/prof/hordiff

outfile

Name of PNG plot

map_params

with hor map_type: depth, value_type, source (CVM-H, CVM-4, Both)

with hordiff map_type: depth, value_type

with prof map_type: lon1, lat1, lon2, lat2, value_type, source (CVM-H, CVM-4, Both)

value_type

Vp, Vs, Rho, Topo, Vs30, Z2500

Plotting from a mesh

Supported plot types include horizontal slices and profile slices from either SORD or AWP formatted meshes.

./PlotGrid.py <map_type> <mesh_file> <outfile> <title> <map_params>

where:

map_type

hor, prof

mesh_file

Name of the SORD or AWP formatted mesh.

outfile

Name of PNG plot

title

Title for the plot

map_params

with hor map_type: k_offset, imax, jmax, kmax, decimation, source, value_type

with prof map_type: imax, jmax, kmax, p1_x, p1_y, p2_x, p2_y, depth, decimation, source, value_type

value_type

Vp, Vs, Rho

source

IJK-32, IJK-20, IJK-12, SORD

Known Issues

Small scale slices or highly decimated slices may take on a pixellated look. This is due to a low pixel density in the plots. The work-around is to reduce the decimation factor or increase the size of the slice.