Difference between revisions of "UCVM User Guide"

From SCECpedia
Jump to navigationJump to search
(Redirected page to UCVM 14.3.0 User Guide)
 
(91 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
#REDIRECT [[UCVM 14.3.0 User Guide]]
 +
 
= Overview =
 
= Overview =
  
[[UCVM]] is a software framework for creating and distributing a state-wide California velocity model.
+
The Unified Community Velocity Model Framework [[UCVM]] is a software package which provides a standard interface to several regional community velocity models. Any differences in coordinate projections within each model are hidden so that the researcher queries any of the models by longitude, latitude, and Z (where Z is depth from surface or elevation relative MSL). UCVM also provides several other unique capabilities and products, including:
 +
 
 +
* Seamlessly combine two or more models into a composite model for querying
 +
* Optionally include a California statewide geotechnical layer and interpolate it with the crustal velocity models
 +
* Extract a 3D mesh or CVM Etree (octree) of material properties from any combination of models
 +
* Standard California statewide elevation and Vs30 data map is provided
 +
* Miscellaneous tools are provided for creating 2D etree maps, and optimizing etrees
 +
* Numerically smooth discontinuities at the interfaces between different regional models
 +
* Add support for future velocity models with the extendable interface
 +
 
 +
 
 +
Current list of supported velocity models: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, SCEC CVM-NCI, Magistrale WFCVM, USGS CenCalVM, Graves Cape Mendocino, Lin-Thurber Statewide, Tape SoCal.
 +
 
 +
The API itself is written in the C language. The software is MPI-capable, and has been shown to scale to 3300+ cores on NICS Kraken in a mesh generator. It allows application programs to tailor which set of velocity models are linked in to minimize memory footprint. It is also compatible with grid resources that have data stage-in policies for jobs, and light-weight kernels that support static libraries only. This package has been tested on NCCS Jaguar, NICS Kraken, and USC HPCC.
  
  
Line 7: Line 22:
  
 
The system requirements are as follows:
 
The system requirements are as follows:
# UNIX operating system (Linux, Solaris, MacOS)
+
# UNIX operating system (Linux)
 
# GNU gcc/gfortran compilers (MPI wrappers such as mpicc are OK)
 
# GNU gcc/gfortran compilers (MPI wrappers such as mpicc are OK)
 
# tar for opening the compressed files
 
# tar for opening the compressed files
# Euclid Etree library [http://www.cs.cmu.edu/~euclid/ Euclid Etree]
+
# [http://www.cs.cmu.edu/~euclid/ Euclid Etree library] (recommend [http://hypocenter.usc.edu/research/ucvm/euclid3-1.3.tar.gz euclid3-1.3.tar.gz])
# Proj.4 projection library [http://trac.osgeo.org/proj/ Proj.4]
+
# [http://trac.osgeo.org/proj/ Proj.4 Projection Library] (recommend [http://download.osgeo.org/proj/proj-4.7.0.tar.gz proj-4.7.0.tar.gz])
  
 +
 +
Optional dependencies include any of the following standard velocity models and packages:
 +
 +
# Standard community velocity models: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, SCEC CVM-NCI, Magistrale WFCVM, USGS CenCalVM, Graves Cape Mendocino, Lin-Thurber Statewide, Tape SoCal
 +
# NetCDF (network Common Data Form) library [http://www.unidata.ucar.edu/downloads/netcdf/index.jsp NetCDF]
  
 
= Installation =
 
= Installation =
Line 20: Line 40:
 
#Start at SCEC website: [http://scec.usc.edu/scecpedia/UCVM http://scec.usc.edu/scecpedia/UCVM]
 
#Start at SCEC website: [http://scec.usc.edu/scecpedia/UCVM http://scec.usc.edu/scecpedia/UCVM]
 
#Navigate to the Downloads section of the [[UCVM]] web page.
 
#Navigate to the Downloads section of the [[UCVM]] web page.
#Click the download link to download the latest source code distribution file and any listed dependencies. This is typically posted as a tar and gzipped file (tgz format). The source code distribution file is large (280 Mb), so the download make take awhile.
+
#Click the download link to download the latest source code distribution file and any listed dependencies. This is typically posted as a tar and gzipped file (tgz format). The source code distribution file is large (350 Mb), so the download make take awhile.
  
  
 
== CheckSum Test ==
 
== CheckSum Test ==
  
Not currently implemented.
+
Verify that the provided md5 checksum file matches the computed md5 for your downloaded tarball:
 +
 
 +
<pre>
 +
$ md5sum -c ucvm-12.2.0.tar.gz.md5
 +
</pre>
  
  
Line 42: Line 66:
  
 
For the Euclid Etree library:
 
For the Euclid Etree library:
% tar xvf euclid-latest.tar
+
<pre>
% cd ./euclide-3.12/libsrc
+
$ tar zxvf euclid3-1.3.tar.gz
% make all
+
$ cd ./euclid3-1.3/libsrc
 +
$ make all
 +
</pre>
  
If installing the Etree library on a Lustre filesystem (NICS Kraken, TACC Ranger, NCCS Jaguar), you must enable IOBUF. Please see [[Cvmbycols#Technical_Note_for_Lustre_Filesystems]] for special installation instructions.
+
If installing the Etree library on a Lustre filesystem on a Cray system (NICS Kraken, NCCS Jaguar), you must enable the Cray IOBUF module prior to compiling the library. Also,
 +
edit the ./libsrc/Makefile to add the compile flags "-DUSE_IOBUF -DUSE_IOBUF_LOCAL_MACROS".
  
For the Proj.4 library:
+
For the Proj.4 library (version 4.7.0):
% tar xvf proj-4.7.0.tar
+
<pre>
% cd proj-4.7.0
+
$ tar xvf proj-4.7.0.tar
% ./configure --prefix=<your desired install directory>
+
$ cd proj-4.7.0
% make; make install
+
$ ./configure --prefix=<your desired install directory>
 +
$ make; make install
 +
</pre>
  
 
Be sure to note where you installed these two packages. You will need this information later in the installation process.
 
Be sure to note where you installed these two packages. You will need this information later in the installation process.
  
Then, install the velocity models that you wish to use with UCVM. Ensure they are compiled with gcc/gfortran.
+
Then, install the velocity models that you wish to use with UCVM according to the documentation provided by those models. Ensure they are compiled with gcc/gfortran.
  
 
For the SCEC CVM-H model:
 
For the SCEC CVM-H model:
% tar xvf cvmh-11.9.0.tgz
+
<pre>
% cd cvmh
+
$ tar xvf cvmh-11.9.1.tar.gz
% make all
+
$ cd cvmh-11.9.1
 +
$ ./configure --prefix=<your desired_install directory>
 +
$ make; make install
 +
</pre>
  
 
For the SCEC CVM-S model:
 
For the SCEC CVM-S model:
% tar xvf cvms-11.9.0.tgz
+
<pre>
% cd cvms/src
+
$ tar xvf cvms-11.11.0.tar.gz
% make all
+
$ cd cvms/src
 +
$ make all
 +
</pre>
  
 
Again, be sure to note where you installed the community velocity models.
 
Again, be sure to note where you installed the community velocity models.
  
Declare environment variables with the paths to the Etree library, Proj.4 library, and velocity models that you previously installed.
+
Declare environment variables with the paths to the Etree library, Proj.4 library, and velocity models that you previously installed. Also, update your LD_LIBRARY_PATH environment variable to include the new library directories.
  
ETREE_DIR=<Etree install path>
+
<pre>
PROJ4_DIR=<Proj.4 install path>
+
ETREE_DIR=<Etree install path>
CVMH_DIR=<CVM-H install path>
+
PROJ4_DIR=<Proj.4 install path>
CVMS_DIR=<CVM-S install path>
+
CVMH_DIR=<CVM-H install path>
 +
CVMS_DIR=<CVM-S install path>
 +
LD_LIBRARY_PATH=${ETREE_DIR}/libsrc:${PROJ4_DIR}/lib:${CVMH_DIR}/lib:${CVMS_DIR}/lib:${LD_LIBRARY_PATH}
 +
UCVM_INSTALL_DIR=<desired UCVM install path>
 +
</pre>
  
 
In a bash shell, for example, the ETREE_DIR environment variable can be declared with the command:
 
In a bash shell, for example, the ETREE_DIR environment variable can be declared with the command:
 
+
<pre>
% ETREE_DIR=/opt/etree;export ETREE_DIR
+
$ ETREE_DIR=/opt/etree;export ETREE_DIR
 +
</pre>
  
 
In a csh shell:
 
In a csh shell:
 
+
<pre>
% setenv ETREE_DIR /opt/etree
+
$ setenv ETREE_DIR /opt/etree
 +
</pre>
  
 
Now unpack the UCVM software distribution:
 
Now unpack the UCVM software distribution:
% tar zxvf ucvm-11.9.0.tar.gz
+
<pre>
 +
$ tar zxvf ucvm-12.2.0.tar.gz
 +
</pre>
  
 
The build configuration differs slightly depending on where you wish to run UCVM. On a standard Linux system with gcc, run the following
 
The build configuration differs slightly depending on where you wish to run UCVM. On a standard Linux system with gcc, run the following
commands to build the software with CVM-H and CVM-S enabled:
+
commands to configure the software with CVM-H and CVM-S enabled:
 +
<pre>
 +
$ cd ucvm-12.2.0
 +
$ ./configure --prefix=${UCVM_INSTALL_DIR} --with-etree-include-path="${ETREE_DIR}/libsrc"
 +
--with-etree-lib-path="${ETREE_DIR}/libsrc" --with-proj4-include-path="${PROJ4_DIR}/include"
 +
--with-proj4-lib-path="${PROJ4_DIR}/lib" --enable-model-cvmh --enable-model-cvms
 +
--with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib"
 +
--with-gctpc-lib-path="${CVMH_DIR}/lib" --with-cvms-include-path="${CVMS_DIR}/include"
 +
--with-cvms-lib-path="${CVMS_DIR}/lib" --with-cvmh-model-path="${CVMH_DIR}/model" --with-cvms-model-path="${CVMS_DIR}/src"
 +
</pre>
  
% cd ucvm-11.9.0
+
The above example enables CVM-H and CVM-S but any of the following models may be enabled: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, USGS CenCalVM, and WFCVM. For each velocity model you want to use, the configure script needs to be told to explicitly use that model (with the --enable-model-* flag) and where to find the velocity model's library files (.a or .so), header files (.h), and model files so that they can be linked into UCVM.
% ./configure --prefix=${UCVM_INSTALL_DIR} --with-etree-include-path="${ETREE_DIR}/libsrc"
 
--with-etree-lib-path="${ETREE_DIR}/libsrc" --with-proj4-include-path="${PROJ4_DIR}/include"
 
--with-proj4-lib-path="${PROJ4_DIR}/lib" --with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib"
 
--with-gctpc-lib-path="${CVMH_DIR}/gctpc/lib" --with-cvms-include-path="${CVMS_DIR}/include" --with-cvms-lib-path="${CVMS_DIR}/lib"
 
--with-cvmh-model-path="${CVMH_DIR}/bin" --with-cvms-model-path="${CVMS_DIR}/src"
 
% make
 
  
Where "${UCVM_INSTALL_DIR}" is the target install directory of your choice. You may define it as an enviroment variable or just specify it directly as the argument to prefix. The above example used CVM-H and CVM-S but any of the following models may be enabled: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, USGS CenCalVM, and WFCVM. For each velocity model you want to use, the configure script needs to know where to find the velocity model's library files (.a or .so), header files (.h), and model files so that they can be linked into UCVM.  
+
Note that unless you force static linking with the "--enable-static" command line option, you must update your LD_LIBRARY_PATH environment variable with the paths to the Etree, Proj.4, and model shared libraries.
  
Check that the package was built correctly with the following command:
+
The configure command is quite long, so an example shell script with this command is located in ./scripts/autoconf/basic_install.sh. It assumes that you previously defined the environment variables described in this installation guide. The script can be executed with the following shell commands:
 +
<pre>
 +
$ cd scripts/autoconf
 +
$ ./basic_install.sh
 +
</pre>
  
% make check
+
Compile the software by executing the following command in the main directory:
 +
<pre>
 +
$ make
 +
</pre>
  
Verify that all unit tests and acceptance tests pass.
+
Check that the package was built correctly by executing the following command in the main directory:
 +
<pre>
 +
$ make check
 +
</pre>
  
Install the software to the desired target location with the following command:
+
Verify that all unit tests and acceptance tests pass. Depending on your system and your UCVM configuration, testing may take up to 15-30 minutes.
  
% make install
+
Install the software to the desired target location by executing the following command in the main directory:
 +
<pre>
 +
$ make install
 +
</pre>
  
 +
A full list of supported standard velocity models and options can be listed with the command:
 +
<pre>
 +
$ ./configure --help
 +
</pre>
  
A full list of supported standard velocity models and options can be listed with the command
+
== Updating the Configuration ==
  
% ./configure --help
+
The UCVM installer will automatically create a configuration file in ${UCVM_INSTALL_DIR}/conf/ucvm.conf with entries for all enabled velocity models and some standard settings. However, you may wish to edit this file to toggle model flags or to add new user models in the future. For example, the CVM-H model has a number of flags that may be set in the UCVM configuration file. Please see [[UCVM_User_Guide#Framework_Configuration]] for more details.
  
  
 
== Advanced Install ==
 
== Advanced Install ==
  
Additional options must be passed to the ./configure script on a more exotic Linux system, such as a host that requires static linking (NICS Kraken), or has a Lustre high-performace filesystem (TACC Ranger). The following example shows how to enable static linking and IOBUF buffering:
+
Additional options must be passed to the ./configure script on a more exotic Linux system, such as a host that requires static linking (NICS Kraken), or has a Lustre high-performace filesystem (NCCS Jaguar, NICS Kraken). The following example shows how to enable static linking and Cray IOBUF buffering:
 +
<pre>
 +
$ module add iobuf
 +
$ ./configure --prefix=${UCVM_INSTALL_DIR} --enable-static --enable-iobuf --enable-model-cvmh --enable-model-cvms
 +
--with-etree-include-path="${ETREE_DIR}/libsrc" --with-etree-lib-path="${ETREE_DIR}/libsrc"
 +
--with-proj4-include-path="${PROJ4_DIR}/include" --with-proj4-lib-path="${PROJ4_DIR}/lib"
 +
--with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib"
 +
--with-gctpc-lib-path="${CVMH_DIR}/lib" --with-cvms-include-path="${CVMS_DIR}/include"
 +
--with-cvms-lib-path="${CVMS_DIR}/lib" --with-cvmh-model-path="${CVMH_DIR}/model" --with-cvms-model-path="${CVMS_DIR}/src" CC=cc
 +
$ make; make check; make install
 +
</pre>
  
% cd ucvm-11.9.0
+
<p>It is generally advisable to compile all components of UCVM using the same compilation system. NICS Kraken, for example, uses the Cray compiler wrappers (cc not gcc). As such, on this system, you must use CC to compile the Euclid library.</p>
% module add iobuf
 
% ./configure --prefix=${UCVM_INSTALL_DIR} --enable-static --enable-iobuf --with-etree-include-path="${ETREE_DIR}/libsrc"
 
--with-etree-lib-path="${ETREE_DIR}/libsrc" --with-proj4-include-path="${PROJ4_DIR}/include"
 
--with-proj4-lib-path="${PROJ4_DIR}/lib" --with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib"
 
--with-gctpc-lib-path="${CVMH_DIR}/gctpc/lib" --with-cvms-include-path="${CVMS_DIR}/include" --with-cvms-lib-path="${CVMS_DIR}/lib"
 
--with-cvmh-model-path="${CVMH_DIR}/bin" --with-cvms-model-path="${CVMS_DIR}/src" CC=cc
 
% make; make check; make install
 
  
 
A few notes:
 
A few notes:
 
* Some systems require applications to be linked statically (example above).
 
* Some systems require applications to be linked statically (example above).
* Lustre filesystems require linking with IOBUF module (NICS Kraken example above).
+
* Lustre filesystems may require linking with Cray IOBUF module (NICS Kraken example above).
 
* The configure script needs to be told where to find the library file(s), the header file(s), and the model files for each velocity model that you wish to enable. The configure script will check that the libraries and headers are present, and it will setup UCVM with that model enabled and properly configured.
 
* The configure script needs to be told where to find the library file(s), the header file(s), and the model files for each velocity model that you wish to enable. The configure script will check that the libraries and headers are present, and it will setup UCVM with that model enabled and properly configured.
 
* The script will attempt to auto-detect a GNU compliant compiler. If an MPI wrapper is found, the MPI version of UCVM is compiled, otherwise the serial version is built. The compiler may be overridden with the option 'CC="mpicc"' for example.
 
* The script will attempt to auto-detect a GNU compliant compiler. If an MPI wrapper is found, the MPI version of UCVM is compiled, otherwise the serial version is built. The compiler may be overridden with the option 'CC="mpicc"' for example.
Line 141: Line 206:
 
<pre>
 
<pre>
 
   --enable-static        enable static linking
 
   --enable-static        enable static linking
 +
  --enable-netcdf        enable netCDF module
 
   --enable-iobuf          enable IOBUF module
 
   --enable-iobuf          enable IOBUF module
 +
  --enable-model-cvmh    enable model SCEC CVM-H
 +
  --enable-model-cvms    enable model SCEC CVM-S
 +
  --enable-model-cencal  enable model USGS CenCalVM
 +
  --enable-model-cvmsi    enable model SCEC CVM-SI
 +
  --enable-model-cvmnci  enable model SCEC CVM-NCI
 +
  --enable-model-wfcvm    enable model WFCVM
 +
  --enable-model-cvmlt    enable model Lin-Thurber
 +
  --enable-model-cmrg    enable model CMRG
 +
  --enable-model-tape    enable model Tape
  
 
   --with-etree-include-path
 
   --with-etree-include-path
 
                           location of the Etree headers
 
                           location of the Etree headers
 
   --with-etree-lib-path  location of the Etree libraries
 
   --with-etree-lib-path  location of the Etree libraries
 
 
   --with-proj4-include-path
 
   --with-proj4-include-path
 
                           location of the Proj.4 headers
 
                           location of the Proj.4 headers
 
   --with-proj4-lib-path  location of the Proj.4 libraries
 
   --with-proj4-lib-path  location of the Proj.4 libraries
 
+
  --with-netcdf-include-path
 +
                          location of the netCDF headers
 +
  --with-netcdf-lib-path  location of the netCDF libraries
 
   --with-cencal-include-path
 
   --with-cencal-include-path
 
                           location of the USGS CenCalVM headers
 
                           location of the USGS CenCalVM headers
Line 158: Line 234:
 
   --with-cencal-extmodel-path
 
   --with-cencal-extmodel-path
 
                           location of the USGS CenCalVM extended etree
 
                           location of the USGS CenCalVM extended etree
 
 
   --with-cvmh-include-path
 
   --with-cvmh-include-path
 
                           location of the SCEC CVM-H headers
 
                           location of the SCEC CVM-H headers
 
   --with-cvmh-lib-path    location of the SCEC CVM-H libraries
 
   --with-cvmh-lib-path    location of the SCEC CVM-H libraries
 
   --with-cvmh-model-path  location of the SCEC CVM-H model files
 
   --with-cvmh-model-path  location of the SCEC CVM-H model files
   --with-gctpc-lib-path  location of the CVM-H GCTPC libraries
+
   --with-gctpc-lib-path  location of the CVM-H GCTPC libraries (CVM-H 11.2.0)
 
 
 
   --with-cvms-include-path
 
   --with-cvms-include-path
 
                           location of the SCEC CVM-S headers
 
                           location of the SCEC CVM-S headers
 
   --with-cvms-lib-path    location of the SCEC CVM-S libraries
 
   --with-cvms-lib-path    location of the SCEC CVM-S libraries
 
   --with-cvms-model-path  location of the SCEC CVM-S model files
 
   --with-cvms-model-path  location of the SCEC CVM-S model files
 
 
   --with-cvmsi-include-path
 
   --with-cvmsi-include-path
 
                           location of the SCEC CVM-SI headers
 
                           location of the SCEC CVM-SI headers
 
   --with-cvmsi-lib-path  location of the SCEC CVM-SI libraries
 
   --with-cvmsi-lib-path  location of the SCEC CVM-SI libraries
 
   --with-cvmsi-model-path location of the SCEC CVM-SI model files
 
   --with-cvmsi-model-path location of the SCEC CVM-SI model files
 
+
  --with-cvmnci-include-path
 +
                          location of the SCEC CVM-NCI headers
 +
  --with-cvmnci-lib-path  location of the SCEC CVM-NCI libraries
 +
  --with-cvmnci-model-path
 +
                          location of the SCEC CVM-NCI model files
 
   --with-wfcvm-include-path
 
   --with-wfcvm-include-path
 
                           location of the WFCVM headers
 
                           location of the WFCVM headers
 
   --with-wfcvm-lib-path  location of the WFCVM libraries
 
   --with-wfcvm-lib-path  location of the WFCVM libraries
 
   --with-wfcvm-model-path location of the WFCVM model files
 
   --with-wfcvm-model-path location of the WFCVM model files
 +
  --with-cvmlt-include-path
 +
                          location of the CVMLT headers
 +
  --with-cvmlt-lib-path  location of the CVMLT libraries
 +
  --with-cvmlt-model-path location of the CVMLT model files
 +
  --with-cmrg-include-path
 +
                          location of the CMRG headers
 +
  --with-cmrg-lib-path    location of the CMRG libraries
 +
  --with-cmrg-model-path  location of the CMRG model files
 +
  --with-tape-include-path
 +
                          location of the TAPE headers
 +
  --with-tape-lib-path    location of the TAPE libraries
 +
  --with-tape-model-path  location of the TAPE model files
 +
</pre>
 +
 +
== Unit and Acceptance Tests ==
 +
 +
Both the unit tests and acceptance tests may be executed with the following command from the main directory:
 +
 +
<pre>
 +
$ make check
 
</pre>
 
</pre>
  
 +
All tests should result in a PASS.
 +
 +
 +
== Trouble-shooting ==
 +
 +
=== Shared Library Paths ===
 +
 +
If you see an error similar to the following while running either the tests or the UCVM programs:
  
== Unit Tests ==
+
error while loading shared libraries: libsomelib.so: cannot open shared object file: No such file or directory
  
Unit tests may be executed with the command:
+
This indicates that UCVM was linked against one or more shared libraries and the dynamic library loader cannot find the actual .so library at run-time. The solution is to update your LD_LIBRARY_PATH to include the directory containing the library mentioned in the error message. For example, the following command adds a new search directory to LD_LIBRARY_PATH in a csh shell:
 +
<pre>
 +
$ setenv LD_LIBRARY_PATH /home/USER/opt/somepkg/lib:${LD_LIBRARY_PATH}
 +
</pre>
  
% make check
+
Alternatively, if you do not want to work with shared libraries, UCVM can be compiled statically with the "--enable-static" configuration option.
  
All tests should result in a PASS.
+
=== Proj. 4 Error: major axis or radius = 0 or not given ===
  
 +
On systems with home filesystems that are not viewable to compute nodes (such as NICS Kraken), you may encounter errors with Proj.4 when trying to run any component of UCVM on compute nodes. This is due to the fact that Proj.4 actually relies on a file called proj_defs.dat which is located in the ${MAKE_INSTALL_LOCATION}/share/proj directory. So for example, suppose you had configured Proj.4 with ./configure --prefix=/not_viewable_to_compute_nodes/proj-4.7.0, Proj. 4 would then search for proj_defs.dat in /not_viewable_to_compute_nodes/proj-4.7.0/share/proj/proj_defs.dat. This will cause UCVM to throw the error "Proj.4 Error: major axis or radius = 0 or not given" and your job will fail.
  
== Acceptance Tests ==
+
The only way it seems to solve this issue is to actually make sure your --prefix directory is actually visible to the compute nodes and do a make install there. Documentation suggests that you can set the PROJ_LIB environment variable, however this seems to not work correctly without modifications to the Proj.4 source code.
  
Acceptance tests may be executed with the command:
+
=== Building Against CVM-H 11.2.0 ===
  
% make check
+
UCVM is designed to work with CVM-H 11.9.1 by default, but it can optionally be built with CVM-H 11.2.0. Use this modified configure command to configure the package with CVM-H 11.2.0 and CVM-S:
  
All tests should result in a PASS.
+
<pre>
 +
$ cd ucvm-12.2.0
 +
$ ./configure --prefix=${UCVM_INSTALL_DIR} --with-etree-include-path="${ETREE_DIR}/libsrc"
 +
--with-etree-lib-path="${ETREE_DIR}/libsrc" --with-proj4-include-path="${PROJ4_DIR}/include"
 +
--with-proj4-lib-path="${PROJ4_DIR}/lib" --enable-model-cvmh --enable-model-cvms
 +
--with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib"
 +
--with-gctpc-lib-path="${CVMH_DIR}/gctpc/lib" --with-cvms-include-path="${CVMS_DIR}/include"
 +
--with-cvms-lib-path="${CVMS_DIR}/lib" --with-cvmh-model-path="${CVMH_DIR}/bin" --with-cvms-model-path="${CVMS_DIR}/src"
 +
CFLAGS="-D_UCVM_MODEL_CVMH_11_2_0"
 +
</pre>
  
 +
Note that the CVM-H model and gctpc paths have changed, and there is now a CFLAGS argument. After configuring, you may compile, check, and install normally.
  
 
= Framework Configuration =
 
= Framework Configuration =
Line 213: Line 332:
 
# Change to reflect UCVM installation directory.
 
# Change to reflect UCVM installation directory.
 
#
 
#
ucvm_modelpath=/home/scec-00/USER/opt/aftershock/ucvm-11.9.0/model/ucvm
+
ucvm_interface=map_etree
 +
ucvm_mappath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/ucvm/ucvm.e
  
  
Line 222: Line 342:
 
#
 
#
 
# SCEC CVM-H
 
# SCEC CVM-H
cvmh_modelpath=/home/scec-00/USER/opt/aftershock/cvmh/bin
+
cvmh_modelpath=/home/scec-00/USER/opt/aftershock/cvmh-11.9.1/model
 
#
 
#
 
# USGS Bay Area high-rez and extended etrees
 
# USGS Bay Area high-rez and extended etrees
Line 229: Line 349:
 
#
 
#
 
# SCEC CVM-SI
 
# SCEC CVM-SI
cvmsi_modelpath=/home/scec-00/USER/opt/aftershock/cvmsi/model
+
cvmsi_modelpath=/home/scec-00/USER/opt/aftershock/cvmsi/model/i8
 +
#
 +
# SCEC CVM-NCI
 +
cvmnci_modelpath=/home/scec-00/USER/opt/aftershock/cvmnci/model/i2
 
#
 
#
 
# Wasatch Front CVM
 
# Wasatch Front CVM
Line 235: Line 358:
 
#
 
#
 
# Lin-Thurber Statewide
 
# Lin-Thurber Statewide
lin-thurber_modelpath=/home/scec-00/USER/opt/aftershock/ucvm-11.9.0/model/lin-thurber
+
cvmlt_modelpath=/home/scec-00/USER/opt/aftershock/cvmlt/model
 
#
 
#
 
# Cape Mendocino RG
 
# Cape Mendocino RG
cmrg_modelpath=/home/scec-00/USER/opt/aftershock/ucvm-11.9.0/model/cmrg
+
cmrg_modelpath=/home/scec-00/USER/opt/aftershock/cvm-cmrg/model/cmrg.conf
 +
#
 +
# Tape SoCal Model
 +
tape_modelpath=/home/scec-00/USER/opt/aftershock/cvm-tape/model/m16
 +
#
 +
# 1D
 +
1d_modelpath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/1d/1d.conf
 +
#
 +
# 1D GTL
 +
1dgtl_modelpath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/1d/1d.conf
  
  
Line 261: Line 393:
 
# Optional Yong-Wald vs30 map
 
# Optional Yong-Wald vs30 map
 
yong_interface=map_etree
 
yong_interface=map_etree
yong_mappath=/home/scec-00/USER/opt/aftershock/ucvm-11.9.0/model/ucvm/ucvm_yong_wald.e
+
yong_mappath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/ucvm/ucvm_yong_wald.e
  
  
 
# Model flags
 
# Model flags
 
#
 
#
#cvmh_param=USE_1D_BKG,True
+
cvmh_param=USE_1D_BKG,True
 +
cvmh_param=USE_GTL,True
 
</pre>
 
</pre>
  
Line 279: Line 412:
 
! Description
 
! Description
 
|-
 
|-
| ucvm_modelpath
+
| ucvm_interface
| ${UCVM_INSTALL_DIR}/model/ucvm
+
| map_etree
| Directory containing the default projection specification and UCVM DEM/Vs30 Etree map.
+
| The interface code needed to parse the UCVM map
 +
|-
 +
| ucvm_mappath
 +
| ${UCVM_INSTALL_DIR}/model/ucvm/ucvm.e
 +
| String identifying location of the UCVM map Etree
 
|-
 
|-
 
| *_modelpath
 
| *_modelpath
Line 301: Line 438:
 
| *_param
 
| *_param
 
| Flag,Value
 
| Flag,Value
| Set a model-specific configuration Flag to Value. The only defined flag currently is USE_1D_BKG for CVM-H, for toggling the 1D background model.  
+
| Set a model-specific configuration Flag to Value. The defined flags are USE_1D_BKG and USE_GTL for CVM-H, for toggling the 1D background model and GTL.  
 
|}
 
|}
  
Line 330: Line 467:
 
| cencal
 
| cencal
 
| USGS Bay Area CenCalVM
 
| USGS Bay Area CenCalVM
 +
| External, installed separately and linked at install
 +
| 1
 +
|-
 +
| cvmnci
 +
| SCEC CVM-NCI
 
| External, installed separately and linked at install
 
| External, installed separately and linked at install
 
| 1
 
| 1
Line 348: Line 490:
 
| 1
 
| 1
 
|-
 
|-
| lin-thurber
+
| lt
 
| Lin-Thurber California Statewide
 
| Lin-Thurber California Statewide
| Internal, included in UCVM and automatically linked
+
| External, installed separately and linked at install
 
| 1
 
| 1
 
|-
 
|-
 
| cmrg
 
| cmrg
 
| Cape Mendocino Rob Graves
 
| Cape Mendocino Rob Graves
| Internal, included in UCVM and automatically linked
+
| External, installed separately and linked at install
 +
| 1
 +
|-
 +
| tape
 +
| Carl Tape SoCal
 +
| External, installed separately and linked at install
 +
| 1
 +
|-
 +
| cmuetree
 +
| CMU CVM Etree
 +
| Internal, included in UCVM, user specifies etree path in ./conf/ucvm.conf
 
| 1
 
| 1
 
|}
 
|}
Line 390: Line 542:
 
|-
 
|-
 
| elygtl
 
| elygtl
| Ely Vs30-derived GTL
+
| Ely Vs30-derived GTL (Ely et al., 2010)
 
|-
 
|-
 
| 1dgtl
 
| 1dgtl
 
| Generic 1D, identical to 1d crustal model
 
| Generic 1D, identical to 1d crustal model
 
|}
 
|}
 +
 +
 +
Interpolation functions are used to smooth GTL material properties with the underlying crustal model material properties. This smoothing is performed over an interpolation zone along the Z axis. Interpolation functions can be assigned on a per-GTL basis. Two predefined interpolation functions are provided to the user:
 +
 +
{| class="wikitable" border="1"
 +
|-
 +
! Label
 +
! Interpolation Function
 +
|-
 +
| linear
 +
| Linear interpolation
 +
|-
 +
| ely
 +
| Ely interpolation relation (Ely et al., 2010)
 +
|}
 +
 +
 +
If the user enables a GTL model but does not specify an interpolation function, linear interpolation is used by default. If no interpolation zone is specified, a depth range of 0 m - 350m is used.
  
  
Line 425: Line 595:
 
| map_etree
 
| map_etree
 
| SCEC Map Etree
 
| SCEC Map Etree
 +
|}
 +
 +
 +
== Supported Model Flags ==
 +
 +
Underlying models may be configured in one of two ways: with a key/value string in the UCVM configuration file, or by passing configuration parameters with the <code>ucvm_setparam()</code>function. The following model flags are supported:
 +
 +
{| class="wikitable" border="1"
 +
|-
 +
! Parameter
 +
! Value
 +
! Models
 +
|-
 +
| USE_1D_BKG
 +
| True/False (default False)
 +
| CVM-H
 +
|-
 +
| USE_GTL
 +
| True/False (default True)
 +
| CVM-H
 
|}
 
|}
  
Line 465: Line 655:
  
  
For other models that either are represented by tables, or have their own native API, UCVM may be extended to query from these models. However, this requires code modifications to the core UCVM library located in ./src/ucvm and recompilation. The general steps are:
+
For other models that either are represented by tables, or have their own native API, UCVM may be extended to query from these models. However, this requires code modifications to the core UCVM library located in ./src/ucvm and recompilation. The general steps are:
  
 
* Define new model label UCVM_MODEL_MODELNAME in ucvm_dtypes.h
 
* Define new model label UCVM_MODEL_MODELNAME in ucvm_dtypes.h
Line 472: Line 662:
 
** Add ucvm_model_modelname.h include at top
 
** Add ucvm_model_modelname.h include at top
 
** Add model lookup to ucvm_add_model() function
 
** Add model lookup to ucvm_add_model() function
 +
** Add model to ucvm_get_resources() function
 
* Modify ./src/ucvm/Makefile.am:
 
* Modify ./src/ucvm/Makefile.am:
 
** Add ucvm_model_modelname.o to list of libucvm.a dependencies
 
** Add ucvm_model_modelname.o to list of libucvm.a dependencies
* Reconf Makefiles with ./scripts/autoconf/reconf.sh
+
* Modify ./configure.ac:
* Recompile UCVM, passing the following options to configure:
+
** Define "enable-modelname", "with-modelname-lib-path", and "with-modelname-incl-path" configure options
** Path to the native model header files with: CFLAGS='-I<hdr_path>'
+
** Add library and header checks for your new model
** Path to the native model library files with: LDFLAGS='-L<lib_path> -l<libname>"
+
* Modify installer to customize path to your new model in ucvm.conf:
 +
** Add your new model to ./conf/Makefile.am
 +
** Add config entry for your new model to ./conf/ucvm_template.conf
 +
* Regenerate the Makefiles with ./scripts/autoconf/reconf.sh
 +
* Reconfigure and recompile UCVM, making sure to pass your new model flags to the configure utility
  
  
Line 489: Line 684:
  
 
User-defined maps can be created with the ./bin/grd2etree utility.
 
User-defined maps can be created with the ./bin/grd2etree utility.
 
  
 
= Framework Description =
 
= Framework Description =
Line 495: Line 689:
 
== Concept of Operation ==
 
== Concept of Operation ==
  
UCVM combines multiple regional velocity models and two internal UCVM models into one "meta-model" for the purposes of querying Vp, Vs, and density. Individual model setup and query details are abstracted under a uniform interface, and the application sees only the meta-model. This simplifies application code and makes it easier for those programs to support new velocity models in the future.
+
=== Combining Velocity Models into a Composite Model ===
 +
 
 +
UCVM combines multiple regional velocity models, along with a DEM map and a Vs30 map, into one composite model for the purposes of querying surface elevation, Vs30, Vp, Vs, and density. Individual model setup and query details are abstracted under a uniform interface, and the application sees only the composite model. This simplifies application code and makes it easier for those programs to support new velocity models in the future.
 +
 
 +
[[File:UCVM_Tiling_Concept.png|256px|thumb|right|UCVM model tiling method.]]
 +
 
 +
Velocity models are combined into the composite by tiling them on top of one another. When UCVM is initialized, the user selects an ordered list of models to query for data. Query points are then submitted to each velocity model in that list, one model at a time in list order. The first model to return valid velocity data for the point is considered to have fullfilled that data request and subsequent models are not queried. Generally, no smoothing is performed at the interfaces between models (an exception is interpolation between a GTL and crustal model as described below). For each query point, the following data is returned: surface elevation, Vs30, Vp, Vs, and density. The figure at right illustrates how this tiling is performed.
 +
 
 +
Regional models may include: CVM-H, CVM-H, USGS Bay Area, 1D, or other user-defined models. Most models and maps have a maximum extent, outside of which no data is available. The exception is the generic 1D model as it has infinite extents. Applications must check the returned velocity and density values to ensure they are valid.
  
Velocity models are combined into the meta-model by tiling them on top of one another. When UCVM is initialized, the user selects an ordered list of models to query for data. Query points are then submitted to each velocity model in that list, one model at a time in list order. The first model to return valid velocity data for the point is considered to have fullfilled that data request and subsequent models are not queried. Generally, no smoothing is performed at the interfaces between models (an exception is interpolation between a GTL and crustal model as described below). For each query point, the following data is returned: surface elevation, Vs30, Vp, Vs, and density.
 
  
Regional models may include: CVM-H, CVM-H, USGS Bay Area, 1D, etc. Internal models include the UCVM DEM and Vs30 map. Most models, both regional and internal, have a maximum extent, outside of which no data is available. The exception is the generic 1D model as it has infinite extents. Applications must check the returned velocity and density values to ensure they are valid.
+
=== Querying with one or more GTLs ===
  
 
The API distinguishes between two types of regional velocity models: GTL models, and crustal models. GTL models are shallow near-surface velocity models called geotechnical layers. Crustal models are deeper velocity models such as tomographic models. When a user selects to query from one or more crustal models only, the API executes the previously described behavior. However, if one or more GTL models are selected, additional processing is performed.
 
The API distinguishes between two types of regional velocity models: GTL models, and crustal models. GTL models are shallow near-surface velocity models called geotechnical layers. Crustal models are deeper velocity models such as tomographic models. When a user selects to query from one or more crustal models only, the API executes the previously described behavior. However, if one or more GTL models are selected, additional processing is performed.
  
When a GTL model is given in the ordered list of models, the API creates a default interpolation zone along the z-axis over which interpolation between this GTL and the underlying crustal models is performed. It also assigns a default (linear) interpolation function to smooth the velocities of the GTL and underlying crustal models. Both the interpolation zone and interpolation function can be changed by the user. Querying for a point in this operating mode is now slightly different from the previous crustal-only case. As before, all query points are submitted to the crustal models in the ordered list of models (skipping any GTLs). However, the z coordinate of the points may be shifted down to the lower edge of the interpolation zone if the point falls within it. The models are queried with this possibly modified 3D coordinate and the velocities/density values are saved. The same list of query points is then submitted to the GTL models in the ordered list of models (skipping any crustals) and the z coordinate of those points within the interpolation zone may be shifted up to the top edge. The models are queried with this possibly modified 3D coordinate and the velocity/density values are saved. The list of query points is then traversed one final time. Those points that fall above the interpolation zone will have their velocities/densities set to those of the GTL as processed by the interpolation function assigned to that GTL. Those points that fall within the interpolation zone and have valid GTL/crustal values will have their GTL/crustal values smoothed according to the assigned interpolation function. Points that fall below the interpolation ozne and have valid crustal values will have their velocities/densities set to the crustal values. Note that points with invalid/incompatible velocities in either the GTL or crustal dataset cannot be smoothed and the interpolation results will be marked as invalid.
+
When a GTL model is given in the ordered list of models, the API creates a default interpolation zone along the z-axis over which interpolation between this GTL and the underlying crustal models is performed. It also assigns a default (linear) interpolation function to smooth the velocities of the GTL and underlying crustal models. Both the interpolation zone and interpolation function can be changed by the user.  
 +
 
 +
Querying for a point in this operating mode is now slightly different from the previous crustal-only case. As before, all query points are submitted to the crustal models in the ordered list of models (skipping any GTLs). However, the z coordinate of the points may be shifted down to the lower edge of the interpolation zone if the point falls within it. The crustal models are queried with this possibly modified 3D coordinate and the velocities/density values are saved. The same list of query points is then submitted to the GTL models in the ordered list of models (skipping any crustals) and the z coordinate of those points within the interpolation zone may be shifted up to the top edge. The GTL models are queried with this possibly modified 3D coordinate and the velocity/density values are saved.  
 +
 
 +
The list of query points is then traversed one final time. Those points that fall above the interpolation zone will have their velocities/densities set to those of the GTL as processed by the interpolation function assigned to that GTL. Those points that fall within the interpolation zone and have valid GTL/crustal values will have their GTL/crustal values smoothed according to the assigned interpolation function. Points that fall below the interpolation zone and have valid crustal values will have their velocities/densities set to the crustal values. Note that points with invalid/incompatible velocities in either the GTL or crustal dataset cannot be smoothed and the interpolation results will be marked as invalid.
  
  
Line 544: Line 749:
 
         - If running interactively, type Cntl-D to end input coord list.
 
         - If running interactively, type Cntl-D to end input coord list.
  
Version: 11.9.0
+
Version: 12.2.0
 
</pre>
 
</pre>
  
Line 558: Line 763:
 
* The Etree data structure is efficient and the library provides disk caching and buffering, resulting in very fast query performance.
 
* The Etree data structure is efficient and the library provides disk caching and buffering, resulting in very fast query performance.
  
How to use ucvm2etree:
+
Two versions of this utility are provided, a serial version and an MPI version.
 +
 
 +
 
 +
==== Serial ucvm2etree ====
 +
 
 +
The serial ucvm2etree is suitable for small to medium sized etrees, and is executed as follows:
 
<pre>
 
<pre>
 
Usage: ucvm2etree [-h] -f config
 
Usage: ucvm2etree [-h] -f config
Line 566: Line 776:
 
         -h: Help message
 
         -h: Help message
  
Version: 11.9.0
+
Version: 12.2.0
 
</pre>
 
</pre>
 +
  
 
The following is an example ucvm2etree configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/ucvm2etree_example.conf:
 
The following is an example ucvm2etree configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/ucvm2etree_example.conf:
  
 
<pre>
 
<pre>
# ucvm2etree conf file
+
# Domain corners coordinates (clockwise, degrees):
 +
proj=geo-bilinear
 +
lon_0=-119.288842
 +
lat_0=34.120549
 +
 
 +
lon_1=-118.354016
 +
lat_1=35.061096
  
# Projection and region coordinates (degrees):
+
lon_2=-116.846030
proj=+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0
+
lat_2=34.025873
lon_0=-129.75
+
 
lat_0=40.75
+
lon_3=-117.780976
rot=55.0
+
lat_3=33.096503
  
 
# Domain dimensions (meters):
 
# Domain dimensions (meters):
x-size=1800000.0
+
x-size=180000.0000
y-size=900000.0
+
y-size=135000.0000
z-size=112500.0
+
z-size=61875.0000
  
 
# Blocks partition parameters:
 
# Blocks partition parameters:
 
nx=32
 
nx=32
ny=16
+
ny=24
  
# Max freq (Hz), points per wavelength
+
# Max freq, points per wavelength, Vs min
 
max_freq=0.5
 
max_freq=0.5
 
ppwl=4.0
 
ppwl=4.0
 +
vs_min=200.0
  
# Etree max cellsize to allow in meters
+
# Max allowed size of octants in meters
max_cellsize=10000.0
+
max_octsize=10000.0
  
 
# Etree parameters and info
 
# Etree parameters and info
title=Cal0.5Hz200ms
+
title=ChinoHills_0.5Hz_200ms
 
author=P_Small
 
author=P_Small
 
date=05/2011
 
date=05/2011
outputfile=./ucvm_cvmh_cal_0.5hz_200ms.e
+
outputfile=./cmu_cvmh_chino_0.5hz_200ms.e
 
format=etree
 
format=etree
  
# CVMs to use
+
# UCVM parameters
cvm=cvmh
+
#ucvmstr=cvms
ucvmconf=../../conf/aftershock/ucvm.conf
+
ucvmstr=cvmh
 +
ucvm_interp_zrange=0.0,350.0
 +
ucvmconf=../../conf/kraken/ucvm.conf
  
# Min Vs, Vp
+
# Scratch
vs_min=200.0
+
scratch=/lustre/scratch/patricks/scratch
  
# Scratch
+
#
scratch=/tmp
+
# Buffering parameters used by MPI version only
 +
#
 +
# Etree buffer size in MB
 +
buf_etree_cache=128
 +
# Max octants to buffer for flat file during extraction
 +
buf_extract_mem_max_oct=4194304
 +
# Max octants to save in flat file before reporting full during extraction
 +
buf_extract_ffile_max_oct=16000000
 +
# Max octants to read from input flat file during sorting
 +
buf_sort_ffile_max_oct=20000000
 +
# Minimum number of octants between reports during merging
 +
buf_merge_report_min_oct=10000000
 +
# MPI send/recv octant buffer size during merging
 +
buf_merge_sendrecv_buf_oct=4096
 +
# Etree read/write octant buffer size during merging
 +
buf_merge_io_buf_oct=4194304
 
</pre>
 
</pre>
 +
 +
 +
==== Parallel ucvm2etree ====
 +
 +
The MPI version is useful for large and very large etrees. It involves three programs: ucvm2etree-extract-MPI, ucvm2etree-sort-MPI, and ucvm2etree-merge-MPI. All three take the same configuration file as the one used by the serial version. The ucvmetree-extract-MPI utility generates the set of points that are to be used in the etree, extracts those points from the specified CVMs, and saves them in a set of N files (where N is the process count). The ucvm2etree-sort-MPI utility sorts the points in each flat file by octant key. The third utility, ucvm2etree-merge-MPI, merges these locally sorted points into the final etree file.
  
 
Import the new CVM Etree into UCVM by adding these lines to ${UCVM_INSTALL_DIR}/conf/ucvm.conf:
 
Import the new CVM Etree into UCVM by adding these lines to ${UCVM_INSTALL_DIR}/conf/ucvm.conf:
Line 623: Line 863:
 
   newmodel          : name for your new model
 
   newmodel          : name for your new model
 
   config.outputfile : outputfile from config passed into ucvm2etree
 
   config.outputfile : outputfile from config passed into ucvm2etree
 +
</pre>
 +
 +
The following sections describe these three programs in more detail:
 +
 +
===== ucvm2etree-extract-MPI =====
 +
 +
Divides the etree region into C columns for extraction. This is an embarrassingly parallel operation. A dispatcher (rank 0) farms out each column to a worker in a pool of N cores for extraction. Each worker queries UCVM for the points in its column and writes a flat-file formatted etree. After program execution, there are N sub-etree files, each locally unsorted. The extractor must be run on 2^Y + 1 cores where Y>0 and (2^Y) < C. The output flat file format is a list of octants(24 byte addr, 16 byte key, 12 byte payload) in arbitrary Z-order.
 +
 +
Since the number of points in a column depends on the minimum Vs values within that column, some columns will have high octant counts and others will have very low octant counts. Having sub-etrees that vary greatly in size is not optimal for the sorting operations that follow, so ucvm2etree-extract-MPI implements a simple octant balancing mechanism. When a worker has extracted more than X octants (the default 16M octants), it reports to the dispatcher that it cannot extract any more columns and terminates. This strategy approximately balances the sub-etrees so that they may be loaded into memory by ucvm2etree-sort-MPI. In the case of very large extractions where the dispatcher reports that all workers have reported they are full yet columns remain to be extracted, increase the job size by a factor of 2 until there is room for all the columns.
 +
 +
 +
===== ucvm2etree-sort-MPI =====
 +
 +
Sorts the sub-etrees produced by ucvm2etree-extract-MPI so that each file is in local pre-order (Z-order). Again, the is an embarrassingly parallel operation. Each rank in the job reads in one of the sub-etrees produced by the previous program, sorts the octants in Z-order, and writes the sorted octants to a new sub-etree. The sorter must be run on 2^Y cores where Y>0. The worker pool must be large enough to allow each worker to load all the octants from its assigned file into memory. By default, this octant limit is 20M octants. If a rank reports that the size of the sub-etree exceeds memory capacity, the 20M buffer size constant may be increased if memory allows, or alternatively, cvmbycols-extract-MPI may be rerun with a larger job size to reduce the number of octants per file.
 +
 +
 +
===== ucvm2etree-merge-MPI =====
 +
 +
Merges N locally sorted etrees in flat file format into a final, compacted etree. This is essentially a merge sort on the keys from the addresses read from the local files. The cores at the lowest level of the merge tree each read in octants from two flat files in pre-order, merge sort the two sets of addresses, then pass the locally sorted list of addresses to a parent node for additional merging. This proceeds until the points rise to rank 1 which has a completely sorted list of etree addresses. Rank 0 takes this sorted list and performs a transactional append on the final Etree. The merger must be run on 2^N cores.
 +
 +
The program reads in input files that are in flat file format. In can output a merged Etree in either Etree format or flat file format. Although, due to space considerations, it strips the output flat file format to a pre-order list ot octants(16 byte key, 12 byte payload). The missing addr field is redundant and can be regenerated from the key field.
 +
 +
 +
===== MPI Version Buffer Values =====
 +
 +
There are a number of buffer values in the ucvm.conf file that must be accurately set for extraction, sorting, and merging to take place. Please see below for a list of those values and a description of the purpose of each one.
 +
 +
'''buf_etree_cache'''
 +
 +
Units: MB<br />
 +
Recommended Value: 128 usually works well, you can increase or decrease as need be<br />
 +
For Process: ucvm2etree-extract-MPI<br />
 +
Description: Used in the merging process during the etree_open call. According to the documentation of etree_open, this is more precisely defined as the "internal buffer space being allocated in megabytes". The value defined represents the memory buffer for etree traversing, finding, writing, etc.
 +
 +
'''buf_extract_mem_max_oct'''
 +
 +
Units: Number of octants<br />
 +
Recommended Value: 2097152<br />
 +
For Process: ucvm2etree-extract-MPI<br />
 +
Description: Used in the extraction process as the number of bytes to buffer before writing to disk. It reserves, in memory, a placeholder for this number of octants. The size of each octant is 52 bytes and as such, the actual memory allocation is 52 * this value.
 +
 +
'''buf_extract_ffile_max_oct'''
 +
 +
Units: Number of octants<br />
 +
Recommended Value: Varies. It must be greater than total output file size divided by the number of processes.<br />
 +
For Process: ucvm2etree-extract-MPI<br />
 +
Description: This value is ''very critical''! This defines the maximum output file size that each worker process can write. If this is not large enough, the worker will halt and return a "worker is full message". The best way to set this is to guess how large your extraction file might be (a relatively detailed Chino Hills e-tree, for example, is 250GB, and divide that number by 52 * the number or processes you have. So if I was building that mesh, on 32 processes, this value would need to be at least 162 million. However, this value also ties in with the number below, so '''please read it carefully as well'''.
 +
 +
'''buf_sort_ffile_max_oct'''
 +
 +
Units: Number of octants<br />
 +
Recommended Value: Must be at least equal to buf_extract_ffile_max_oct<br />
 +
For Process: ucvm2etree-sort-MPI<br />
 +
Description: Used in the sorting process to determine the max number of octants to read into memory. Therefore, for each process running on the server, there must be number of processes * 52 * buf_sort_ffile_max_oct bytes of memory available. So in our example above, if we were running those 32 processes on 16 dual core machines, we would need 162 million * 52 * 2 cores bytes of RAM or about 16 GB. All sorting is done in memory, not using the file on the disk.
 +
 +
'''buf_merge_report_min_oct'''
 +
 +
Units: Number of octants<br />
 +
Recommended Value: Personal preference - default is 10000000.<br />
 +
For Process: ucvm2etree-merge-MPI<br />
 +
Description:. This defines the number of octants that must be processed between progress reports in the merge process. This progress reports take the form of "Appended [at least buf_merge_report_min_oct] in N.NNs seconds". If you would like more frequent updates, set this value to be smaller.
 +
 +
'''buf_merge_sendrecv_buf_oct'''
 +
 +
Units: Bytes<br />
 +
Recommended Value: 4096<br />
 +
For Process: ucvm2etree-merge-MPI<br />
 +
Description: The number of bytes to merge per loop during the merge process. It is recommended to keep this value as is.
 +
 +
'''buf_merge_io_buf_oct'''
 +
 +
Units: Number of octants<br />
 +
Recommended Value: 4194304<br />
 +
For Process: ucvm2etree-merge-MPI<br />
 +
Description: Number of octants to read in and store in memory for merging. After being merged, the memory is then flushed to disk and the process repeats. Since this is a value in octants, it will require the machine to have at least buf_merge_io_buf_oct * 52 bytes of memory.
 +
<br />
 +
<br />
 +
 +
==== Etree Summary ====
 +
 +
In order to understand how ucvm2etree works, some knowledge of the Etree data structure is required. The fundamental unit of measure in an Etree is the tick. This is a distance expressed in meters, and is derived from the length of the longest side of the geographic region:
 +
<pre>
 +
tick size = (max(len x, len y, len z) / 2^31) m
 +
</pre>
 +
In essence, the geographic region is divided up into a set of 2^31 ticks.
 +
 +
The funamental unit of data in an Etree is the octant. An octant contains an address (location code) and a payload (material properties). Octants are organized into a tree structure called an octree, where each node has up to eight children. Each level in this octree represents an octant size (alternatively: edge size, or resolution). Moving from level N to level N+1 in the octree increases resolution by 2x. Moving from level N to level N-1 reduces resolution by 2x. The resolution for a level can be determined with:
 +
 +
<pre>
 +
res from level N = (max_len / 2^N) m
 +
</pre>
 +
 +
 +
The number of ticks in an octant of level N is:
 +
 +
<pre>
 +
ticks for level N = 2^N
 +
</pre>
 +
 +
 +
And thus the octant size in level N is:
 +
 +
<pre>
 +
octant size = (2^N * (max(len x, len y, len z) / 2^31)) m
 +
</pre>
 +
 +
 +
Given a particular resolution, the appropriate level to represent it in the etree can be determined with:
 +
 +
<pre>
 +
level N from res = (int)(log(max_len/res)/log(2)) + 1  (rounded up to next integer)
 +
</pre>
 +
 +
 +
These relations are important in understanding the algorithm described in the following sections.
 +
 +
 +
===== Algorithm Description =====
 +
 +
This program uses several algorithmic approaches to increase extraction speed and minimize the amount of disk space required to store the Etree. These are described in the following sections.
 +
 +
 +
====== Gridding of Geographic Coordinates ======
 +
 +
Gridding of the geographic bounding box is performed either using bilinear interpolation of lat/long corners, or by a UCVM projection specification (proj4 projection string, rot angle, lat0, lon0). Grid points are located at cell center.
 +
 +
 +
====== Optimized Etree Resolution Based on Frequency, PPWL, and Minimum Vs ======
 +
 +
One of the strengths of the Etree (octree) representation is that resolution can be adjusted so that a particular region is sampled at a lower/higher interval than surrounding areas. You are able to achieve high resolution in just those places where it is needed, and the rest of the region can be sampled at a much lower rez thereby saving considerable disk space.
 +
 +
With this flexibility comes the difficulty of determining what resolution is needed at all points within a meshing region to support Computational Seismology simulations. ucvm2etree uses an algorithm to deduce this resolution based off of three input parameters: maximum frequency to support, the number of points per wavelength that is desired, and the minimum Vs to support. The following relation is used to derive the resolution for a region:
 +
 +
<pre>
 +
res = vs_min / (ppwl * max_freq)
 +
 +
where:
 +
res: resolution for the region with vs_min, in meters
 +
vs_min: Minimum Vs found in region, in meters/sec
 +
ppwl: desired points per wavelength
 +
max_freq: maximum frequency to support, in Hz
 
</pre>
 
</pre>
  
 +
 +
Using the above relation, the 3D space is decomposed into small columns, and the resolution in each column is tailored to the vs values found at each depth. The algorithm can be summarized as follows:
 +
* Divide region in set of x,y columns, where the length of each column equals its width.
 +
* For each column:
 +
** Starting at depth = 0m, extract a 2D grid for the column at the resolution needed to support the max frequency, pounts/wavelength, and minimum Vs, with a depth that is one-half the current octant edge size (that is, Z axis is cell center as well).
 +
*** If the local minimum Vs is in this 2D grid allows for a lower resolution, and the current z value is divisible by the lower resolution, requery the CVM at the lower resolution. Otherwise, if the local min Vs in this grid needs a higher resolution, increase the resolution to what is required (up to the maximum resolution dictated by the configured global minimum Vs) and requery the CVM.
 +
*** Write this layer to a flat file (a single file when run serially, or a file local to this core when run in parallel)
 +
*** Step the depth down by the resolution of this 2D grid and loop to the next layer in the column.
 +
** Loop to the next column
 +
 +
 +
The ucvm2etree software enforces a minimum and maximum resolution for octants. Given a certain set of input parameters, these resolutions can be derived as follows:
 +
<pre>
 +
If vs_min = 200 m/s, max_freq = 4.0 Hz, and ppwl = 8, max_dim_length = 180000.0 m, max_cellsize = 500.0 m, then
 +
 +
Maximum Resolution:
 +
res = 200.0 / (8.0 * 4.0) = 6.25 m  (minimum resolution needed to support Vs=200 m/s)
 +
N = (log(180000.0/6.25)/log(2)) + 1 = 15
 +
max_res = 180000.0 / 2^15 ticks ~= 5.49 m
 +
 +
Therefore, maximum resolution (minimum edge size) for these parameters is 5.49 m.
 +
 +
Minimum Resolution:
 +
res = 500.0 m  (the maximum cell size to support)
 +
N = (log(180000.0/500.0)/log(2)) + 1 = 9
 +
min_res = 180000.0 / 2^9 ticks ~= 351.56 m
 +
 +
Therefore, minimum resolution (maximum edge size) for these parameters is 351.56 m.
 +
</pre>
 +
* Minimum resolution represents a floor on the possible resolution of an etree. This is protection against very large octants from being formed by high Vs values.
 +
 +
 +
====== High-performance Etree merging and compaction ======
 +
 +
Generation of an Etree in a parallel environment is challenging since the Euclid etree interface only allows serial write access to the etree file. Points may only be inserted, or they may be appended in key order, by a single writer. This makes etee construction and compaction time consuming. To circumvent this issue, a high performance etree merge/compaction algorithm has been developed.
 +
 +
Simply put, it is essentially a merge sort performed on 2^N small flat files that contain all of the points needed for the final etree. The flat files contain material properties previously extracted from UCVM. Once these small, local flat files exist on disk, an MPI job of 2^N cores is then able to sort these octants by their keys in O(n*log n) time. Once all the octants are sorted, a single core may then open the final etree/flat file for writing in append mode, and simply stream the sorted octants to disk efficiently.
 +
 +
Conceptually, the cores in this 2^N MPI job are organized into a tree-like structure with each node containing two children. The bottom layer of this tree has nodes that each do a pre-order traversal on two flat files. They sort the combined octants read from each etree in pre-order, and then pass them up to a parent node. The nodes from the top layer to the second to bottom layer each receive locally sorted lists of octants from their two children and once again sort the combined list of octants in pre-order. This newly sorted list is passed upward in this manner until the root node of the tree receives two locally sorted halves of the entire set of etree points. It then sorts them, and appends the fully sorted stream of octants to the final file.
 +
 +
 +
=== ucvm2mesh ===
 +
 +
This utility creates a 3D regular mesh extracted from a set of CVMs. The CVM Etree may then be used in a wave propagation simulation such as AWP-ODC. It can produce meshes in a number of formats
 +
 +
* AWP 3 value: float Vp, float Vs, float Rho
 +
* AWP 5 value: float Vp, float Vs, float Rho, float Qp, float Qs
 +
* AWP 8 vlaue: int i, int j, int k, float Vp, float Vs, float Rho, float Qp, float Qs
 +
* SORD: One file each for Vp, Vs, Rho (double precision)
 +
* netCDF (with optional netCDF package installed, and the mesh2netcdf utility)
 +
 +
 +
Two versions of this utility are provided, a serial version and an MPI version.
 +
 +
 +
==== Serial ucvm2mesh ====
 +
 +
The serial ucvm2etree is suitable for small to medium sized meshes, and is executed as follows:
 +
<pre>
 +
Usage: ucvm2mesh [-h] -f config
 +
 +
Flags:
 +
        -f: Configuration file
 +
        -h: Help message
 +
 +
Version: 12.2.0
 +
</pre>
 +
 +
 +
The following is an example ucvm2mesh configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/ucvm2mesh_example.conf:
 +
 +
<pre>
 +
# List of CVMs to query
 +
ucvmlist=cvmh
 +
 +
# UCVM conf file
 +
ucvmconf=../conf/kraken/ucvm.conf
 +
 +
# Gridding cell centered or vertex
 +
gridtype=CENTER
 +
 +
# Spacing of cells
 +
spacing=2000.0
 +
 +
# Projection
 +
proj=+proj=utm +datum=WGS84 +zone=11
 +
rot=-40.0
 +
x0=-122.3
 +
y0=34.7835
 +
z0=0.0
 +
 +
# Number of cells along each dim
 +
nx=384
 +
ny=248
 +
nz=25
 +
 +
# Partitioning of grid among processors
 +
px=2
 +
py=2
 +
pz=5
 +
 +
# Vs/Vp minimum
 +
vp_min=0
 +
vs_min=0
 +
 +
# Mesh and grid files, format
 +
meshfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.media
 +
gridfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.grid
 +
meshtype=IJK-12
 +
 +
# Location of scratch dir
 +
scratch=/lustre/scratch/user/scratch
 +
</pre>
 +
 +
 +
==== Parallel ucvm2mesh ====
 +
 +
The MPI version, ucvm2mesh-mpi, is useful for large and very large meshes. It accepts the same configuration file as the serial version. It provides an additional command-line option for stage-out of the mesh files from the locations specified in the conf file to a final directory location. This is useful on HPC systems that use temporary scratch space.
 +
<pre>
 +
Usage: ucvm2mesh-mpi [-h] [-o dir] -f config
 +
 +
Flags:
 +
        -f: Configuration file
 +
        -o: Stage-out directory
 +
        -h: Help message
 +
 +
Version: 12.2.0
 +
</pre>
 +
 +
==== Performance Enhancements for ucvm2mesh on Lustre Filesystems ====
 +
 +
When running ucvm2mesh on Lustre filesystems (and other systems with similar structures), you may experience some performance issues that result in less than optimal performance. By default, Lustre uses one OST or disk per file. As such, running ucvm2mesh-mpi with a significant number of cores results in all those processes attempting to seek and write to the same file on the same OST/disk. This can result in a disk bottleneck that gives very poor performance, where it may take up to 10 minutes to write a few hundred MBs.
 +
 +
This can be fixed by increasing the number of OSTs/disks to which the output directory and files are written. This is called striping. Generally, you will want to set the stripes for the output directory to be either the recommended amount for large files or the maximum amount allowed. The command to set the number of stripes on Kraken is lfs setstripe [dir] --count [count]. Count can be a fixed number or it can be -1 for the maximum supported by the operating system. The Kraken documentation suggests that 80 is a reasonable amount for very large files. So, for example, suppose we were generating a large mesh with our stage out directory being /lustre/scratch/username/meshes/outstripe. We would first make that directory and then run:
 +
 +
lfs setstripe /lustre/scratch/username/meshes/outstripe --count 80
 +
 +
Then we would have our configuration file output to that directory:
 +
 +
meshfile=/lustre/scratch/username/meshes/outstripe/USC_cvms_awp.media
 +
gridfile=/lustre/scratch/username/meshes/outstripe/USC_cvms_awp.media.grid
 +
 +
When we qsub the job, it will automatically take advantage of striping and distribute the load across multiple targets.
 +
 +
It is also important to remember that more cores is not always better. Additional cores can result in increased write requests that may cause performance decreases. It is important to find the optimal balance between the number of processes writing to the file and the number of stripes. For exampe, for a large 600km x 400km x 40km meshes, a core count of 600 and stripe count of 80 on NICS Kraken will result in reasonably good performance.
  
 
=== grd_query ===
 
=== grd_query ===
Line 649: Line 1,175:
 
         - If running interactively, type Cntl-D to end input coord list.
 
         - If running interactively, type Cntl-D to end input coord list.
  
Version: 11.9.0
+
Version: 12.2.0
 
</pre>
 
</pre>
  
Line 664: Line 1,190:
 
         -h: Help message
 
         -h: Help message
  
Version: 11.9.0
+
Version: 12.2.0
 +
</pre>
 +
 
 +
 
 +
The following is an example grd2etree configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/grd2etree_example.conf:
 +
 
 +
<pre>
 +
# grd2etree UCM Map config file
 +
 
 +
# Domain corners coordinates (degrees):
 +
proj=+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0
 +
lon_0=-129.75
 +
lat_0=40.75
 +
rot=55.0
 +
 
 +
# Domain dimensions (meters):
 +
x-size=1800000.0
 +
y-size=900000.0
 +
 
 +
# Spacing
 +
spacing=250.0
 +
 
 +
# Etree parameters and info
 +
title=UCVM_Elev_Vs30_Map_Wills_Wald
 +
author=P_Small
 +
date=05/2011
 +
outputfile=../../model/ucvm/ucvm.e
 +
 
 +
# Grid data directories
 +
elev_hr_dir=/home/scec-00/USER/opt/ned
 +
elev_lr_dir=/home/scec-00/USER/opt/bath
 +
vs30_hr_dir=/home/scec-00/USER/opt/vs30/wills_gridfloat
 +
vs30_lr_dir=/home/scec-00/USER/opt/vs30/wald_gridfloat
 
</pre>
 
</pre>
  
Line 682: Line 1,240:
 
=== patchmodel ===
 
=== patchmodel ===
  
The utility provides a post-processing "patching" mechanism to smooth out discontinuities at the interfaces between different regional models. The user identifies a 3D geographic region in the form of an open box where smoothing is to be performed. The patchmodel program is run to extract the material properties along the surfaces of this box at a fixed resolution (eg: 250 m) and it saves them into a new model which may then be imported into and queried by UCVM. Points that fall within the smoothing region are interpolated using an inverse distance weighting function with the 10% closest surface points on the same z-plane. A power factor of 2.0 is used for weighting. Linear interpolation is used along the z-axis to smooth out the edge values.
+
The utility provides a post-processing "patching" mechanism to smooth out discontinuities at the interfaces between different regional models. The user identifies a 3D geographic region in the form of an open box where smoothing is to be performed. The patchmodel program is run to extract the material properties along the vertical surfaces of this box at a fixed resolution (eg: 250 m) from 0m depth to "z-size" depth and it saves them into a new model which may then be imported into and queried by UCVM. Points that fall within the smoothing region are interpolated using an inverse distance weighting function with the 10% closest surface points on the same z-plane. A power factor of 2.0 is used for weighting. Linear interpolation is used along the z-axis to smooth out the edge values.
  
 
<pre>
 
<pre>
Line 691: Line 1,249:
 
         -h: Help message
 
         -h: Help message
  
Version: 11.9.0
+
Version: 12.2.0
 +
</pre>
 +
 
 +
 
 +
The following is an example patchmodel configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/patch_example.conf:
 +
 
 +
<pre>
 +
# Example Patchmodel config file
 +
 
 +
# Version
 +
version=cvmh_p1_2011/05/24
 +
 
 +
# UCVM config
 +
ucvmstr=cencal,cvmh,lin-thurber,1d
 +
ucvm_interp_zrange=0.0,350.0
 +
ucvmconf=../../conf/aftershock/ucvm.conf
 +
 
 +
# Projection
 +
proj=+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0
 +
lon_0=-121.3
 +
lat_0=35.6
 +
rot=90.0
 +
 
 +
# Domain dimensions (meters):
 +
x-size=550000.0
 +
y-size=50000.0
 +
z-size=112500.0
 +
 
 +
# Spacing (meters)
 +
spacing=250.0
 +
 
 +
# Output paths
 +
modelname=cvmh_p1
 +
modelpath=/home/scec-00/USER/opt/aftershock/ucvm/model/patch
 
</pre>
 
</pre>
  
Line 699: Line 1,290:
 
<pre>
 
<pre>
 
  newmodel_interface=model_patch
 
  newmodel_interface=model_patch
  newmodel_modelpath=config.modelpath/config.modelname
+
  newmodel_modelpath=config.modelpath/config.modelname.conf
  
 
  where:
 
  where:
Line 705: Line 1,296:
 
   config.modelpath : modelpath from config passed into patchmodel
 
   config.modelpath : modelpath from config passed into patchmodel
 
   config.modelname : modelname from config passed into patchmodel
 
   config.modelname : modelname from config passed into patchmodel
 +
 +
Note: Be sure to append ".conf" to newmodel_modelpath
 
</pre>
 
</pre>
  
Line 770: Line 1,363:
 
/* Query underlying models */
 
/* Query underlying models */
 
int ucvm_query(int n, ucvm_point_t *pnt, ucvm_data_t *data);
 
int ucvm_query(int n, ucvm_point_t *pnt, ucvm_data_t *data);
 +
 +
/* Get info on installed models */
 +
int ucvm_install_info(char *info, int len);
  
 
#endif
 
#endif
Line 777: Line 1,373:
 
* Initialization (<code>ucvm_init()</code>)
 
* Initialization (<code>ucvm_init()</code>)
 
* Add models in order of preference using <code>ucvm_add_model()</code>. These may be either crustal or GTL models. See below for specific details.
 
* Add models in order of preference using <code>ucvm_add_model()</code>. These may be either crustal or GTL models. See below for specific details.
 +
* Set any models-specific flags using <code>ucvm_setparam</code>. These will override any flags specified in ucvm.conf. For example, to enable the 1D background model with CVM-H through the API, execute the <code>ucvm_setparam()</code> function as follows:
 +
<pre>
 +
ucvm_setparam(UCVM_PARAM_MODEL_CONF, UCVM_MODEL_CVMH, "USE_1D_BKG", "True");
 +
</pre>
 
* If GTLs have been added:
 
* If GTLs have been added:
 
** Associate an interpolation function with each using <code>ucvm_assoc_ifunc()</code>. Otherwise, linear interpolation will be used.
 
** Associate an interpolation function with each using <code>ucvm_assoc_ifunc()</code>. Otherwise, linear interpolation will be used.
Line 803: Line 1,403:
 
</pre>
 
</pre>
 
* The programmer may also define their own models and integrate them into UCVM. A user model interface is defined by creating the init(), finalize(), getversion(), getlabel(), setparam(), and query() functions for the model interface, and populating the <code>ucvm_model_t</code> structure with these function pointers and some additional record keeping information. Configuration information for a specific model instance of that interface is placed in a <code>ucvm_modelconf_t</code> structure. Once a model is fully described with a <code>ucvm_model_t</code> interface and a <code>ucvm_modelconf_t</code> configuration, it can be registered for use with the <code>ucvm_add_user_model()</code> function and queried with the <code>ucvm_query()</code> function.  
 
* The programmer may also define their own models and integrate them into UCVM. A user model interface is defined by creating the init(), finalize(), getversion(), getlabel(), setparam(), and query() functions for the model interface, and populating the <code>ucvm_model_t</code> structure with these function pointers and some additional record keeping information. Configuration information for a specific model instance of that interface is placed in a <code>ucvm_modelconf_t</code> structure. Once a model is fully described with a <code>ucvm_model_t</code> interface and a <code>ucvm_modelconf_t</code> configuration, it can be registered for use with the <code>ucvm_add_user_model()</code> function and queried with the <code>ucvm_query()</code> function.  
 +
  
 
=== Data Type Descriptions ===
 
=== Data Type Descriptions ===
Line 860: Line 1,461:
 
               ucvm_data_t *data);
 
               ucvm_data_t *data);
 
} ucvm_model_t;
 
} ucvm_model_t;
</pre>
 
 
 
=== Setting Model Flags ===
 
 
Underlying models may be configured in one of two ways: with a key/value string in the UCVM configuration file, or by passing configuration parameters with the <code>ucvm_setparam()</code>function. The following model flags are supported:
 
 
<pre>
 
key: "USE_1D_BKG"    valid values: "True"/"False"    (CVM-H only)
 
</pre>
 
 
 
For example, to enable the 1D background model with CVM-H execute the <code>ucvm_setparam()</code> function as follows:
 
 
<pre>
 
ucvm_setparam(UCVM_PARAM_MODEL_CONF, UCVM_MODEL_CVMH, "USE_1D_BKG", "True");
 
 
</pre>
 
</pre>
  
Line 942: Line 1,527:
 
-118.0 34.0 0.0
 
-118.0 34.0 0.0
 
-118.0 34.0 100.0
 
-118.0 34.0 100.0
-118.0 34.0 1000.0
+
-118.0 34.0 1000.0 <Cntrl-D>
 
  -118.0000    34.0000      0.000    295.679    390.000      cvmh    824.177    195.000  1084.062      none      0.000      0.000      0.000      crust    824.177    195.000  1084.062
 
  -118.0000    34.0000      0.000    295.679    390.000      cvmh    824.177    195.000  1084.062      none      0.000      0.000      0.000      crust    824.177    195.000  1084.062
 
  -118.0000    34.0000    100.000    295.679    390.000      cvmh  2371.500    727.748  2051.320      none      0.000      0.000      0.000      crust  2371.500    727.748  2051.320
 
  -118.0000    34.0000    100.000    295.679    390.000      cvmh  2371.500    727.748  2051.320      none      0.000      0.000      0.000      crust  2371.500    727.748  2051.320
 
  -118.0000    34.0000  1000.000    295.679    390.000      cvmh  2616.063  1082.813  2127.783      none      0.000      0.000      0.000      crust  2616.063  1082.813  2127.783
 
  -118.0000    34.0000  1000.000    295.679    390.000      cvmh  2616.063  1082.813  2127.783      none      0.000      0.000      0.000      crust  2616.063  1082.813  2127.783
 +
</pre>
 +
 +
 +
=== Additional Examples ===
 +
 +
The following example directs ucvm_query to query the Ely GTL and CVM-S crustal model by depth, using the Ely interpolation function to smooth the values. The Z-axis interpolation zone is set to 0-400m.
 +
 +
<pre>
 +
% ./ucvm_query -m elygtl:ely,cvms -c gd -z 0.0,400.0 -f ../conf/ucvm.conf
 +
</pre>
 +
 +
 +
The following example queries the USGS Bay Area (CenCalVM) and CVM-H models by elevation, tiling them into a composite model (where CenCalVM has priority).
 +
 +
<pre>
 +
% ./ucvm_query -m cencal,cvmh -c ge -f ../conf/ucvm.conf
 +
</pre>
 +
 +
 +
The following example queries the 1D model by depth, with surface elevation and Vs30 values taken from the alternate Yong-Wald Vs30 map.
 +
 +
<pre>
 +
% ./ucvm_query -m 1d -p yong -c gd -f ../conf/ucvm.conf
 
</pre>
 
</pre>
  
Line 964: Line 1,572:
 
  LDFLAGS = $(LDFLAGS) -L$(UCVM_INSTALL_DIR)/lib ucvm -L($CVMS_INSTALL_DIR)/lib cvms -L$(ETREE_INSTALL_DIR)/libsrc etree -L$(PROJ4_INSTALL_DIR)/lib proj  
 
  LDFLAGS = $(LDFLAGS) -L$(UCVM_INSTALL_DIR)/lib ucvm -L($CVMS_INSTALL_DIR)/lib cvms -L$(ETREE_INSTALL_DIR)/libsrc etree -L$(PROJ4_INSTALL_DIR)/lib proj  
 
           -lgfortran -pthreads -lm
 
           -lgfortran -pthreads -lm
 +
 +
 +
=== Example Programs ===
 +
 +
Please see the ./examples directory of the software distribution for compiled examples that employ the C API.
  
  
 
= History of UCVM Releases =
 
= History of UCVM Releases =
 +
 +
* 12.2.0 (Feb 9, 2012): Initial release
  
  
Line 978: Line 1,593:
  
 
Please reference at least Small et al (2011) if you use this software framework; other references should be considered, depending on the purpose.
 
Please reference at least Small et al (2011) if you use this software framework; other references should be considered, depending on the purpose.
 +
 +
 +
= Technical Notes =
 +
 +
== SCEC CVM Etree Convention ==
 +
 +
=== Etree Model Format ===
 +
* Variable resolution based off of Vs, max supported frequency, and points per wavelength. The resolution at any particular region will be dependent upon the local Vs value (relation shown below). However, a minimum Vs at Etree creation will clamp the resolution to a maximum value regardless of how low the local Vs is found to be.
 +
<pre>
 +
rez = vs / (ppwl * freq)
 +
 +
where:
 +
vs: Vs at point of interest (m/s)
 +
ppwl: points per wavelength
 +
freq: max supported frequency (Hz)
 +
</pre>
 +
* Model data is referenced relative to free surface. Free surface is defined at air/ground, water/ground interface. UCVM topography and bathymetry can be used to determine if a point is in the air/water and default properties returned for those domains.
 +
* Stored properties are Vp, Vs, and density.
 +
 +
 +
=== Etree Metadata Description ===
 +
 +
The Etree metadata tag describes the payload of the octants, the extents of the coverage region, and the map projection. It will have the following format, with fields delimited by vertical bars:
 +
 +
<pre>
 +
metatype_version|title|author|date|vs_min|max_freq|ppwl|format|proj.4_string|geo_origin|rot_angle|xyz_dims|xyz_ticks
 +
 +
where:
 +
  metatype_version: "SCEC_CVM_V1" string of max length 64 chars
 +
  title: free-format title string of max length 64 chars
 +
  author: free-format author string of max length 64 chars
 +
  date: YY/MM/DD date string
 +
  vs_min: vs minimum used to determine max resolution
 +
  max_freq: maximum supported frequency in Hz
 +
  ppwl: Points per wavelength
 +
  format: data format specification of octants. Uses string format: "name,data_type,size,units". eg: "vp,float,4,m/s;vs,float,4,m/s;rho,float,4,kg/m^3"
 +
  proj.4_string: Proj.4 projection string, eg: "+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0"
 +
  geo_origin: Origin point of CVM region in geo coordinates, eg: "-129.25,41.0"
 +
  rot_angle: Rotation angle of bounding box in degrees, eg "55.0"
 +
  xyz_dims: Dimensions of bounding box in km, eg "1800,800,100"
 +
  xyz_ticks: Number of ticks along each dimension
 +
</pre>
 +
 +
 +
== SCEC Patch Model Format ==
 +
 +
A patch model consists of five files: a configuration text file, and four binary files with the surface data. All of these files are automatically generated by the patchmodel utility. The sections below describe the file formats.
 +
 +
 +
=== Configuration file ===
 +
 +
The patch model configuration file specifies the origin, dimensions, and map projection of the 3D region covered by the patch. An example follows:
 +
 +
<pre>
 +
# cvmh patch conf file
 +
 +
# Version
 +
version=cvmh_p1_2011/05/24
 +
# Projection
 +
proj=+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0
 +
lon_0=-121.300000
 +
lat_0=35.600000
 +
rot=90.000000
 +
# Spacing
 +
spacing=250.000000
 +
# Dimensions
 +
x-size=550000.000000
 +
y-size=50000.000000
 +
z-size=112500.000000
 +
# Paths to surface files
 +
surf_0_0_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_0_0.bin
 +
surf_1_0_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_1_0.bin
 +
surf_0_1_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_0_1.bin
 +
surf_1_1_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_1_1.bin
 +
</pre>
 +
 +
 +
=== Surface Files ===
 +
 +
The surface file format is a 2D array of three floating point values: vp, vs, and density. The assignment of x versus y axis in each file is dependent on the orientation of the surface with respect to the box's origin point.
 +
 +
 +
== SCEC Map Etree Convention ==
 +
 +
=== Etree Model Format ===
 +
 +
* 3D Etree with a single octant depth along Z-axis (effectively a 2D grid)
 +
* Fixed resolution based off of maximum desired grid spacing in grd2etree configuration.
 +
* Sampled with grid registration to allow smoothing of adjacent octants in UCVM.
 +
* Stored values at each grid point are: surface elevation (float), Vs30 (float).
 +
 +
 +
=== Etree Metadata Description ===
 +
 +
The Etree metadata tag describes the payload of the octants, the extents of the coverage region, and the map projection. It will have the following format, with fields delimited by vertical bars:
 +
 +
<pre>
 +
metatype_version|title|author|date|spacing|format|proj.4_string|geo_origin|rot_angle|xy_dims|xyz_ticks
 +
 +
where:
 +
  metatype_version: "SCEC_MAP_V1" string of max length 64 chars
 +
  title: free-format title string of max length 64 chars
 +
  author: free-format author string of max length 64 chars
 +
  date: YY/MM/DD date string
 +
  spacing: grid spacing in meters
 +
  format: data format specification of octants. Uses string format: "name,data_type,size,units". eg: "surf,float,4,m/s;vs30,float,4,m/s;"
 +
  proj.4_string: Proj.4 projection string, eg: "+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0"
 +
  geo_origin: Origin point of CVM region in geo coordinates, eg: "-129.25,41.0"
 +
  rot_angle: Rotation angle of bounding box in degrees, eg "55.0"
 +
  xy_dims: Dimensions of bounding box in km, eg "1800,800"
 +
  xyz_ticks: Number of ticks along each dimension
 +
</pre>
  
  
Line 989: Line 1,716:
 
#'''Wald, D. J., and T. I. Allen (2007)''', Topographic slope as a proxy for seismic site conditions and amplification, Bull. Seism. Soc. Am., 97 (5), 1379-1395, doi:10.1785/0120060267.  
 
#'''Wald, D. J., and T. I. Allen (2007)''', Topographic slope as a proxy for seismic site conditions and amplification, Bull. Seism. Soc. Am., 97 (5), 1379-1395, doi:10.1785/0120060267.  
 
#'''Wills, C. J., and K. B. Clahan (2006)''', Developing a map of geologically defined site-condition categories for California, Bull. Seism. Soc. Am., 96 (4A), 1483-1501, doi:10.1785/0120050179.
 
#'''Wills, C. J., and K. B. Clahan (2006)''', Developing a map of geologically defined site-condition categories for California, Bull. Seism. Soc. Am., 96 (4A), 1483-1501, doi:10.1785/0120050179.
#'''Yong, A., Hough, S.E., Iwahashi, J., and A. Braverman (2012)''', A terrain-based site conditions map of California with implications for the contiguous United States, Bull. Seism. Soc. Am., accepted.
+
#'''Yong, A., Hough, S.E., Iwahashi, J., and A. Braverman (2012)''', A terrain-based site conditions map of California with implications for the contiguous United States, Bull. Seism. Soc. Am., Vol. 102, No. 1, pp. 114–128, February 2012, doi: 10.1785/0120100262.
 
 
 
 
= Miscellaneous Technical Notes =
 
  
  

Latest revision as of 23:52, 9 June 2014

Contents

Overview

The Unified Community Velocity Model Framework UCVM is a software package which provides a standard interface to several regional community velocity models. Any differences in coordinate projections within each model are hidden so that the researcher queries any of the models by longitude, latitude, and Z (where Z is depth from surface or elevation relative MSL). UCVM also provides several other unique capabilities and products, including:

  • Seamlessly combine two or more models into a composite model for querying
  • Optionally include a California statewide geotechnical layer and interpolate it with the crustal velocity models
  • Extract a 3D mesh or CVM Etree (octree) of material properties from any combination of models
  • Standard California statewide elevation and Vs30 data map is provided
  • Miscellaneous tools are provided for creating 2D etree maps, and optimizing etrees
  • Numerically smooth discontinuities at the interfaces between different regional models
  • Add support for future velocity models with the extendable interface


Current list of supported velocity models: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, SCEC CVM-NCI, Magistrale WFCVM, USGS CenCalVM, Graves Cape Mendocino, Lin-Thurber Statewide, Tape SoCal.

The API itself is written in the C language. The software is MPI-capable, and has been shown to scale to 3300+ cores on NICS Kraken in a mesh generator. It allows application programs to tailor which set of velocity models are linked in to minimize memory footprint. It is also compatible with grid resources that have data stage-in policies for jobs, and light-weight kernels that support static libraries only. This package has been tested on NCCS Jaguar, NICS Kraken, and USC HPCC.


Requirements

The system requirements are as follows:

  1. UNIX operating system (Linux)
  2. GNU gcc/gfortran compilers (MPI wrappers such as mpicc are OK)
  3. tar for opening the compressed files
  4. Euclid Etree library (recommend euclid3-1.3.tar.gz)
  5. Proj.4 Projection Library (recommend proj-4.7.0.tar.gz)


Optional dependencies include any of the following standard velocity models and packages:

  1. Standard community velocity models: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, SCEC CVM-NCI, Magistrale WFCVM, USGS CenCalVM, Graves Cape Mendocino, Lin-Thurber Statewide, Tape SoCal
  2. NetCDF (network Common Data Form) library NetCDF

Installation

Download

  1. Start at SCEC website: http://scec.usc.edu/scecpedia/UCVM
  2. Navigate to the Downloads section of the UCVM web page.
  3. Click the download link to download the latest source code distribution file and any listed dependencies. This is typically posted as a tar and gzipped file (tgz format). The source code distribution file is large (350 Mb), so the download make take awhile.


CheckSum Test

Verify that the provided md5 checksum file matches the computed md5 for your downloaded tarball:

$ md5sum -c ucvm-12.2.0.tar.gz.md5


Basic Install

The general steps for installing UCVM are:

  1. Install Etree and Proj.4 packages
  2. Install one or more velocity models
  3. Define environment variables describing where Etree/Proj.4/models are located
  4. Run the UCVM installer


First, install the Etree and Proj.4 packages which are required by UCVM. The INSTALL/README instructions in each distribution describe how to install them, but generally:

For the Euclid Etree library:

$ tar zxvf euclid3-1.3.tar.gz
$ cd ./euclid3-1.3/libsrc
$ make all

If installing the Etree library on a Lustre filesystem on a Cray system (NICS Kraken, NCCS Jaguar), you must enable the Cray IOBUF module prior to compiling the library. Also, edit the ./libsrc/Makefile to add the compile flags "-DUSE_IOBUF -DUSE_IOBUF_LOCAL_MACROS".

For the Proj.4 library (version 4.7.0):

$ tar xvf proj-4.7.0.tar
$ cd proj-4.7.0
$ ./configure --prefix=<your desired install directory>
$ make; make install

Be sure to note where you installed these two packages. You will need this information later in the installation process.

Then, install the velocity models that you wish to use with UCVM according to the documentation provided by those models. Ensure they are compiled with gcc/gfortran.

For the SCEC CVM-H model:

$ tar xvf cvmh-11.9.1.tar.gz
$ cd cvmh-11.9.1
$ ./configure --prefix=<your desired_install directory>
$ make; make install

For the SCEC CVM-S model:

$ tar xvf cvms-11.11.0.tar.gz
$ cd cvms/src
$ make all

Again, be sure to note where you installed the community velocity models.

Declare environment variables with the paths to the Etree library, Proj.4 library, and velocity models that you previously installed. Also, update your LD_LIBRARY_PATH environment variable to include the new library directories.

ETREE_DIR=<Etree install path>
PROJ4_DIR=<Proj.4 install path>
CVMH_DIR=<CVM-H install path>
CVMS_DIR=<CVM-S install path>
LD_LIBRARY_PATH=${ETREE_DIR}/libsrc:${PROJ4_DIR}/lib:${CVMH_DIR}/lib:${CVMS_DIR}/lib:${LD_LIBRARY_PATH}
UCVM_INSTALL_DIR=<desired UCVM install path>

In a bash shell, for example, the ETREE_DIR environment variable can be declared with the command:

$ ETREE_DIR=/opt/etree;export ETREE_DIR

In a csh shell:

$ setenv ETREE_DIR /opt/etree

Now unpack the UCVM software distribution:

$ tar zxvf ucvm-12.2.0.tar.gz

The build configuration differs slightly depending on where you wish to run UCVM. On a standard Linux system with gcc, run the following commands to configure the software with CVM-H and CVM-S enabled:

$ cd ucvm-12.2.0
$ ./configure --prefix=${UCVM_INSTALL_DIR} --with-etree-include-path="${ETREE_DIR}/libsrc" 
 --with-etree-lib-path="${ETREE_DIR}/libsrc" --with-proj4-include-path="${PROJ4_DIR}/include" 
 --with-proj4-lib-path="${PROJ4_DIR}/lib" --enable-model-cvmh --enable-model-cvms 
 --with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib" 
 --with-gctpc-lib-path="${CVMH_DIR}/lib" --with-cvms-include-path="${CVMS_DIR}/include" 
 --with-cvms-lib-path="${CVMS_DIR}/lib" --with-cvmh-model-path="${CVMH_DIR}/model" --with-cvms-model-path="${CVMS_DIR}/src"

The above example enables CVM-H and CVM-S but any of the following models may be enabled: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, USGS CenCalVM, and WFCVM. For each velocity model you want to use, the configure script needs to be told to explicitly use that model (with the --enable-model-* flag) and where to find the velocity model's library files (.a or .so), header files (.h), and model files so that they can be linked into UCVM.

Note that unless you force static linking with the "--enable-static" command line option, you must update your LD_LIBRARY_PATH environment variable with the paths to the Etree, Proj.4, and model shared libraries.

The configure command is quite long, so an example shell script with this command is located in ./scripts/autoconf/basic_install.sh. It assumes that you previously defined the environment variables described in this installation guide. The script can be executed with the following shell commands:

$ cd scripts/autoconf
$ ./basic_install.sh

Compile the software by executing the following command in the main directory:

$ make

Check that the package was built correctly by executing the following command in the main directory:

$ make check

Verify that all unit tests and acceptance tests pass. Depending on your system and your UCVM configuration, testing may take up to 15-30 minutes.

Install the software to the desired target location by executing the following command in the main directory:

$ make install

A full list of supported standard velocity models and options can be listed with the command:

$ ./configure --help

Updating the Configuration

The UCVM installer will automatically create a configuration file in ${UCVM_INSTALL_DIR}/conf/ucvm.conf with entries for all enabled velocity models and some standard settings. However, you may wish to edit this file to toggle model flags or to add new user models in the future. For example, the CVM-H model has a number of flags that may be set in the UCVM configuration file. Please see UCVM_User_Guide#Framework_Configuration for more details.


Advanced Install

Additional options must be passed to the ./configure script on a more exotic Linux system, such as a host that requires static linking (NICS Kraken), or has a Lustre high-performace filesystem (NCCS Jaguar, NICS Kraken). The following example shows how to enable static linking and Cray IOBUF buffering:

$ module add iobuf
$ ./configure --prefix=${UCVM_INSTALL_DIR} --enable-static --enable-iobuf --enable-model-cvmh --enable-model-cvms
 --with-etree-include-path="${ETREE_DIR}/libsrc" --with-etree-lib-path="${ETREE_DIR}/libsrc" 
 --with-proj4-include-path="${PROJ4_DIR}/include" --with-proj4-lib-path="${PROJ4_DIR}/lib" 
 --with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib" 
 --with-gctpc-lib-path="${CVMH_DIR}/lib" --with-cvms-include-path="${CVMS_DIR}/include" 
 --with-cvms-lib-path="${CVMS_DIR}/lib" --with-cvmh-model-path="${CVMH_DIR}/model" --with-cvms-model-path="${CVMS_DIR}/src" CC=cc
$ make; make check; make install

It is generally advisable to compile all components of UCVM using the same compilation system. NICS Kraken, for example, uses the Cray compiler wrappers (cc not gcc). As such, on this system, you must use CC to compile the Euclid library.

A few notes:

  • Some systems require applications to be linked statically (example above).
  • Lustre filesystems may require linking with Cray IOBUF module (NICS Kraken example above).
  • The configure script needs to be told where to find the library file(s), the header file(s), and the model files for each velocity model that you wish to enable. The configure script will check that the libraries and headers are present, and it will setup UCVM with that model enabled and properly configured.
  • The script will attempt to auto-detect a GNU compliant compiler. If an MPI wrapper is found, the MPI version of UCVM is compiled, otherwise the serial version is built. The compiler may be overridden with the option 'CC="mpicc"' for example.
  • Additional compiler flags and linker flags may be passed to ./configure with the CFLAGS and LDFLAGS options. Generally, this is not needed but allows the package to be built on platforms with specific compiling/linking requirements.


The full list of options follows:

  --enable-static         enable static linking
  --enable-netcdf         enable netCDF module
  --enable-iobuf          enable IOBUF module
  --enable-model-cvmh     enable model SCEC CVM-H
  --enable-model-cvms     enable model SCEC CVM-S
  --enable-model-cencal   enable model USGS CenCalVM
  --enable-model-cvmsi    enable model SCEC CVM-SI
  --enable-model-cvmnci   enable model SCEC CVM-NCI
  --enable-model-wfcvm    enable model WFCVM
  --enable-model-cvmlt    enable model Lin-Thurber
  --enable-model-cmrg     enable model CMRG
  --enable-model-tape     enable model Tape

  --with-etree-include-path
                          location of the Etree headers
  --with-etree-lib-path   location of the Etree libraries
  --with-proj4-include-path
                          location of the Proj.4 headers
  --with-proj4-lib-path   location of the Proj.4 libraries
  --with-netcdf-include-path
                          location of the netCDF headers
  --with-netcdf-lib-path  location of the netCDF libraries
  --with-cencal-include-path
                          location of the USGS CenCalVM headers
  --with-cencal-lib-path  location of the USGS CenCalVM libraries
  --with-cencal-model-path
                          location of the USGS CenCalVM high-rez etree
  --with-cencal-extmodel-path
                          location of the USGS CenCalVM extended etree
  --with-cvmh-include-path
                          location of the SCEC CVM-H headers
  --with-cvmh-lib-path    location of the SCEC CVM-H libraries
  --with-cvmh-model-path  location of the SCEC CVM-H model files
  --with-gctpc-lib-path   location of the CVM-H GCTPC libraries (CVM-H 11.2.0)
  --with-cvms-include-path
                          location of the SCEC CVM-S headers
  --with-cvms-lib-path    location of the SCEC CVM-S libraries
  --with-cvms-model-path  location of the SCEC CVM-S model files
  --with-cvmsi-include-path
                          location of the SCEC CVM-SI headers
  --with-cvmsi-lib-path   location of the SCEC CVM-SI libraries
  --with-cvmsi-model-path location of the SCEC CVM-SI model files
  --with-cvmnci-include-path
                          location of the SCEC CVM-NCI headers
  --with-cvmnci-lib-path  location of the SCEC CVM-NCI libraries
  --with-cvmnci-model-path
                          location of the SCEC CVM-NCI model files
  --with-wfcvm-include-path
                          location of the WFCVM headers
  --with-wfcvm-lib-path   location of the WFCVM libraries
  --with-wfcvm-model-path location of the WFCVM model files
  --with-cvmlt-include-path
                          location of the CVMLT headers
  --with-cvmlt-lib-path   location of the CVMLT libraries
  --with-cvmlt-model-path location of the CVMLT model files
  --with-cmrg-include-path
                          location of the CMRG headers
  --with-cmrg-lib-path    location of the CMRG libraries
  --with-cmrg-model-path  location of the CMRG model files
  --with-tape-include-path
                          location of the TAPE headers
  --with-tape-lib-path    location of the TAPE libraries
  --with-tape-model-path  location of the TAPE model files

Unit and Acceptance Tests

Both the unit tests and acceptance tests may be executed with the following command from the main directory:

$ make check

All tests should result in a PASS.


Trouble-shooting

Shared Library Paths

If you see an error similar to the following while running either the tests or the UCVM programs:

error while loading shared libraries: libsomelib.so: cannot open shared object file: No such file or directory

This indicates that UCVM was linked against one or more shared libraries and the dynamic library loader cannot find the actual .so library at run-time. The solution is to update your LD_LIBRARY_PATH to include the directory containing the library mentioned in the error message. For example, the following command adds a new search directory to LD_LIBRARY_PATH in a csh shell:

$ setenv LD_LIBRARY_PATH /home/USER/opt/somepkg/lib:${LD_LIBRARY_PATH}

Alternatively, if you do not want to work with shared libraries, UCVM can be compiled statically with the "--enable-static" configuration option.

Proj. 4 Error: major axis or radius = 0 or not given

On systems with home filesystems that are not viewable to compute nodes (such as NICS Kraken), you may encounter errors with Proj.4 when trying to run any component of UCVM on compute nodes. This is due to the fact that Proj.4 actually relies on a file called proj_defs.dat which is located in the ${MAKE_INSTALL_LOCATION}/share/proj directory. So for example, suppose you had configured Proj.4 with ./configure --prefix=/not_viewable_to_compute_nodes/proj-4.7.0, Proj. 4 would then search for proj_defs.dat in /not_viewable_to_compute_nodes/proj-4.7.0/share/proj/proj_defs.dat. This will cause UCVM to throw the error "Proj.4 Error: major axis or radius = 0 or not given" and your job will fail.

The only way it seems to solve this issue is to actually make sure your --prefix directory is actually visible to the compute nodes and do a make install there. Documentation suggests that you can set the PROJ_LIB environment variable, however this seems to not work correctly without modifications to the Proj.4 source code.

Building Against CVM-H 11.2.0

UCVM is designed to work with CVM-H 11.9.1 by default, but it can optionally be built with CVM-H 11.2.0. Use this modified configure command to configure the package with CVM-H 11.2.0 and CVM-S:

$ cd ucvm-12.2.0
$ ./configure --prefix=${UCVM_INSTALL_DIR} --with-etree-include-path="${ETREE_DIR}/libsrc" 
 --with-etree-lib-path="${ETREE_DIR}/libsrc" --with-proj4-include-path="${PROJ4_DIR}/include" 
 --with-proj4-lib-path="${PROJ4_DIR}/lib" --enable-model-cvmh --enable-model-cvms 
 --with-cvmh-include-path="${CVMH_DIR}/include" --with-cvmh-lib-path="${CVMH_DIR}/lib" 
 --with-gctpc-lib-path="${CVMH_DIR}/gctpc/lib" --with-cvms-include-path="${CVMS_DIR}/include" 
 --with-cvms-lib-path="${CVMS_DIR}/lib" --with-cvmh-model-path="${CVMH_DIR}/bin" --with-cvms-model-path="${CVMS_DIR}/src"
 CFLAGS="-D_UCVM_MODEL_CVMH_11_2_0"

Note that the CVM-H model and gctpc paths have changed, and there is now a CFLAGS argument. After configuring, you may compile, check, and install normally.

Framework Configuration

The main package configuration file is ${UCVM_INSTALL_DIR}/conf/ucvm.conf. This is where the paths to all configured models and maps are specified, as well as any model flags are defined. The UCVM installer sets up this file automatically. Yet there are a number of situations where you will want to modify it, such as to add a new model.

The following is an example configuration:

# UCVM config file

# UCVM model path
#
# Change to reflect UCVM installation directory.
#
ucvm_interface=map_etree
ucvm_mappath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/ucvm/ucvm.e


# Pre-defined models
#
# SCEC CVM-S
cvms_modelpath=/home/scec-00/USER/opt/aftershock/cvms/src
#
# SCEC CVM-H
cvmh_modelpath=/home/scec-00/USER/opt/aftershock/cvmh-11.9.1/model
#
# USGS Bay Area high-rez and extended etrees
cencal_modelpath=/home/scec-00/USER/opt/etree/USGSBayAreaVM-08.3.0.etree
cencal_extmodelpath=/home/scec-00/USER/opt/etree/USGSBayAreaVMExt-08.3.0.etree
#
# SCEC CVM-SI
cvmsi_modelpath=/home/scec-00/USER/opt/aftershock/cvmsi/model/i8
#
# SCEC CVM-NCI
cvmnci_modelpath=/home/scec-00/USER/opt/aftershock/cvmnci/model/i2
#
# Wasatch Front CVM
wfcvm_modelpath=/home/scec-00/USER/opt/aftershock/wfcvm/src
#
# Lin-Thurber Statewide
cvmlt_modelpath=/home/scec-00/USER/opt/aftershock/cvmlt/model
#
# Cape Mendocino RG
cmrg_modelpath=/home/scec-00/USER/opt/aftershock/cvm-cmrg/model/cmrg.conf
#
# Tape SoCal Model
tape_modelpath=/home/scec-00/USER/opt/aftershock/cvm-tape/model/m16
#
# 1D
1d_modelpath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/1d/1d.conf
#
# 1D GTL
1dgtl_modelpath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/1d/1d.conf


# User-defined models
#
# Change to reflect paths to user model files
#
# Current supported interfaces: model_etree, model_patch
#
#user1_interface=model_etree
#user1_modelpath=/home/username/model_etree/user1.etree
#user2_interface=model_patch
#user2_modelpath=/home/username/model_patch/user2.conf


# User-defined maps
#
# Change to reflect paths to user map files
#
# Current supported interfaces: map_etree
#
# Optional Yong-Wald vs30 map
yong_interface=map_etree
yong_mappath=/home/scec-00/USER/opt/aftershock/ucvm-11.11.0_RC/model/ucvm/ucvm_yong_wald.e


# Model flags
#
cvmh_param=USE_1D_BKG,True
cvmh_param=USE_GTL,True


The following table describes each configuration item, where "*" indicates the model/map label. Model/map labels can be predefined strings such as "cmvh" for CVM-H, or any arbitrary string label for the case of a user-defined model.

Key Value(s) Description
ucvm_interface map_etree The interface code needed to parse the UCVM map
ucvm_mappath ${UCVM_INSTALL_DIR}/model/ucvm/ucvm.e String identifying location of the UCVM map Etree
*_modelpath A directory, URL, or file String identifying location of model's primary or high-rez data
*_extmodelpath A directory, URL, or file String identifying location of model's extended data (required for USGS CenCalVM extended Etree).
*_interface Models: model_etree, model_patch
Maps: model_map
The interface code needed to parse this user-defined model/map.
*_mappath A directory, URL, or file String identifying location of user-defined map Etree
*_param Flag,Value Set a model-specific configuration Flag to Value. The defined flags are USE_1D_BKG and USE_GTL for CVM-H, for toggling the 1D background model and GTL.


Supported Standard Crustal Velocity Models

UCVM has built-in support for a number of standard community velocity models (CVMs). These standard models are referenced in ucvm_query and the API by their string labels. The following table lists the labels for these predefined CVMs. These are reserved labels and cannot be used for a user-defined model.

Note that "predefined" does not necessarily mean "installed". These are models that UCVM can query if it is linked with that model's libraries at install time. Some are pre-linked, such as lin-thurber and cmrg, the rest must be explicitly linked at install.

Label Model Type Active Limit
cvmh SCEC CVM-H External, installed separately and linked at install 1
cvms SCEC CVM-S External, installed separately and linked at install 1
cencal USGS Bay Area CenCalVM External, installed separately and linked at install 1
cvmnci SCEC CVM-NCI External, installed separately and linked at install 1
cvmsi SCEC CVM-SI External, installed separately and linked at install 1
wfcvm Wasatch Front CVM (Utah) External, installed separately and linked at install 1
1d Hadley-Kanamori 1D Internal, included in UCVM and automatically linked 1
lt Lin-Thurber California Statewide External, installed separately and linked at install 1
cmrg Cape Mendocino Rob Graves External, installed separately and linked at install 1
tape Carl Tape SoCal External, installed separately and linked at install 1
cmuetree CMU CVM Etree Internal, included in UCVM, user specifies etree path in ./conf/ucvm.conf 1


UCVM also defines two "model interfaces". A model interface is a specific file format convention for representing a velocity model. Any velocity model written to conform to one of these model interfaces may be easily imported and queried from UCVM. In addition, many such models may be active simultaneously, whereas only one version of a predefined model may be active at any given time. The two model interfaces are: SCEC CVM Etrees, and SCEC Patch Models. SCEC CVM Etrees employ a special schema and metadata format. SCEC Patch Models have a special conf file and binary file format. The section UCVM_User_Guide#Adding_a_User-defined_Velocity_Model discusses how to create and import these velocity models.

The currently supported model interfaces are summarized in the following table:

Interface Description Active Limit
model_etree SCEC CVM Etree 100
model_patch SCEC Patch Model 100


Supported Standard GTL Velocity Models

UCVM has built-in support for two geotechnical layer models. These near-surface models are intended to provide superior velocity information at shallow depths. The following table lists the labels for these predefined GTLs. These are reserved labels and cannot be used for a user-defined GTL.

Label GTL
elygtl Ely Vs30-derived GTL (Ely et al., 2010)
1dgtl Generic 1D, identical to 1d crustal model


Interpolation functions are used to smooth GTL material properties with the underlying crustal model material properties. This smoothing is performed over an interpolation zone along the Z axis. Interpolation functions can be assigned on a per-GTL basis. Two predefined interpolation functions are provided to the user:

Label Interpolation Function
linear Linear interpolation
ely Ely interpolation relation (Ely et al., 2010)


If the user enables a GTL model but does not specify an interpolation function, linear interpolation is used by default. If no interpolation zone is specified, a depth range of 0 m - 350m is used.


Supported Standard Maps

UCVM has built-in support for a number of standard maps for California. These standard maps contain elevation data (DEM) and Vs30 data for the region and are referenced in ucvm_query and the API by their string labels. The following table lists the labels for these predefined maps. These are reserved labels and cannot be used for a user-defined map.

Label Map
ucvm USGS NED 1 arcsec DEM, and Wills-Wald Vs30 (default)
yong USGS NED 1 arcsec DEM, and Yong-Wald Vs30


There is also a special map interface that supports reading SCEC Map Etrees. These are maps that conform to a special Etree schema and metadata format convention. The user may create their own maps in this format and import them into UCVM. Any number of maps may be defined, but only one may be active at any time. Maps are defined by the user in the UCVM configuration file (see UCVM_User_Guide#Adding_a_User-defined_Map).

The map interfaces are summarized in the following table:

Interface Description
map_etree SCEC Map Etree


Supported Model Flags

Underlying models may be configured in one of two ways: with a key/value string in the UCVM configuration file, or by passing configuration parameters with the ucvm_setparam()function. The following model flags are supported:

Parameter Value Models
USE_1D_BKG True/False (default False) CVM-H
USE_GTL True/False (default True) CVM-H


Adding New Velocity Models

Enabling Another Standard Velocity Model After Installation

If after installation you wish to link in another standard velocity model, you must re-install UCVM using the process described in UCVM_User_Guide#Installation. Add the configure options appropriate for the new model you wish to link into UCVM.


Upgrading the Version of a Standard Velocity Model After Installation

Generally, UCVM must be recompiled and re-installed using the new model libraries and headers. However, if you are certain that only the model files have changed (and the query interface has NOT changed), you can simply update that model's modelpath configuration in ${UCVM_INSTALL_DIR}/conf/ucvm.conf to reference the new files.


Using Multiple Versions of a Supported Velocity Model Simultaneously

Not supported.


Adding a User-defined Velocity Model

UCVM may be extended to support any user-defined velocity model. The simplest way to add a new model is to store that model as a SCEC Etree or Patch Model, and update ${UCVM_INSTALL_DIR}/conf/ucvm.conf with the model interface and path. For example:

For SCEC Etrees:

user1_interface=model_etree
user1_modelpath=/home/username/model_etree/user1.etree


For Patch models:

user2_interface=model_patch
user2_modelpath=/home/username/model_patch/user2.conf


SCEC Etrees can be created with the ./bin/ucvm2etree utility. Patch models can be created with the ./bin/patchmodel utility.


For other models that either are represented by tables, or have their own native API, UCVM may be extended to query from these models. However, this requires code modifications to the core UCVM library located in ./src/ucvm and recompilation. The general steps are:

  • Define new model label UCVM_MODEL_MODELNAME in ucvm_dtypes.h
  • Create ucvm_model_modelname.h/.c containing the glue code that links the UCVM query interface to the native model interface
  • Modify ./src/ucvm/ucvm.c:
    • Add ucvm_model_modelname.h include at top
    • Add model lookup to ucvm_add_model() function
    • Add model to ucvm_get_resources() function
  • Modify ./src/ucvm/Makefile.am:
    • Add ucvm_model_modelname.o to list of libucvm.a dependencies
  • Modify ./configure.ac:
    • Define "enable-modelname", "with-modelname-lib-path", and "with-modelname-incl-path" configure options
    • Add library and header checks for your new model
  • Modify installer to customize path to your new model in ucvm.conf:
    • Add your new model to ./conf/Makefile.am
    • Add config entry for your new model to ./conf/ucvm_template.conf
  • Regenerate the Makefiles with ./scripts/autoconf/reconf.sh
  • Reconfigure and recompile UCVM, making sure to pass your new model flags to the configure utility


Adding a User-defined Map

Update ${UCVM_INSTALL_DIR}/conf/ucvm.conf with the map interface and Etree path:

usermap1_interface=map_etree
usermap2_mappath=/home/username/maps/usermap1.e

User-defined maps can be created with the ./bin/grd2etree utility.

Framework Description

Concept of Operation

Combining Velocity Models into a Composite Model

UCVM combines multiple regional velocity models, along with a DEM map and a Vs30 map, into one composite model for the purposes of querying surface elevation, Vs30, Vp, Vs, and density. Individual model setup and query details are abstracted under a uniform interface, and the application sees only the composite model. This simplifies application code and makes it easier for those programs to support new velocity models in the future.

UCVM model tiling method.

Velocity models are combined into the composite by tiling them on top of one another. When UCVM is initialized, the user selects an ordered list of models to query for data. Query points are then submitted to each velocity model in that list, one model at a time in list order. The first model to return valid velocity data for the point is considered to have fullfilled that data request and subsequent models are not queried. Generally, no smoothing is performed at the interfaces between models (an exception is interpolation between a GTL and crustal model as described below). For each query point, the following data is returned: surface elevation, Vs30, Vp, Vs, and density. The figure at right illustrates how this tiling is performed.

Regional models may include: CVM-H, CVM-H, USGS Bay Area, 1D, or other user-defined models. Most models and maps have a maximum extent, outside of which no data is available. The exception is the generic 1D model as it has infinite extents. Applications must check the returned velocity and density values to ensure they are valid.


Querying with one or more GTLs

The API distinguishes between two types of regional velocity models: GTL models, and crustal models. GTL models are shallow near-surface velocity models called geotechnical layers. Crustal models are deeper velocity models such as tomographic models. When a user selects to query from one or more crustal models only, the API executes the previously described behavior. However, if one or more GTL models are selected, additional processing is performed.

When a GTL model is given in the ordered list of models, the API creates a default interpolation zone along the z-axis over which interpolation between this GTL and the underlying crustal models is performed. It also assigns a default (linear) interpolation function to smooth the velocities of the GTL and underlying crustal models. Both the interpolation zone and interpolation function can be changed by the user.

Querying for a point in this operating mode is now slightly different from the previous crustal-only case. As before, all query points are submitted to the crustal models in the ordered list of models (skipping any GTLs). However, the z coordinate of the points may be shifted down to the lower edge of the interpolation zone if the point falls within it. The crustal models are queried with this possibly modified 3D coordinate and the velocities/density values are saved. The same list of query points is then submitted to the GTL models in the ordered list of models (skipping any crustals) and the z coordinate of those points within the interpolation zone may be shifted up to the top edge. The GTL models are queried with this possibly modified 3D coordinate and the velocity/density values are saved.

The list of query points is then traversed one final time. Those points that fall above the interpolation zone will have their velocities/densities set to those of the GTL as processed by the interpolation function assigned to that GTL. Those points that fall within the interpolation zone and have valid GTL/crustal values will have their GTL/crustal values smoothed according to the assigned interpolation function. Points that fall below the interpolation zone and have valid crustal values will have their velocities/densities set to the crustal values. Note that points with invalid/incompatible velocities in either the GTL or crustal dataset cannot be smoothed and the interpolation results will be marked as invalid.


Querying Models Via Native Interface versus Querying UCVM

For the most part, querying any model through its native interface will yield the same material properties as UCVM. There are a few exceptions, which are noted here:

  • When querying by elevation with CVM-H and any GTL, points that fall above the UCVM free surface and within the CVM-H domain are normalized to be relative to the CVM-H free surface. This is done to eliminate small discrepancies between the UCVM DEM and the CVM-H DEM.
  • The same correction is performed when querying by elevation with CenCalVM and any GTL.
  • When querying CenCalVM by depth, UCVM does not directly pass that point to the cencalvm interface since the definition of depth differs between UCVM and cencalvm. Instead, UCVM derives the elevation of the free surface (ground/air, ground/water) at the point of interest by drilling down through the model at that point and finding the highest octant with valid material properties. UCVM then converts the query point depth to elevation and queries cencalvm by elevation.


Utilities

ucvm_query

This is the command-line tool for querying CVMs. Any set of crustal and GTL velocity models may be selected and queried in order of preference. Points may be queried by (lon,lat,dep) or (lon,lat,elev) and the coordinate conversions for a particular model are handled transparently.

Usage: ucvm_query [-m models<:ifunc>] [-p user_map] [-c coordtype] [-f config] [-z zmin,zmax] < file.in

Flags:
        -h This help message.
        -m Comma delimited list of crustal/GTL models to query in order
           of preference. GTL models may optionally be suffixed with ':ifunc'
           to specify interpolation function.
        -p User-defined map to use for elevation and vs30 data.
        -c Z coordinate mode: geo-depth (gd, default), geo-elev (ge).
        -f Configuration file. Default is ./ucvm.conf.
        -z Optional depth range for gtl/crust interpolation.

Input format is:
        lon lat Z

Output format is:
        lon lat Z surf vs30 crustal cr_vp cr_vs cr_rho gtl gtl_vp gtl_vs gtl_rho cmb_algo cmb_vp cmb_vs cmb_rho

Notes:
        - If running interactively, type Cntl-D to end input coord list.

Version: 12.2.0

The config file is generally ${UCVM_INSTALL_DIR}/conf/ucvm.conf, but you may pass in any custom configuration file.


ucvm2etree

This utility creates a SCEC-formatted CVM Etree extracted from a set of CVMs. The CVM Etree may then be imported into UCVM and queried like any other velocity model. This confers a number of advantages:

  • Several different velocity models can be combined into one composite model and distributed as a single file.
  • Any number of CVM Etrees can be easily imported into UCVM without recompilation (just a config change) since Etree support is built in.
  • The Etree data structure is efficient and the library provides disk caching and buffering, resulting in very fast query performance.

Two versions of this utility are provided, a serial version and an MPI version.


Serial ucvm2etree

The serial ucvm2etree is suitable for small to medium sized etrees, and is executed as follows:

Usage: ucvm2etree [-h] -f config

Flags:
        -f: Configuration file
        -h: Help message

Version: 12.2.0


The following is an example ucvm2etree configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/ucvm2etree_example.conf:

# Domain corners coordinates (clockwise, degrees):
proj=geo-bilinear
lon_0=-119.288842
lat_0=34.120549

lon_1=-118.354016
lat_1=35.061096

lon_2=-116.846030
lat_2=34.025873

lon_3=-117.780976
lat_3=33.096503

# Domain dimensions (meters):
x-size=180000.0000
y-size=135000.0000
z-size=61875.0000

# Blocks partition parameters:
nx=32
ny=24

# Max freq, points per wavelength, Vs min
max_freq=0.5
ppwl=4.0
vs_min=200.0

# Max allowed size of octants in meters
max_octsize=10000.0

# Etree parameters and info
title=ChinoHills_0.5Hz_200ms
author=P_Small
date=05/2011
outputfile=./cmu_cvmh_chino_0.5hz_200ms.e
format=etree

# UCVM parameters
#ucvmstr=cvms
ucvmstr=cvmh
ucvm_interp_zrange=0.0,350.0
ucvmconf=../../conf/kraken/ucvm.conf

# Scratch
scratch=/lustre/scratch/patricks/scratch

#
# Buffering parameters used by MPI version only
#
# Etree buffer size in MB
buf_etree_cache=128
# Max octants to buffer for flat file during extraction
buf_extract_mem_max_oct=4194304
# Max octants to save in flat file before reporting full during extraction
buf_extract_ffile_max_oct=16000000
# Max octants to read from input flat file during sorting
buf_sort_ffile_max_oct=20000000
# Minimum number of octants between reports during merging
buf_merge_report_min_oct=10000000
# MPI send/recv octant buffer size during merging
buf_merge_sendrecv_buf_oct=4096
# Etree read/write octant buffer size during merging
buf_merge_io_buf_oct=4194304


Parallel ucvm2etree

The MPI version is useful for large and very large etrees. It involves three programs: ucvm2etree-extract-MPI, ucvm2etree-sort-MPI, and ucvm2etree-merge-MPI. All three take the same configuration file as the one used by the serial version. The ucvmetree-extract-MPI utility generates the set of points that are to be used in the etree, extracts those points from the specified CVMs, and saves them in a set of N files (where N is the process count). The ucvm2etree-sort-MPI utility sorts the points in each flat file by octant key. The third utility, ucvm2etree-merge-MPI, merges these locally sorted points into the final etree file.

Import the new CVM Etree into UCVM by adding these lines to ${UCVM_INSTALL_DIR}/conf/ucvm.conf:

 newmodel_interface=model_etree
 newmodel_modelpath=config.outputfile

 where:
   newmodel          : name for your new model
   config.outputfile : outputfile from config passed into ucvm2etree

The following sections describe these three programs in more detail:

ucvm2etree-extract-MPI

Divides the etree region into C columns for extraction. This is an embarrassingly parallel operation. A dispatcher (rank 0) farms out each column to a worker in a pool of N cores for extraction. Each worker queries UCVM for the points in its column and writes a flat-file formatted etree. After program execution, there are N sub-etree files, each locally unsorted. The extractor must be run on 2^Y + 1 cores where Y>0 and (2^Y) < C. The output flat file format is a list of octants(24 byte addr, 16 byte key, 12 byte payload) in arbitrary Z-order.

Since the number of points in a column depends on the minimum Vs values within that column, some columns will have high octant counts and others will have very low octant counts. Having sub-etrees that vary greatly in size is not optimal for the sorting operations that follow, so ucvm2etree-extract-MPI implements a simple octant balancing mechanism. When a worker has extracted more than X octants (the default 16M octants), it reports to the dispatcher that it cannot extract any more columns and terminates. This strategy approximately balances the sub-etrees so that they may be loaded into memory by ucvm2etree-sort-MPI. In the case of very large extractions where the dispatcher reports that all workers have reported they are full yet columns remain to be extracted, increase the job size by a factor of 2 until there is room for all the columns.


ucvm2etree-sort-MPI

Sorts the sub-etrees produced by ucvm2etree-extract-MPI so that each file is in local pre-order (Z-order). Again, the is an embarrassingly parallel operation. Each rank in the job reads in one of the sub-etrees produced by the previous program, sorts the octants in Z-order, and writes the sorted octants to a new sub-etree. The sorter must be run on 2^Y cores where Y>0. The worker pool must be large enough to allow each worker to load all the octants from its assigned file into memory. By default, this octant limit is 20M octants. If a rank reports that the size of the sub-etree exceeds memory capacity, the 20M buffer size constant may be increased if memory allows, or alternatively, cvmbycols-extract-MPI may be rerun with a larger job size to reduce the number of octants per file.


ucvm2etree-merge-MPI

Merges N locally sorted etrees in flat file format into a final, compacted etree. This is essentially a merge sort on the keys from the addresses read from the local files. The cores at the lowest level of the merge tree each read in octants from two flat files in pre-order, merge sort the two sets of addresses, then pass the locally sorted list of addresses to a parent node for additional merging. This proceeds until the points rise to rank 1 which has a completely sorted list of etree addresses. Rank 0 takes this sorted list and performs a transactional append on the final Etree. The merger must be run on 2^N cores.

The program reads in input files that are in flat file format. In can output a merged Etree in either Etree format or flat file format. Although, due to space considerations, it strips the output flat file format to a pre-order list ot octants(16 byte key, 12 byte payload). The missing addr field is redundant and can be regenerated from the key field.


MPI Version Buffer Values

There are a number of buffer values in the ucvm.conf file that must be accurately set for extraction, sorting, and merging to take place. Please see below for a list of those values and a description of the purpose of each one.

buf_etree_cache

Units: MB
Recommended Value: 128 usually works well, you can increase or decrease as need be
For Process: ucvm2etree-extract-MPI
Description: Used in the merging process during the etree_open call. According to the documentation of etree_open, this is more precisely defined as the "internal buffer space being allocated in megabytes". The value defined represents the memory buffer for etree traversing, finding, writing, etc.

buf_extract_mem_max_oct

Units: Number of octants
Recommended Value: 2097152
For Process: ucvm2etree-extract-MPI
Description: Used in the extraction process as the number of bytes to buffer before writing to disk. It reserves, in memory, a placeholder for this number of octants. The size of each octant is 52 bytes and as such, the actual memory allocation is 52 * this value.

buf_extract_ffile_max_oct

Units: Number of octants
Recommended Value: Varies. It must be greater than total output file size divided by the number of processes.
For Process: ucvm2etree-extract-MPI
Description: This value is very critical! This defines the maximum output file size that each worker process can write. If this is not large enough, the worker will halt and return a "worker is full message". The best way to set this is to guess how large your extraction file might be (a relatively detailed Chino Hills e-tree, for example, is 250GB, and divide that number by 52 * the number or processes you have. So if I was building that mesh, on 32 processes, this value would need to be at least 162 million. However, this value also ties in with the number below, so please read it carefully as well.

buf_sort_ffile_max_oct

Units: Number of octants
Recommended Value: Must be at least equal to buf_extract_ffile_max_oct
For Process: ucvm2etree-sort-MPI
Description: Used in the sorting process to determine the max number of octants to read into memory. Therefore, for each process running on the server, there must be number of processes * 52 * buf_sort_ffile_max_oct bytes of memory available. So in our example above, if we were running those 32 processes on 16 dual core machines, we would need 162 million * 52 * 2 cores bytes of RAM or about 16 GB. All sorting is done in memory, not using the file on the disk.

buf_merge_report_min_oct

Units: Number of octants
Recommended Value: Personal preference - default is 10000000.
For Process: ucvm2etree-merge-MPI
Description:. This defines the number of octants that must be processed between progress reports in the merge process. This progress reports take the form of "Appended [at least buf_merge_report_min_oct] in N.NNs seconds". If you would like more frequent updates, set this value to be smaller.

buf_merge_sendrecv_buf_oct

Units: Bytes
Recommended Value: 4096
For Process: ucvm2etree-merge-MPI
Description: The number of bytes to merge per loop during the merge process. It is recommended to keep this value as is.

buf_merge_io_buf_oct

Units: Number of octants
Recommended Value: 4194304
For Process: ucvm2etree-merge-MPI
Description: Number of octants to read in and store in memory for merging. After being merged, the memory is then flushed to disk and the process repeats. Since this is a value in octants, it will require the machine to have at least buf_merge_io_buf_oct * 52 bytes of memory.

Etree Summary

In order to understand how ucvm2etree works, some knowledge of the Etree data structure is required. The fundamental unit of measure in an Etree is the tick. This is a distance expressed in meters, and is derived from the length of the longest side of the geographic region:

tick size = (max(len x, len y, len z) / 2^31) m

In essence, the geographic region is divided up into a set of 2^31 ticks.

The funamental unit of data in an Etree is the octant. An octant contains an address (location code) and a payload (material properties). Octants are organized into a tree structure called an octree, where each node has up to eight children. Each level in this octree represents an octant size (alternatively: edge size, or resolution). Moving from level N to level N+1 in the octree increases resolution by 2x. Moving from level N to level N-1 reduces resolution by 2x. The resolution for a level can be determined with:

res from level N = (max_len / 2^N) m


The number of ticks in an octant of level N is:

ticks for level N = 2^N


And thus the octant size in level N is:

octant size = (2^N * (max(len x, len y, len z) / 2^31)) m


Given a particular resolution, the appropriate level to represent it in the etree can be determined with:

level N from res = (int)(log(max_len/res)/log(2)) + 1   (rounded up to next integer)


These relations are important in understanding the algorithm described in the following sections.


Algorithm Description

This program uses several algorithmic approaches to increase extraction speed and minimize the amount of disk space required to store the Etree. These are described in the following sections.


Gridding of Geographic Coordinates

Gridding of the geographic bounding box is performed either using bilinear interpolation of lat/long corners, or by a UCVM projection specification (proj4 projection string, rot angle, lat0, lon0). Grid points are located at cell center.


Optimized Etree Resolution Based on Frequency, PPWL, and Minimum Vs

One of the strengths of the Etree (octree) representation is that resolution can be adjusted so that a particular region is sampled at a lower/higher interval than surrounding areas. You are able to achieve high resolution in just those places where it is needed, and the rest of the region can be sampled at a much lower rez thereby saving considerable disk space.

With this flexibility comes the difficulty of determining what resolution is needed at all points within a meshing region to support Computational Seismology simulations. ucvm2etree uses an algorithm to deduce this resolution based off of three input parameters: maximum frequency to support, the number of points per wavelength that is desired, and the minimum Vs to support. The following relation is used to derive the resolution for a region:

res = vs_min / (ppwl * max_freq)

where:
 res: resolution for the region with vs_min, in meters
 vs_min: Minimum Vs found in region, in meters/sec
 ppwl: desired points per wavelength
 max_freq: maximum frequency to support, in Hz


Using the above relation, the 3D space is decomposed into small columns, and the resolution in each column is tailored to the vs values found at each depth. The algorithm can be summarized as follows:

  • Divide region in set of x,y columns, where the length of each column equals its width.
  • For each column:
    • Starting at depth = 0m, extract a 2D grid for the column at the resolution needed to support the max frequency, pounts/wavelength, and minimum Vs, with a depth that is one-half the current octant edge size (that is, Z axis is cell center as well).
      • If the local minimum Vs is in this 2D grid allows for a lower resolution, and the current z value is divisible by the lower resolution, requery the CVM at the lower resolution. Otherwise, if the local min Vs in this grid needs a higher resolution, increase the resolution to what is required (up to the maximum resolution dictated by the configured global minimum Vs) and requery the CVM.
      • Write this layer to a flat file (a single file when run serially, or a file local to this core when run in parallel)
      • Step the depth down by the resolution of this 2D grid and loop to the next layer in the column.
    • Loop to the next column


The ucvm2etree software enforces a minimum and maximum resolution for octants. Given a certain set of input parameters, these resolutions can be derived as follows:

If vs_min = 200 m/s, max_freq = 4.0 Hz, and ppwl = 8, max_dim_length = 180000.0 m, max_cellsize = 500.0 m, then

Maximum Resolution:
res = 200.0 / (8.0 * 4.0) = 6.25 m  (minimum resolution needed to support Vs=200 m/s)
N = (log(180000.0/6.25)/log(2)) + 1 = 15
max_res = 180000.0 / 2^15 ticks ~= 5.49 m

Therefore, maximum resolution (minimum edge size) for these parameters is 5.49 m.

Minimum Resolution:
res = 500.0 m  (the maximum cell size to support)
N = (log(180000.0/500.0)/log(2)) + 1 = 9
min_res = 180000.0 / 2^9 ticks ~= 351.56 m

Therefore, minimum resolution (maximum edge size) for these parameters is 351.56 m.
  • Minimum resolution represents a floor on the possible resolution of an etree. This is protection against very large octants from being formed by high Vs values.


High-performance Etree merging and compaction

Generation of an Etree in a parallel environment is challenging since the Euclid etree interface only allows serial write access to the etree file. Points may only be inserted, or they may be appended in key order, by a single writer. This makes etee construction and compaction time consuming. To circumvent this issue, a high performance etree merge/compaction algorithm has been developed.

Simply put, it is essentially a merge sort performed on 2^N small flat files that contain all of the points needed for the final etree. The flat files contain material properties previously extracted from UCVM. Once these small, local flat files exist on disk, an MPI job of 2^N cores is then able to sort these octants by their keys in O(n*log n) time. Once all the octants are sorted, a single core may then open the final etree/flat file for writing in append mode, and simply stream the sorted octants to disk efficiently.

Conceptually, the cores in this 2^N MPI job are organized into a tree-like structure with each node containing two children. The bottom layer of this tree has nodes that each do a pre-order traversal on two flat files. They sort the combined octants read from each etree in pre-order, and then pass them up to a parent node. The nodes from the top layer to the second to bottom layer each receive locally sorted lists of octants from their two children and once again sort the combined list of octants in pre-order. This newly sorted list is passed upward in this manner until the root node of the tree receives two locally sorted halves of the entire set of etree points. It then sorts them, and appends the fully sorted stream of octants to the final file.


ucvm2mesh

This utility creates a 3D regular mesh extracted from a set of CVMs. The CVM Etree may then be used in a wave propagation simulation such as AWP-ODC. It can produce meshes in a number of formats

  • AWP 3 value: float Vp, float Vs, float Rho
  • AWP 5 value: float Vp, float Vs, float Rho, float Qp, float Qs
  • AWP 8 vlaue: int i, int j, int k, float Vp, float Vs, float Rho, float Qp, float Qs
  • SORD: One file each for Vp, Vs, Rho (double precision)
  • netCDF (with optional netCDF package installed, and the mesh2netcdf utility)


Two versions of this utility are provided, a serial version and an MPI version.


Serial ucvm2mesh

The serial ucvm2etree is suitable for small to medium sized meshes, and is executed as follows:

Usage: ucvm2mesh [-h] -f config

Flags:
        -f: Configuration file
        -h: Help message

Version: 12.2.0


The following is an example ucvm2mesh configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/ucvm2mesh_example.conf:

# List of CVMs to query
ucvmlist=cvmh

# UCVM conf file
ucvmconf=../conf/kraken/ucvm.conf

# Gridding cell centered or vertex
gridtype=CENTER

# Spacing of cells
spacing=2000.0

# Projection
proj=+proj=utm +datum=WGS84 +zone=11
rot=-40.0
x0=-122.3
y0=34.7835
z0=0.0

# Number of cells along each dim
nx=384
ny=248
nz=25

# Partitioning of grid among processors
px=2
py=2
pz=5

# Vs/Vp minimum
vp_min=0
vs_min=0

# Mesh and grid files, format
meshfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.media
gridfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.grid
meshtype=IJK-12

# Location of scratch dir
scratch=/lustre/scratch/user/scratch


Parallel ucvm2mesh

The MPI version, ucvm2mesh-mpi, is useful for large and very large meshes. It accepts the same configuration file as the serial version. It provides an additional command-line option for stage-out of the mesh files from the locations specified in the conf file to a final directory location. This is useful on HPC systems that use temporary scratch space.

Usage: ucvm2mesh-mpi [-h] [-o dir] -f config

Flags:
        -f: Configuration file
        -o: Stage-out directory 
        -h: Help message

Version: 12.2.0

Performance Enhancements for ucvm2mesh on Lustre Filesystems

When running ucvm2mesh on Lustre filesystems (and other systems with similar structures), you may experience some performance issues that result in less than optimal performance. By default, Lustre uses one OST or disk per file. As such, running ucvm2mesh-mpi with a significant number of cores results in all those processes attempting to seek and write to the same file on the same OST/disk. This can result in a disk bottleneck that gives very poor performance, where it may take up to 10 minutes to write a few hundred MBs.

This can be fixed by increasing the number of OSTs/disks to which the output directory and files are written. This is called striping. Generally, you will want to set the stripes for the output directory to be either the recommended amount for large files or the maximum amount allowed. The command to set the number of stripes on Kraken is lfs setstripe [dir] --count [count]. Count can be a fixed number or it can be -1 for the maximum supported by the operating system. The Kraken documentation suggests that 80 is a reasonable amount for very large files. So, for example, suppose we were generating a large mesh with our stage out directory being /lustre/scratch/username/meshes/outstripe. We would first make that directory and then run:

lfs setstripe /lustre/scratch/username/meshes/outstripe --count 80

Then we would have our configuration file output to that directory:

meshfile=/lustre/scratch/username/meshes/outstripe/USC_cvms_awp.media gridfile=/lustre/scratch/username/meshes/outstripe/USC_cvms_awp.media.grid

When we qsub the job, it will automatically take advantage of striping and distribute the load across multiple targets.

It is also important to remember that more cores is not always better. Additional cores can result in increased write requests that may cause performance decreases. It is important to find the optimal balance between the number of processes writing to the file and the number of stripes. For exampe, for a large 600km x 400km x 40km meshes, a core count of 600 and stripe count of 80 on NICS Kraken will result in reasonably good performance.

grd_query

This command-line tool queries data from a set of ArcGIS grid files in GridFloat format.

Usage: grd_query [-e] [-v] -d datadir [-b bkgdir] < file.in

Flags:
        -e Employ GRD bathymetry heuristic.
        -v Employ GRD vs30 heuristic.
        -d Main ArcGIS Gridfloat data directory. Default is ./ned.
        -b Background/Bathymetry data directory. Default is ./bath.
        -h This help message.

Input format is:
        lon lat

Output format is:
        lon lat val valid

Notes:
        - If running interactively, type Cntl-D to end input coord list.

Version: 12.2.0


grd2etree

This utility extracts a SCEC-formatted Etree map from a set of DEM and Vs30 grid files in ArcGIS Gridfloat format.

Usage: grd2etree [-h] -f config

Flags:
        -f: Configuration file
        -h: Help message

Version: 12.2.0


The following is an example grd2etree configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/grd2etree_example.conf:

# grd2etree UCM Map config file

# Domain corners coordinates (degrees):
proj=+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0
lon_0=-129.75
lat_0=40.75
rot=55.0

# Domain dimensions (meters):
x-size=1800000.0
y-size=900000.0

# Spacing
spacing=250.0

# Etree parameters and info
title=UCVM_Elev_Vs30_Map_Wills_Wald
author=P_Small
date=05/2011
outputfile=../../model/ucvm/ucvm.e

# Grid data directories
elev_hr_dir=/home/scec-00/USER/opt/ned
elev_lr_dir=/home/scec-00/USER/opt/bath
vs30_hr_dir=/home/scec-00/USER/opt/vs30/wills_gridfloat
vs30_lr_dir=/home/scec-00/USER/opt/vs30/wald_gridfloat


Import the new Etree Map into UCVM by adding these lines to ${UCVM_INSTALL_DIR}/conf/ucvm.conf:

 newmodel_interface=map_etree
 newmodel_modelpath=config.outputfile

 where:
   newmodel          : name for your new model
   config.outputfile : outputfile from config passed into grd2etree


patchmodel

The utility provides a post-processing "patching" mechanism to smooth out discontinuities at the interfaces between different regional models. The user identifies a 3D geographic region in the form of an open box where smoothing is to be performed. The patchmodel program is run to extract the material properties along the vertical surfaces of this box at a fixed resolution (eg: 250 m) from 0m depth to "z-size" depth and it saves them into a new model which may then be imported into and queried by UCVM. Points that fall within the smoothing region are interpolated using an inverse distance weighting function with the 10% closest surface points on the same z-plane. A power factor of 2.0 is used for weighting. Linear interpolation is used along the z-axis to smooth out the edge values.

Usage: patchmodel [-h] -f config

Flags:
        -f: Configuration file
        -h: Help message

Version: 12.2.0


The following is an example patchmodel configuration file. It can also be found in ${UCVM_INSTALL_DIR}/conf/example/patch_example.conf:

# Example Patchmodel config file

# Version
version=cvmh_p1_2011/05/24

# UCVM config
ucvmstr=cencal,cvmh,lin-thurber,1d
ucvm_interp_zrange=0.0,350.0
ucvmconf=../../conf/aftershock/ucvm.conf

# Projection
proj=+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0
lon_0=-121.3
lat_0=35.6
rot=90.0

# Domain dimensions (meters):
x-size=550000.0
y-size=50000.0
z-size=112500.0

# Spacing (meters)
spacing=250.0

# Output paths
modelname=cvmh_p1
modelpath=/home/scec-00/USER/opt/aftershock/ucvm/model/patch


Import the new patch into UCVM by adding these lines to ${UCVM_INSTALL_DIR}/conf/ucvm.conf:

 newmodel_interface=model_patch
 newmodel_modelpath=config.modelpath/config.modelname.conf

 where:
   newmodel         : name for your new model
   config.modelpath : modelpath from config passed into patchmodel
   config.modelname : modelname from config passed into patchmodel

 Note: Be sure to append ".conf" to newmodel_modelpath


ecompact

This optimization utility compacts an etree file with any payload and schema. It does this by removing empty space in the Etree data structure.

Usage: ecompact in_etree out_etree


ecoalesce

This optimization utility coalesces an etree file that conforms to either the SCEC CVM Etree or CMU CVM Etree conventions. It does this by replacing eight adjacent octants with identical material properties at level N with a single octant containing the same material properties at level N-1.

Usage: ecoalesce in_etree out_etree


C Application Programming Interface

UCVM provides a programming interface for querying Vp, Vs, and rho from any user-defined model. The main UCVM interface is as follows:

#ifndef UCVM_H
#define UCVM_H

#include <stdarg.h>
#include "ucvm_dtypes.h"

/* Initializer */
int ucvm_init(const char *config);

/* Finalizer */
int ucvm_finalize();

/* Enable specific model(s), by string list, by label, or 
   by ucvm_model_t */
int ucvm_add_model_list(const char *list);
int ucvm_add_model(const char *label);
int ucvm_add_user_model(ucvm_model_t *m, ucvm_modelconf_t *conf);

/* Use specific map (elev, vs30) by label */
int ucvm_use_map(const char *label);

/* Associate specific interp func with GTL model, by label 
   or by ucvm_ifunc_t */
int ucvm_assoc_ifunc(const char *mlabel, const char *ilabel);
int ucvm_assoc_user_ifunc(const char *mlabel, ucvm_ifunc_t *ifunc);

/* Get label for a model */
int ucvm_model_label(int m, char *label, int len);

/* Get label for an interpolation function */
int ucvm_ifunc_label(int f, char *label, int len);

/* Get version for a model */
int ucvm_model_version(int m, char *ver, int len);

/* Set parameters (see ucvm_dtypes.h for valid param flags) */
int ucvm_setparam(ucvm_param_t param, ...);

/* Query underlying models */
int ucvm_query(int n, ucvm_point_t *pnt, ucvm_data_t *data);

/* Get info on installed models */
int ucvm_install_info(char *info, int len);

#endif

The general order of operations for using this interface is:

  • Initialization (ucvm_init())
  • Add models in order of preference using ucvm_add_model(). These may be either crustal or GTL models. See below for specific details.
  • Set any models-specific flags using ucvm_setparam. These will override any flags specified in ucvm.conf. For example, to enable the 1D background model with CVM-H through the API, execute the ucvm_setparam() function as follows:
ucvm_setparam(UCVM_PARAM_MODEL_CONF, UCVM_MODEL_CVMH, "USE_1D_BKG", "True");
  • If GTLs have been added:
    • Associate an interpolation function with each using ucvm_assoc_ifunc(). Otherwise, linear interpolation will be used.
    • Set an interpolation depth range using ucvm_setparam(UCVM_PARAM_IFUNC_ZRANGE, ...). Otherwise the interpolation range is set to 0,0.
  • Query points of interest with ucvm_query()
  • Finalization (ucvm_finalize())


Notes on enabling models:

  • Models are referenced either by a pre-defined string id or by a populated ucvm_model_t structure.
  • Multiple models may be registered, and they are queried in order of registration.
  • A helper function ucvm_add_model_list() is provided to quickly add multiple models. The string format is "gtl_model<:gtl_interp>,crustal_model,..." where "<:gtl_interp>" is an optional GTL interpolation function specification. For example, the string "elygtl:ely,cvmh" specifies the Ely GTL and CVM-H, adn the use of the Ely interpolation function to smooth between the GTL and crustal models.
  • Pre-defined models include: CVM-S, CVM-H, CenCal, Lin-Thurber, Rob Graves Cape Mendocino, 1D, Ely GTL, any SCEC Etree or patch listed in the UCVM configuration file. These models are enabled by calling ucvm_add_model() with their name string identifier as listed in either ucvm_dtypes.h (pre-defined) or in the UCVM configuration file (user-defined SCEC Etree, UCVM patch).
  • Most predefined model interfaces only support one model instance at a time. However, multiple Etrees and patches may be enabled. Etrees and patches are defined in the UCVM configuration file with the following commands:
# Etree definition and path to etree
usermodel1_interface=model_etree
usermodel1_modelpath=/home/...

# Patch definition and path to configuration file
usermodel2_interface=model_patch
usermodel2_modelpath=/home/...

where:
usermodel1/usermodel2 : replaced with desired string identifier for the etree/patch model
  • The programmer may also define their own models and integrate them into UCVM. A user model interface is defined by creating the init(), finalize(), getversion(), getlabel(), setparam(), and query() functions for the model interface, and populating the ucvm_model_t structure with these function pointers and some additional record keeping information. Configuration information for a specific model instance of that interface is placed in a ucvm_modelconf_t structure. Once a model is fully described with a ucvm_model_t interface and a ucvm_modelconf_t configuration, it can be registered for use with the ucvm_add_user_model() function and queried with the ucvm_query() function.


Data Type Descriptions

The full list of defined data types are listed in /include/ucvm_dtypes.h. The following is a general description of the useful application definitions:

Points are specified with a ucvm_point_t structure:

/* 3D point */
typedef struct ucvm_point_t 
{
  double coord[3];
} ucvm_point_t;


Return data is specified with a ucvm_data_t structure:

typedef struct ucvm_data_t 
{
  double surf; /* Elevation of free surface in meters */
  double vs30; /* Vs30 value in m/s */
  ucvm_domain_t domain;  /* Domain where the point lies: GTL model, interpolation zone, crustal model, none. */
  double shift_cr; /* Depth shift applied to point when querying crustal models. Only used when a GTL is enabled. */
  double shift_gtl; /* Depth shift applied to point when querying GTL models. Only used when a GTL is enabled. */
  ucvm_prop_t crust; /* Material properties from crustal model */
  ucvm_prop_t gtl; /* Material properties from GTL model */
  ucvm_prop_t cmb; /* Combined GTL/crustal material properties. This is the final property set that applications should reference. */
} ucvm_data_t;

where ucvm_prop_t is:

/* Material properties */
typedef struct ucvm_prop_t 
{
  int source; /* Model idenfitifer */
  double vp; /* Velocity in m/s */
  double vs; /* Velocity in m/s */
  double rho; /* Density in g/m^3 */
} ucvm_prop_t;


Model interfaces (both crustal and GTL) are defined with a ucvm_model_t structure:

/* Model */
typedef struct ucvm_model_t 
{
  ucvm_mtype_t mtype;
  int (*init)(int id, ucvm_region_t *r, const char *config);
  int (*finalize)();
  int (*getversion)(int id, char *ver, int len);
  int (*getlabel)(int id, char *ver, int len);
  int (*setparam)(int id, int param, ...);
  int (*query)(int id, ucvm_ctype_t cmode,
               int n, ucvm_point_t *pnt, 
               ucvm_data_t *data);
} ucvm_model_t;


Other Interfaces

2D Grid Interface

Provides an interface for generating 2D regular grids in any USGS map projection. Include the header "ucvm_grid.h" to use these functions.

/* Generate grid from projection and dimensions */
int ucvm_grid_gen(ucvm_proj_t *iproj, ucvm_trans_t *trans,
                  ucvm_proj_t *oproj,
                  ucvm_dim_t *dims, double spacing, 
                  ucvm_point_t *pnts);

/* Generate grid from projection and dimensions */
int ucvm_grid_gen_file(ucvm_proj_t *iproj, ucvm_trans_t *trans,
                       ucvm_proj_t *oproj,
                       ucvm_dim_t *dims, double spacing, 
                       const char *filename);

/* Convert point list from one projection to another */
int ucvm_grid_convert(ucvm_proj_t *iproj, 
                      ucvm_proj_t *oproj, 
                      size_t n, ucvm_point_t *pnts);

/* Convert point list from one projection to another */
int ucvm_grid_convert_file(ucvm_proj_t *iproj, 
                           ucvm_proj_t *oproj, 
                           size_t n, const char *filename);


Extracting Values from UCVM

Two methods are provided for extracting material properties from UCVM: command-line tools, and a C API. These are described in the sections below. The following table summarizes the model features supported by each interface:


Extracting Values on the Command Line

The ucvm_query utility is provided for querying a set of community velocity models. Points may either be input interactively or via a text file, and can be either (lon, lat, elevation) or (lon, lat, depth).

The following output shows an interactive example on three points (followed by a Cntl-D to terminate input) using the CVM-S model. Starting in the UCVM installation directory, enter these commands:

% cd bin
% ./ucvm_query -m cvms -f ../conf/ucvm.conf
Using Geo Depth coordinates as default mode.
-118.0 34.0 0.0
-118.0 34.0 100.0
-118.0 34.0 1000.0 <Cntrl-D>
 -118.0000    34.0000      0.000    295.679    390.000       cvms    696.491    213.000   1974.976       none      0.000      0.000      0.000      crust    696.491    213.000   1974.976
 -118.0000    34.0000    100.000    295.679    390.000       cvms   1683.174    603.470   2130.773       none      0.000      0.000      0.000      crust   1683.174    603.470   2130.773
 -118.0000    34.0000   1000.000    295.679    390.000       cvms   3660.809   2056.628   2443.042       none      0.000      0.000      0.000      crust   3660.809   2056.628   2443.042

Similarly, here is an example for the CVM-H model:

% cd bin
% ./ucvm_query -m cvmh -f ../conf/ucvm.conf
Using Geo Depth coordinates as default mode.
-118.0 34.0 0.0
-118.0 34.0 100.0
-118.0 34.0 1000.0 <Cntrl-D>
 -118.0000    34.0000      0.000    295.679    390.000       cvmh    824.177    195.000   1084.062       none      0.000      0.000      0.000      crust    824.177    195.000   1084.062
 -118.0000    34.0000    100.000    295.679    390.000       cvmh   2371.500    727.748   2051.320       none      0.000      0.000      0.000      crust   2371.500    727.748   2051.320
 -118.0000    34.0000   1000.000    295.679    390.000       cvmh   2616.063   1082.813   2127.783       none      0.000      0.000      0.000      crust   2616.063   1082.813   2127.783


Additional Examples

The following example directs ucvm_query to query the Ely GTL and CVM-S crustal model by depth, using the Ely interpolation function to smooth the values. The Z-axis interpolation zone is set to 0-400m.

% ./ucvm_query -m elygtl:ely,cvms -c gd -z 0.0,400.0 -f ../conf/ucvm.conf


The following example queries the USGS Bay Area (CenCalVM) and CVM-H models by elevation, tiling them into a composite model (where CenCalVM has priority).

% ./ucvm_query -m cencal,cvmh -c ge -f ../conf/ucvm.conf


The following example queries the 1D model by depth, with surface elevation and Vs30 values taken from the alternate Yong-Wald Vs30 map.

% ./ucvm_query -m 1d -p yong -c gd -f ../conf/ucvm.conf


Extracting Values in a C Program

Working code examples that use the UCVM A API can be found in the directory ${UCVM_INSTALL_DIR}/examples. Feel free to use these examples as a basis for your own programs. In general, using the API in your program requires the following source/Makefile updates.

Include the ucvm.h header in your source code at a minimum. Eg:

#include "ucvm.h"


Set the include path to the location of the UCVM headers, the Etree and Proj.4 headers, and all of the standard model include paths. Eg, for GNU C compiler and CVM-S:

CFLAGS = $(CFLAGS) -I$(UCVM_INSTALL_DIR)/include  -I$(CVMS_INSTALL_DIR)/include -I$(ETREE_INSTALL_DIR)/libsrc -I$(PROJ4_INSTALL_DIR)/include


Identify the library paths and names to UCVM static library, the Etree and Proj.4 libraries, and all of the standard model library paths needed by your application to the linker. Depending on platform, also include the pthreads and math libraries. Eg, for GNU linker and CVM-S (note the use of the gfortran lib since CVM-S is Fortran):

LDFLAGS = $(LDFLAGS) -L$(UCVM_INSTALL_DIR)/lib ucvm -L($CVMS_INSTALL_DIR)/lib cvms -L$(ETREE_INSTALL_DIR)/libsrc etree -L$(PROJ4_INSTALL_DIR)/lib proj 
          -lgfortran -pthreads -lm


Example Programs

Please see the ./examples directory of the software distribution for compiled examples that employ the C API.


History of UCVM Releases

  • 12.2.0 (Feb 9, 2012): Initial release


Acknowledgements and Contact Info

Support for the development and maintenance of the UCVM framework has been provided by the Southern California Earthquake Center (SCEC). SCEC is funded by NSF Cooperative Agreement EAR-0106924 and USGS Cooperative Agreement 02HQAG0008.

Contributions to this manual were made by: Patrick Small, and Philip Maechling.

Please email software@scec.org for help on downloading and using UCVM, and for any suggestions for the delivery of the code or for this manual.

Please reference at least Small et al (2011) if you use this software framework; other references should be considered, depending on the purpose.


Technical Notes

SCEC CVM Etree Convention

Etree Model Format

  • Variable resolution based off of Vs, max supported frequency, and points per wavelength. The resolution at any particular region will be dependent upon the local Vs value (relation shown below). However, a minimum Vs at Etree creation will clamp the resolution to a maximum value regardless of how low the local Vs is found to be.
rez = vs / (ppwl * freq)

where:
vs: Vs at point of interest (m/s)
ppwl: points per wavelength
freq: max supported frequency (Hz)
  • Model data is referenced relative to free surface. Free surface is defined at air/ground, water/ground interface. UCVM topography and bathymetry can be used to determine if a point is in the air/water and default properties returned for those domains.
  • Stored properties are Vp, Vs, and density.


Etree Metadata Description

The Etree metadata tag describes the payload of the octants, the extents of the coverage region, and the map projection. It will have the following format, with fields delimited by vertical bars:

metatype_version|title|author|date|vs_min|max_freq|ppwl|format|proj.4_string|geo_origin|rot_angle|xyz_dims|xyz_ticks

where:
  metatype_version: "SCEC_CVM_V1" string of max length 64 chars
  title: free-format title string of max length 64 chars
  author: free-format author string of max length 64 chars
  date: YY/MM/DD date string
  vs_min: vs minimum used to determine max resolution
  max_freq: maximum supported frequency in Hz
  ppwl: Points per wavelength
  format: data format specification of octants. Uses string format: "name,data_type,size,units". eg: "vp,float,4,m/s;vs,float,4,m/s;rho,float,4,kg/m^3"
  proj.4_string: Proj.4 projection string, eg: "+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0"
  geo_origin: Origin point of CVM region in geo coordinates, eg: "-129.25,41.0"
  rot_angle: Rotation angle of bounding box in degrees, eg "55.0"
  xyz_dims: Dimensions of bounding box in km, eg "1800,800,100"
  xyz_ticks: Number of ticks along each dimension


SCEC Patch Model Format

A patch model consists of five files: a configuration text file, and four binary files with the surface data. All of these files are automatically generated by the patchmodel utility. The sections below describe the file formats.


Configuration file

The patch model configuration file specifies the origin, dimensions, and map projection of the 3D region covered by the patch. An example follows:

# cvmh patch conf file

# Version
version=cvmh_p1_2011/05/24
# Projection
proj=+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0
lon_0=-121.300000
lat_0=35.600000
rot=90.000000
# Spacing
spacing=250.000000
# Dimensions
x-size=550000.000000
y-size=50000.000000
z-size=112500.000000
# Paths to surface files
surf_0_0_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_0_0.bin
surf_1_0_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_1_0.bin
surf_0_1_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_0_1.bin
surf_1_1_path=/home/scec-00/USER/opt/aftershock/ucvm/model/patch/cvmh_p1_surf_1_1.bin


Surface Files

The surface file format is a 2D array of three floating point values: vp, vs, and density. The assignment of x versus y axis in each file is dependent on the orientation of the surface with respect to the box's origin point.


SCEC Map Etree Convention

Etree Model Format

  • 3D Etree with a single octant depth along Z-axis (effectively a 2D grid)
  • Fixed resolution based off of maximum desired grid spacing in grd2etree configuration.
  • Sampled with grid registration to allow smoothing of adjacent octants in UCVM.
  • Stored values at each grid point are: surface elevation (float), Vs30 (float).


Etree Metadata Description

The Etree metadata tag describes the payload of the octants, the extents of the coverage region, and the map projection. It will have the following format, with fields delimited by vertical bars:

metatype_version|title|author|date|spacing|format|proj.4_string|geo_origin|rot_angle|xy_dims|xyz_ticks

where:
  metatype_version: "SCEC_MAP_V1" string of max length 64 chars
  title: free-format title string of max length 64 chars
  author: free-format author string of max length 64 chars
  date: YY/MM/DD date string
  spacing: grid spacing in meters
  format: data format specification of octants. Uses string format: "name,data_type,size,units". eg: "surf,float,4,m/s;vs30,float,4,m/s;"
  proj.4_string: Proj.4 projection string, eg: "+proj=aeqd +lat_0=36.0 +lon_0=-120.0 +x_0=0.0 +y_0=0.0"
  geo_origin: Origin point of CVM region in geo coordinates, eg: "-129.25,41.0"
  rot_angle: Rotation angle of bounding box in degrees, eg "55.0"
  xy_dims: Dimensions of bounding box in km, eg "1800,800"
  xyz_ticks: Number of ticks along each dimension


References

  1. Ely, G., T. H. Jordan, P. Small, P. J. Maechling (2010), A Vs30-derived Near-surface Seismic Velocity Model Abstract S51A-1907, presented at 2010 Fall Meeting, AGU, San Francisco, Calif., 13-17 Dec. [Ely2010-AGU.pdf]
  2. Graves, R. (1994), Rupture History and Strong Motion Modeling of the 1992 Cape Mendocino Earthquake, USGS External Grant Report Cape mendocino-19941027.pdf
  3. Lin, G., C. H. Thurber, H. Zhang, E. Hauksson, P. Shearer, F. Waldhauser, T. M. Brocher, and J. Hardebeck (2010), A California statewide three-dimensional seismic velocity model from both absolute and differential Times, Bull. Seism. Soc. Am., 100, in press. supplemental
  4. Small, P., P. Maechling, T. Jordan, G. Ely, and R. Taborda (2011), SCEC UCVM - Unified California Velocity Model, in 2011 Southern California Earthquake Center Annual Meeting, Proceedings and Abstracts, vol. TBD, p. TBD.
  5. Taborda R., López J., O'Hallaron D., Tu T. and Bielak J. (2007), A review of the current approach to CVM-Etrees, SCEC Annual Meeting, Palm Springs, CA, USA, September 8–12. [1]
  6. Wald, D. J., and T. I. Allen (2007), Topographic slope as a proxy for seismic site conditions and amplification, Bull. Seism. Soc. Am., 97 (5), 1379-1395, doi:10.1785/0120060267.
  7. Wills, C. J., and K. B. Clahan (2006), Developing a map of geologically defined site-condition categories for California, Bull. Seism. Soc. Am., 96 (4A), 1483-1501, doi:10.1785/0120050179.
  8. Yong, A., Hough, S.E., Iwahashi, J., and A. Braverman (2012), A terrain-based site conditions map of California with implications for the contiguous United States, Bull. Seism. Soc. Am., Vol. 102, No. 1, pp. 114–128, February 2012, doi: 10.1785/0120100262.


License and Disclaimer

License

Copyright (c)2011 Southern California Earthquake Center (SCEC) and the University of Southern California (USC). This copyright applies to the UCVM codebase as a whole, or any individual distributed applications. The individual contributions of government employees, which may be identified on a per-file basis using Subversion (SVN) (see developer site), are in the public domain.

The software and content provided on this website are made available under the terms of the Apache License, Version 2.0. A copy of the License is provided by the Apache Software Foundation (http://www.apache.org/licenses/LICENSE-2.0).

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.


Disclaimer

THIS SOFTWARE AVAILABLE ON THE UCVM WEBSITE IS PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE SOUTHERN CALIFORNIA EARTHQUAKE CENTER (SCEC), THE UNIVERSITY OF SOUTHERN CALIFORNIA (USC), OR ANY OF THEIR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


Without limiting the foregoing, SCEC and USC make no warranty that:

1. The software will meet your requirements.

2. The software will be uninterrupted, timely, secure or error-free.

3. The results that may be obtained from the use of the software will be effective, accurate or reliable.

4. The quality of the software will meet your expectations.

5. Any errors in the software obtained from the UCVM web site will be corrected.


Software and its documentation made available on the UCVM web site:

6. Could include technical or other mistakes, inaccuracies or typographical errors. UCVM contributors may make changes to the software or documentation made available on its web site.

7. May be out of date and SCEC, its contributors, and USC make no commitment to update such materials.


SCEC, its contributors, and USC assume no responsibility for errors or omissions in the software or documentation available from the UCVM web site.

In no event shall SCEC, its contributors, or USC be liable to you or any third parties for any special, punitive, incidental, indirect or consequential damages of any kind, or any damages whatsoever, including, without limitation, those resulting from loss of use, data or profits, whether or not SCEC, its contributors, or USC has been advised of the possibility of such damages, and on any theory of liability, arising out of or in connection with the use of this software.

The use of the software downloaded through the UCVM site is done at your own discretion and risk and with agreement that you will be solely responsible for any damage to your computer system or loss of data that results from such activities. No advice or information, whether oral or written, obtained by you from SCEC, the UCVM website, its contributors, or USC shall create any warranty for the software.