Cvmbycols

Overview

Cvmbycols is a set of codes for producing compacted Etrees for geographic regions with material properties drawn from a community velocity model. Serial and MPI versions of the program are provided. The tool makes use of several existing software packages, including:

• CMU Euclide Etree, available from http://www.cs.cmu.edu/~euclid/
• Unified California Velocity Model, available from UCVM
• Community Velocity Models, including one or more of these:
  • CVM-H, available from CVM-H
  • CVM-S, available from CVM-S
  • USGS CenCal, available from http://earthquake.usgs.gov/regional/nca/3Dgeologic/

This SCEC version of cvmbycols is based on an older version originally developed by Tiankai Tu et al. at CMU.

Algorithm Description

This program uses several algorithmic approaches to increase extraction speed and minimize the amount of disk space required to store the Etree. These are described in the following sections.

Optimized Etree Resolution Based on Freq, PPWL, and Min Vs

• Divide region in set of x,y columns, where the length of each column equals its width.
• Starting at depth = 0m, extract a 2D grid for the column at the resolution needed to support the max frequency, pounts/wavelength, and minimum Vs.
  • If the local minimum Vs is in this 2D grid allows for a lower resolution, and the current z value is divisible by the lower resolution, requery the CVM at the lower resolution. Otherwise, if the local min Vs in this grid needs a higher resolution, increase the resolution to what is required (up to the maximum resolution dictated by the configured global minimum Vs) and requery the CVM.
  • Write this layer to an Etree (either a final etree or an etree local to this core)
  • Step the depth down by the resolution of this 2D grid and loop to the next layer.
• Loop to the next column.

High-performance Etree merging and compaction

Generation of an Etree in a parallel environment is challenging since the Euclid etree interface only allows serial write access to the etree file. Points may only be inserted, or they may be appended in key order, by a single writer. This makes etee construction and compaction time consuming. To circumvent this issue, a high performance etree merge/compaction algorithm has been developed.

Simply put, it is essentially a merge sort performed on 2^N small etrees that contain all of the points needed for the final etree. The small etrees contain material properties previously extracted from UCVM. This extraction may be done serially or in parallel as described in the sections below. Once these small, local etrees exist on disk, an MPI job of 2^N cores is then able to sort these octants by their keys in O(n*log n) time. Once all the octants are sorted, a single core may then open the final etree for writing in append mode, and simply stream the sorted octants to the final etree efficiently.

Conceptually, the cores in this 2^N MPI job are organized into a tree-like structure with each node containing two children. The bottom layer of this tree has nodes that each do a pre-order traversal on two local etrees. They sort the combined octants read from each etree in pre-order, and then pass them up to a parent node. The nodes from the top layer to the second to bottom layer each receive locally sorted lists of octants from their two children and once again sort the combined list of octants in pre-order. This newly sorted list is passed upward in this manner until the root node of the tree receives two locally sorted halves of the entire set of etree points. It then sorts them, and appends the fully sorted stream of octants to the final etree file.

Programs

The package provides a number of programs and utilities. The Cvmbycols etree creator is described below:

Cvmbycols Serial

This program runs as a single-core command-line application. It works by dividing the etree region into columns, then extracts each column from the underlying CVM and inserts the octants into the final Etree.

Cvmbycols MPI

This program is actually two separate MPI programs. One program performs a parallel extraction from UCVM, and other other a merge to form the final etree. These are very high performance applications, capable of generating large etrees.

Cvmbycols-extract-MPI

Divides the etree region into X columns for extraction. This is an embarrassingly parallel operation. A dispatcher (rank 0) farms out each column to a worker in a pool of N cores for extraction. Each worker writes its own local etree. After program execution, there are N small etrees. The extractor must be run on 2^N + 1 cores.

Cvmbycols-merge-MPI

Merges N small etrees into a final, compacted etree. This is essentially a merge sort on the keys from the addresses extracted from the local etrees. The cores at the lowest level of the merge tree each read in octants from two etrees with a pre-order traversal, merge sort the addresses, then pass the locally sorted list of addresses to a parent node for additional merging. This proceeds until the points rise to rank 1 which has a completely sorted list of etree addresses. Rank 0 takes this sorted list and performs a transactional append on the final etree. The merger must be run on 2^N cores.

Merging is a very fast opertion. Speeds in excess of 3M points/sec have been achieved on NICS Kraken.

Defining properties of an Etree

The following properties are defined:

• Bounding box of geographic region in lon,lat coordinates
• Extents of Etree volume in 3D, in km
• Number of columns in x,y axis to partition 3D space. The resulting columns must be square along the x-y axis.
• Frequency limit to support in Hz
• Number of points per wavelength to support
• Minimum Vs to support
• Maximum octant size to allow
• Name of the output etree file
• CVM to use for material properties (from UCVM)

User Guide

Configuring Cvmbycols

An example configuration file follows:

# Domain corners coordinates (degrees):
proj=geo-bilinear
lon_0=-119.288842
lat_0=34.120549

lon_1=-118.354016
lat_1=35.061096

lon_2=-117.780976
lat_2=33.096503

lon_3=-116.846030
lat_3=34.025873

# Domain dimensions (meters):
x-size=180000.0000
y-size=135000.0000
z-size=61875.0000

# Blocks partition parameters:
nx=256
ny=192

# Max freq, points per wavelength
max_freq=4.0
ppwl=8.0

# Etree min/max resolution as power of 2
max_cellsize=500.0

# Etree parameters and info
title=ChinoHills4Hz200ms
author=P_Small_and_R_Taborda
date=03/2011
outputfile=/lustre/scratch/patricks/cvmbycols/trunk/src/cvmbycols/etree_cvmh_chino_4hz_200ms.e

# CVM to use
#cvm=cvms
cvm=cvmh
ucvmconf=../../conf/kraken/ucvm.conf

# Min Vs, Vp
vs_min=200.0

# Scratch dir
scratch=/lustre/scratch/patricks/tmp

Running Cvmbycols MPI

The following PBS script for NICS Kraken illustrates how cvmbycols may be submitted to the batch scheduler:

#!/bin/sh
#PBS -q medium
#PBS -l size=1548
#PBS -l walltime=08:00:00
#PBS -o /lustre/scratch/${USER}/cvmbycols/trunk/pbs/kraken/chino_cvmh_4hz_200ms.out
#PBS -e /lustre/scratch/${USER}/cvmbycols/trunk/pbs/kraken/chino_cvmh_4hz_200ms.err
#PBS -V

HOME_DIR=/lustre/scratch/${USER}/cvmbycols/trunk/src/cvmbycols

cd ${HOME_DIR}

echo "Starting jobs"
date

aprun -n 1025 -S 4 ./cvmbycols-extract-MPI -f ../../conf/kraken/chino_cvmh_4hz_200ms.conf

aprun -n 1024 -S 4 ./cvmbycols-merge-MPI -f ../../conf/kraken/chino_cvmh_4hz_200ms.conf

echo "Jobs done"
date
exit 0

Note that one additional core is needed as a dispatcher for the extractor.

Source Code

SVN: https://source.usc.edu/svn/cvmbycols

Supported CVMs

The cvmbycols utility is built upon UCVM. As a result, it can extract etrees from the following CMV sources:

• CVM-H
• CVM-S
• USGS Cencal (which itself is an etree)
• 1D