Parallel Broadband

Overview

The original SCEC Broadband Platform executes a simulation workflow in a serial manner. Each code within each module, and the workflow itself, is executed serially. All BBP simulations involve these sequence of steps:

1. rupture generation
2. low-frequency seismogram synthesis for N stations
3. high-frequency seismogram synthesis for N stations
4. application of site effects for N stations
5. goodness of fit calculations
6. plotting

Rupture generation reads in a few input files and outputs a single source rupture description. Seismogram synthesis and site effects run on a list of N stations. Goodness of fit and plotting require information from the previously generated seismograms of all stations.

There are three levels in which the platform can be parallelized:

1. Parallelize the scientific codes used in each module (MPI, OpenMP)
2. Parallelize the individual simulation workflow (parallelize the seismogram synthesis by station)
3. Parallelize groups of simulations (multiple simulation workflows can execute simultaneously)

The approach described here is the second option. Seismogram synthesis and site effects are broken out by station and those modules are run in parallel. A simulation involving 100 stations may be run on up to 100 cores, giving an approximately 100x speed-up in execution speed.

Parallel Broadband

A new software layer has been added on top of the existing BBP software stack that allows a user to parallelize an existing serial workflow and execute it either on multiple cores within a single host or multiple nodes on a HPC resource. The user begins by creating a simulation workflow description in the traditional way (using run_bbp.py) and saving it to disk. Then, the new parallel interface is invoked on that workflow description. The serial workflow is parsed, parallelized, and optionally executed.

Supported System Configurations

Parallel Broadband can be run on three types of systems:

1. Any stand-alone, multi-processor computer running Linux
2. Any HPC resource running Linux and a job scheduler that supports multi-node jobs, and MoM nodes that can ssh into the compute nodes of the job
3. Remotely to any system as described in #2 via GRAM (requires BBP to be installed on both the HPC host and the remote system)

So, Parallel Broadband can be run on a quad-core desktop computer and use all four cores. It can also run as a PBS/Torque job submitted to USC HPCC, and use as many cores as there are stations in the simulation.

Installation

Parallel BBP is fully integrated with the original Broadband platform and shares its installation process. The parallel functionality may require an optional configuration step depending on how the system is run. When run on an HPC host, ssh is used to login to compute nodes from a MoM node. The user may need to setup environment variables (PATH and PYTHONPATH, for example) immediately following that login. The shell script ./comps/setup_bbp_env.sh is the place to declare those environment settings.

If run on a stand-alone computer, customizing that shell script is unnecessary.

Converting Serial Workflow to Parallel Workflow

TBD

Running the Parallel Workflow

TBD