Tomography Codes

There are three codes described here which may be of use in the Central California Tomography project: AWP-ODC (forward, CPU version), AWP-ODC-SGT (reciprocity, CPU version), jbsim3d (seismogram synthesis). Steps to acquire these codes and run them are described below.

AWP-ODC (forward, CPU version)

Checkout

To checkout AWP-ODC, run the following command:

svn checkout --username <your SCEC username> https://source.usc.edu/svn/cybershake/import/trunk/AWP-ODC/AWP-ODC-v1.1.2

Enter your SCEC password (the one you use for SCEC computers) when prompted.

Compiling

To compile AWP-ODC, cd into the src-v1.1.2 directory. Use the GNU compilers and start with the default makefile (makefile). I'm not familiar with Mira, so you may need to change the OFLAGS and PFLAGS to match whatever's in En-Jui's version of pmcl3d which compiles on Mira.

Running

This code takes an IN3D file; it should be the same format that you're currently using. I believe if you set the flag ISFCVLM to 1 it will produce volume output.

AWP-ODC-SGT (CPU version)

Checkout

To checkout AWP-ODC-SGT, issue the following command:

svn checkout --username <your SCEC username> https://source.usc.edu/svn/cybershake/import/trunk/AWP-ODC-SGT .  

It will prompt you for a password; enter your SCEC password. This is the same username and password you would use to access SCEC servers.

Compiling

cd into the src directory. On Blue Waters, use the Blue Waters makefile:

make -f makefile.bluewaters

I've never compiled it on Mira. I would suggest taking the Blue Waters makefile and changing the OFLAGS and PFLAGS to match the flags in the makefile for En-Jui's version of pmcl3d which you have successfully compiled on Mira.

Input files

AWP-ODC-SGT takes an IN3D file; the format is described [https://scec.usc.edu/it/AWP-ODC-SGT here (login required). In particular, the SGT-specific parameters igreen, NTISKP_SGT, SGTGRO, and INSGT must be set.

INSGT is a text input file which contains a list of all the points you'd like to save SGTs for. It has a header line which contains the total number of SGT points, then each line has one of those points as x y z coordinates. For example, a file with 2 points, (0,0,1) and (100,400,5) would look like:

2
0 0 1
100 400 5

A source is also needed for SGT calculations. For CyberShake, we have pre-created impulses for various DTs and the X and Y components - if you look in AWP-ODC-SGT/utils/data, you will see sample sources for 20000 and 40000 timesteps (fx_src_20000, fx_src_40000, etc.) These are simple pulses.

Running

To run it, I use a simple wrapper in the AWP-ODC-SGT directory, awp_odc_wrapper.sh. You can call this from a batch script, though the details of the aprun call will vary depending on the particular system you run it on. It takes the IN3D file as input.

JBSim3d_header

I think this is the best version of the seismogram synthesis codes we use in CyberShake for your use. All of them use the same basic code to do the synthesis, but have different levels of optimization and CyberShake-specific add-ons. This one contains some extra code which helps it interface with AWP-ODC-SGT output. It has a dependency on Getpar to read its arguments.

Note that if you want to tie this in with

Checkout

First, check out Getpar:

svn checkout --username <SCEC username> https://source.usc.edu/svn/cybershake/import/trunk/Getpar .

To check JBSim3d_header out, run:

svn checkout --username <SCEC username> https://source.usc.edu/svn/cybershake/import/trunk/JBSim3d_header .

When prompted, enter the password you use for SCEC computing resources.

Compiling

Compile Getpar first:

1. cd into Getpar/getpar/src.
2. Edit Makefile; comment out line 56 ("../../../Compilers.mk") and edit the next 2 lines to that CC and FC point to the names of the C and Fortran compilers in your environment. On many environments it's gcc and gfortran; on Blue Waters it's cc and ftn; on Mira I think it's bgxlc and bgxlf, but I could be wrong.
3. Then run make.

Once that's done, compile JBSim3d_header. cd into JBSim3d_header/src directory. Load the GNU compilers into your environment, and change CC and FC in makefile to the names of the C and Fortran compilers in your environment, like you did with Getpar. Then run make. It will produce the executable jbsim3d.

Running

To run, jbsim3d requires a series of arguments, in the form key=value on the command line. The keys you need to provide are:

• stat: name of your station
• slon: longitude of the station
• slat: latitude of the station
• outputBinary=1: Output a binary seismogram, not text
• mergeOutput=1: Output a single seismogram with multiple components
• ntout: Number of timesteps in the output seismogram
• dtout: Size of timesteps in the output seismogram
• rupmodfile: Path to SRF file describing earthquake
• sgt_xfile: Path to SGT file with X-component data. This must have certain header information, or must be supplied separately in a header file.
• sgt_yfile: Path to SGT file with Y-component data. This must have certain header information, or must be supplied separately in a header file.
• x_header: Separate header file, with X-component header data.
• y_header: Separate header file, with Y-component header data.
• seis_file: Output seismogram file

In order to generate seismograms, jbsim3d must be run twice. The first time, in addition to the arguments above, also provide extract_sgt=1, extract_sgt_xfile=<extracted_x_sgtfile>, and extract_sgt_yfile=<extracted_y_sgtfile>, and leave out seis_file. What this is doing is taking the large SGT file, which has SGTs for the whole volume, and pulling out only the SGTs needed for this particular earthquake. Once this is done, run jbsim3d again, with extract_sgt=0. Also, point sgt_xfile to <extracted_x_sgtfile>, and sgt_yfile to <extracted_y_sgtfile>.

For now, I would strongly suggest we not worry about synthesizing seismograms. In order to use jbsim3d to synthesize seismograms from AWP SGTs, the separate header files must be created. CyberShake has codes to do this, but right now they take as input certain files which are produced as part of CyberShake which the tomography workflow doesn't have. Some code development will be required to bridge this gap in the tomography workflow.