Difference between revisions of "Accessing CyberShake Seismograms"

From SCECpedia
Jump to navigationJump to search
 
(7 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
== Identifying Seismograms of Interest ==
 
== Identifying Seismograms of Interest ==
  
The first step is to figure out which seismograms are of interest.  To do this, you'll need to know a) the site and run ID of the CyberShake run, and b) the source id, rupture id, and rupture variation id of the seismogram.  You may have done this already.  If not, you can determine the run ID by using MySQL to query the CyberShake database on focal.usc.edu.  To determine a run ID, you will need to know values for a few parameters, such as the ERF, the velocity model, and the rupture variation generator version.  Here's a sample query:
+
The first step is to figure out which seismograms are of interest.  You'll need to access the database to determine this; instructions are available at [[Accessing CyberShake Database Data]]. To do this, you'll need to know a) the site and run ID of the CyberShake run, and b) the source id, rupture id, and rupture variation id of the seismogram.  You may have done this already.  If not, you can determine the run ID by using MySQL to query the CyberShake database on focal.usc.edu.  To determine a run ID, you will need to know values for a few parameters, such as the ERF, the velocity model, and the rupture variation generator version.  Here's a sample query:
  
 
  select R.Run_ID from CyberShake_Runs R, CyberShake_Sites S
 
  select R.Run_ID from CyberShake_Runs R, CyberShake_Sites S
Line 50: Line 50:
 
=== Data organization ===
 
=== Data organization ===
  
The data is organized by site name and run ID.  So, for example, there's a directory for site ACTN and then run ID 4296.  Inside this directory are all the seismograms, PSA, RotD, and duration files for site ACTN, run ID 4296.  The data for each site is approximately 90 GB.
+
The data is organized by site name and run ID.  So, for example, there's a directory for site ACTN and then run ID 4296.  Inside this directory are all the seismograms, PSA, RotD, and duration files for site ACTN, run ID 4296.  The filenames are <Seismogram | PeakVals | RotD | Duration>_<site>_<source ID>_<rupture ID>.<grm | bsa | rotd | dur> .  The data for each run of Study 15.12 is approximately 90 GB.
  
 +
== Seismogram Format ==
  
 +
The seismogram file Seismogram_<site>_<sourceID>_<ruptureID>.grm contains the seismograms for all the rupture variations for the given source ID.
  
==== Using the Seismogram Retrieval Script ====
+
The seismograms are velocity seismograms, with <b>units of cm/s</b>.  The number of timesteps and dt depends on what frequency the run was performed at, as well as whether the run included stochastic high-frequency results, and can be determined by reading the header information.
  
Once you know the run ID and the event of interest, you can use a Python script, /home/scec-02/cybershk/utils/retrieve_seismogram.py (and in SVN at https://source.usc.edu/svn/cybershake/import/trunk/Utils/retrieve_seismogram.py), to retrieve the seismogram for you.  It takes as arguments the run ID, source ID, rupture ID, rupture variation ID, output file, and if you want the seismogram header information included or not.  This script automatically searches the directories below, retrieves the specific two-component seismogram of interest, and writes an output file.  It is highly recommended to use this script rather than looking for the file by hand.
+
The format is (binary):
  
==== Locating Seismogram Files by Hand ====
+
  <Rupture Variation 1 56-byte header>
 
 
If you prefer, you can try to find the files on disk yourself.  There are a couple of places to check.
 
 
 
There is a directory on the SCEC servers which corresponds to the specific run ID.  It will be in one of:
 
 
 
/home/scec-04/tera3d/CyberShake/data/PPFiles/<site>/<runID>
 
/home/scec-04/tera3d/CyberShake/data/from_scec-02/PPFiles/<site>/<runID>
 
/home/scec-02/tera3d/CyberShake2007/data/PPFiles/<site>/<runID>
 
 
 
Inside that directory you will see 1 of 2 possibilities -- we changed our file format about a year ago, so we have data in both formats.
 
 
 
a) In the older format, the directory will contain a number (10-80) of Seismogram*.zip files.  All broadband runs fall into this category.  One of these zip files has the seismogram you want, but there's no straightforward way to tell.  I recommend running the following from the command line (in bash):
 
 
 
for i in `ls *_seismograms.zip`; do echo $i; unzip -l $i | grep Seismogram_<site>_<sourceID>_<ruptureID>_<ruptureVariationID>.grm; done
 
 
 
When you run it, you'll get output like:
 
 
 
CyberShake_LADT_749_3_seismograms.zip
 
CyberShake_LADT_749_40_seismograms.zip
 
CyberShake_LADT_749_41_seismograms.zip
 
CyberShake_LADT_749_42_seismograms.zip
 
24000  02-11-2011 19:34  Seismogram_LADT_128_1100_0.grm
 
CyberShake_LADT_749_43_seismograms.zip
 
CyberShake_LADT_749_44_seismograms.zip
 
CyberShake_LADT_749_45_seismograms.zip
 
 
 
So that would tell you that file Seismogram_LADT_128_1100_0.grm is in CyberShake_LADT_749_42_seismograms.zip.
 
 
 
Once you've IDed the zip file, you can extract the seismogram you want by:
 
 
 
unzip <zip file> <seismogram file> .
 
 
 
Note that if you are retrieving seismograms from a Broadband run, the filenames are slightly different.  For low-frequency seismograms, they will be Seismogram_<site>_<sourceID>_<ruptureID>_<ruptureVariationID>_lf.grm , and the broadband seismogram filenames are Seismogram_<site>_<sourceID>_<ruptureID>_<ruptureVariationID>.grm .
 
 
b) If the seismograms are in the newer format, then there's a different approach.  You'll know because the directory will contain several thousand Seismogram_*.grm files.  The one you want is Seismogram_<site>_<sourceID>_<ruptureID>.grm, with whatever source ID and rupture ID you are interested in seismograms for.
 
 
 
All the rupture variations for a single rupture are in this file.  The format is (binary):
 
 
 
  <Rupture Variation 1 header>
 
 
  <Rupture Variation 1, X component - NT 4-byte floats>
 
  <Rupture Variation 1, X component - NT 4-byte floats>
 
  <Rupture Variation 1, Y component - NT 4-byte floats>
 
  <Rupture Variation 1, Y component - NT 4-byte floats>
  <Rupture Variation 2 header>
+
  <Rupture Variation 2 56-byte header>
 
  <Rupture Variation 2, X component - NT 4-byte floats>
 
  <Rupture Variation 2, X component - NT 4-byte floats>
 
  <Rupture Variation 2, Y component - NT 4-byte floats>
 
  <Rupture Variation 2, Y component - NT 4-byte floats>
 
  ...
 
  ...
  <Rupture Variation n header>
+
  <Rupture Variation n 56-byte header>
 
  <Rupture Variation n, X component - NT 4-byte floats>
 
  <Rupture Variation n, X component - NT 4-byte floats>
 
  <Rupture Variation n, Y component - NT 4-byte floats>
 
  <Rupture Variation n, Y component - NT 4-byte floats>
  
Where this gets tricky is that the rupture variations are not necessarily in order in the file.  So you will have to read the header data to find the rupture variations you are interested in.  A description of the header structure is given in [[CyberShake_output_data_headers#Seismogram_header]].  C and Python examples to read the header information are provided there.
+
However, the rupture variations are not necessarily in order in the file.  So if you are only interested in a subset of the rupture variations, you will have to read the header data to find the rupture variations you are interested in.  A description of the header structure is given in [[CyberShake_output_data_headers#Seismogram_header]].  C and Python examples to read the header information are provided there.
  
==== Retrieving All Seismograms for a Run ====
+
Sample Python and C code for reading out the seismogram contents for a seismogram with just 1 rupture variation is below.  This code prints the 1000th timesteps for X and Y components, but obviously you could do whatever is needed with the arrays.
  
If you wish to access all the seismograms for a run, you will need to determine the directory which contains the seismograms.  In general, results for run IDs greater than or equal to 1328 will be in /home/scec-04/tera3d/CyberShake/data/PPFiles/<site name>/<run id> .  However, you can definitively determine the directory by using the seismogram retrieval script on a single seismogram for the run.  As part of the output, the script will print the path to that seismogram.  All other seismograms for the same run id will be in the same directory.  For example:
+
=== Python code ===
  
$>/home/scec-02/cybershk/utils/retrieve_seismogram.py -r 4749 -s 7 -p 1 -v 0 -o tmp -i
+
<pre>
Opening /home/scec-04/tera3d/CyberShake/data/PPFiles/USC/4749/Seismogram_USC_7_1.grm.
+
!/usr/bin/env python3
Seismogram found.
 
  
Therefore, all the seismograms for run ID 4749 are in /home/scec-04/tera3d/CyberShake/data/PPFiles/USC/4749 .
+
import sys
 +
import struct
  
=== Reading Seismogram Files ===
+
class Seismogram:
  
Once you have your seismogram file, the last step is to read the data.
+
    def __init__(self, filename):
 +
        self.filename = filename
 +
        self.nt = 0
 +
        self.dt = 0.0
  
The seismograms are velocity seismograms, with <b>units of cm/s</b>. The number of timesteps and dt depends on what frequency the run was performed at, as well as whether the run included stochastic high-frequency results.
+
    def parseHeader(self, header_str):
 +
        #Find null terminator
 +
        split_point = header_str[0:8].find(b'\x00')
 +
        self.version = header_str[0:split_point].decode()
 +
        if self.version!="12.10":
 +
          print("Error: version does not match expected string '12.10', aborting.")
 +
          sys.exit(2)
 +
        #Find null terminator
 +
        split_point = header_str[8:16].find(b'\x00')
 +
        self.site = header_str[8:8+split_point].decode()
 +
        self.source_id = struct.unpack('i', header_str[24:28])[0]
 +
        self.rupture_id = struct.unpack('i', header_str[28:32])[0]
 +
        self.rup_var_id = struct.unpack('i', header_str[32:36])[0]
 +
        self.dt = struct.unpack('f', header_str[36:40])[0]
 +
        self.nt = struct.unpack('i', header_str[40:44])[0]
 +
        self.comps = struct.unpack('i', header_str[44:48])[0]
 +
        self.x_flag = self.y_flag = self.z_flag = False
 +
        if (self.comps & 1)==1:
 +
            self.x_flag = True
 +
        if (self.comps & 2)==2:
 +
            self.y_flag = True
 +
        if (self.comps & 4)==4:
 +
            self.z_flag = True
 +
        self.det_max_freq = struct.unpack('f', header_str[48:52])[0]
 +
        self.stoch_max_freq = struct.unpack('f', header_str[52:56])[0]
  
We changed the format of the seismogram files after Study 2.2. So for any Run ID > 1309, the seismograms are in the 'new' format, described below.
+
    def readData(self):
 +
        fp_in = open(self.filename, "rb")
 +
        header_str = fp_in.read(56)
 +
        self.parseHeader(header_str)
 +
        data_str = fp_in.read(4*self.nt)
 +
        self.x_data = struct.unpack("%df" % self.nt, data_str)
 +
        data_str = fp_in.read(4*self.nt)
 +
        self.y_data = struct.unpack("%df" % self.nt, data_str)
 +
        fp_in.close()
  
==== New (Run ID > 1309) Seismogram Format ====
+
if len(sys.argv)<2:
 +
    print("Usage: %s <seismogram file>" % sys.argv[0])
 +
    sys.exit(1)
  
If you have used the seismogram extraction script, you will have your seismogram of interest. By parsing the header (specified in [[CyberShake_output_data_headers#Seismogram_header]], examples provided), you can determine the timestep size and number of timesteps for this seismogram.
+
seis_file = sys.argv[1]
 +
s = Seismogram(seis_file)
 +
s.readData()
 +
print("seis_data[X][1000] = %f" % s.x_data[1000])
 +
print("seis_data[Y][1000] = %f" % s.y_data[1000])
 +
</pre>
  
Following the header, all seismograms have the X component, followed by the Y component.  Each component has NT 4-byte floats.  So the overall format is:
+
=== C code ===
 
 
<56-byte header>
 
<NT x 4 byte-floats - X component seismogram>
 
<NT x 4 byte-floats - Y component seismogram>
 
 
 
Sample C code for reading out the seismogram contents is below.  This code prints the 1000th timesteps for X and Y components, but obviously you could do whatever is needed with the arrays.
 
  
 
<pre>
 
<pre>
Line 185: Line 184:
 
</pre>
 
</pre>
  
==== Old (Run ID < 1310) Seismogram Format ====
+
<!-- === Plotting Seismograms ===
 
 
If the file is in the older format, the data is stored in binary format, without header information, as:
 
 
 
<NT 4-byte floats - X component seismogram>
 
<NT 4-byte floats - Y component seismogram>
 
 
 
If you run "od -f <seismogram file> | more" you can see the values.  Depending on the destination for the seismogram data, you may wish to write a small C or Python code to read the data out of the file so that you can process it.
 
 
 
=== Plotting Seismograms ===
 
  
 
If you have new-format seismogram files, you can use the script /home/scec-00/cybershk/scripts/plot_N_seis.py to plot the X and Y components of multiple seismograms on the same plot.  The script takes the seismogram files, the legends, a plot title, and an output file as arguments.  If you prefer to plot the spectral content of the seismograms, you can use the script /home/scec-00/cybershk/scripts/plot_N_seis_spectral.py .
 
If you have new-format seismogram files, you can use the script /home/scec-00/cybershk/scripts/plot_N_seis.py to plot the X and Y components of multiple seismograms on the same plot.  The script takes the seismogram files, the legends, a plot title, and an output file as arguments.  If you prefer to plot the spectral content of the seismograms, you can use the script /home/scec-00/cybershk/scripts/plot_N_seis_spectral.py .
Line 202: Line 192:
 
=== Comparing Seismograms ===
 
=== Comparing Seismograms ===
  
If you want to compare two seismograms, you can use the script /home/scec-00/cybershk/scripts/compare_seis.py, The script compares two 2-component seismograms point-by-point to see if they exceed a specified tolerance.
+
If you want to compare two seismograms, you can use the script /home/scec-00/cybershk/scripts/compare_seis.py, The script compares two 2-component seismograms point-by-point to see if they exceed a specified tolerance.//-->

Latest revision as of 18:48, 5 May 2023

This page details the process of identifying and accessing CyberShake seismograms.

Identifying Seismograms of Interest

The first step is to figure out which seismograms are of interest. You'll need to access the database to determine this; instructions are available at Accessing CyberShake Database Data. To do this, you'll need to know a) the site and run ID of the CyberShake run, and b) the source id, rupture id, and rupture variation id of the seismogram. You may have done this already. If not, you can determine the run ID by using MySQL to query the CyberShake database on focal.usc.edu. To determine a run ID, you will need to know values for a few parameters, such as the ERF, the velocity model, and the rupture variation generator version. Here's a sample query:

select R.Run_ID from CyberShake_Runs R, CyberShake_Sites S
where S.CS_Short_Name="LADT"    //Replace this with the site you want
and R.Site_ID=S.CS_Site_ID
and R.ERF_ID=35                 //35 corresponds to UCERF 2.  You'll always want to use this.
and R.SGT_Variation_ID=5        //This is the code used to generate the SGTs.  5 = Rob Graves' code.  6 = AWP-ODC.  7 = a newer, revised version of the Graves code.
and R.Rup_Var_Scenario_ID=4     //This is the code used to generated the rupture variations.  3 = Graves & Pitarka 2007.  4 = Graves & Pitarka 2010.  You'll probably want to  use 4.
and R.Velocity_Model_ID=1       //This is the velocity model.  1 = CVM-S4.  2 = CVM-H v11.2.  4 = CVM-H v11.9.
and (R.Max_Frequency=0.5 or R.Max_Frequency is null)    //This selects only deterministic (0.5 Hz) runs.  If you wanted only runs that include the stochastic high-frequency components, you would change this to "R.Max_Frequency=10.0"
and R.Status="Verified";        //This means the run completed successfully

From this you should be able to get a run ID. You could use this run ID to access all the seismograms (usually about 500,000) for the run, or you could continue to use the database to select seismograms for only certain events.

From an SRF file, you already have the source ID and run ID. The difference between rupture variations is the hypocenter locations and the slip distributions. The hypocenter locations are in the database, so you can look at them with a query like:

select Rup_Var_ID, Hypocenter_Lat, Hypocenter_Lon, Hypocenter_Depth from Rupture_Variations
where ERF_ID=35
and Source_ID=0
and Rupture_ID=0;

To understand the different slips, the easiest thing to do is to plot them.

Retrieving Seismograms

We are in the process of migrating seismograms from past CyberShake studies to Globus collections, to simplify access for users. Currently, only CyberShake Study 15.12 is available through Globus, but if you are interested in accessing seismograms from a different CyberShake study, please email software@scec.org to make a request.

Globus Account

To access the seismograms, you'll need to acquire a Globus account. If you don't already have one, you can create one using your email address by following the instructions here.

Connecting to the collection

To access the seismogram data, or intensity measure data beyond what is stored in the database, you'll need to transfer data from the Study 15.12 Globus collection to your destination system, either by using the Globus web-based interface or the Globus command-line interface.

Once you've connected to the collection, you can transfer the data anywhere with a Globus endpoint - details here. Many clusters, universities, and HPC centers have Globus endpoints set up already. If you can't find an endpoint where you need one, or you'd prefer to transfer the data to your local laptop or desktop, you can create your own endpoint using Globus Connect Personal.

Globus web-based interface

The simplest way to access CyberShake data is through the Globus web-based interface. You can access this collection by following this link.

Globus command-line interface

If you have a complex request (for example, you only want data for certain events) which is difficult to handle in the web GUI, or you would like to automate your transfers, Globus provides the Globus Command Line Interface(CLI). You can issue the same commands as in the web client, and additionally you can create a batch file which contains a list of all the files you'd like to download. With some versions of the CLI I have noticed a limit in terms of the number of files the CLI can handle at a time, so I would recommend no more than 100,000 files per batch file. If you use this approach, you'll need either the name of the collection, which is 'SCEC CyberShake Study 15.12', or the UUID for the collection, which is 2c328eb5-1c89-470d-ac1d-9fa693d28d86. Note that transfers made through the CLI will still be associated with your account, and you can monitor their progress through the web GUI.

Data organization

The data is organized by site name and run ID. So, for example, there's a directory for site ACTN and then run ID 4296. Inside this directory are all the seismograms, PSA, RotD, and duration files for site ACTN, run ID 4296. The filenames are <Seismogram | PeakVals | RotD | Duration>_<site>_<source ID>_<rupture ID>.<grm | bsa | rotd | dur> . The data for each run of Study 15.12 is approximately 90 GB.

Seismogram Format

The seismogram file Seismogram_<site>_<sourceID>_<ruptureID>.grm contains the seismograms for all the rupture variations for the given source ID.

The seismograms are velocity seismograms, with units of cm/s. The number of timesteps and dt depends on what frequency the run was performed at, as well as whether the run included stochastic high-frequency results, and can be determined by reading the header information.

The format is (binary):

<Rupture Variation 1 56-byte header>
<Rupture Variation 1, X component - NT 4-byte floats>
<Rupture Variation 1, Y component - NT 4-byte floats>
<Rupture Variation 2 56-byte header>
<Rupture Variation 2, X component - NT 4-byte floats>
<Rupture Variation 2, Y component - NT 4-byte floats>
...
<Rupture Variation n 56-byte header>
<Rupture Variation n, X component - NT 4-byte floats>
<Rupture Variation n, Y component - NT 4-byte floats>

However, the rupture variations are not necessarily in order in the file. So if you are only interested in a subset of the rupture variations, you will have to read the header data to find the rupture variations you are interested in. A description of the header structure is given in CyberShake_output_data_headers#Seismogram_header. C and Python examples to read the header information are provided there.

Sample Python and C code for reading out the seismogram contents for a seismogram with just 1 rupture variation is below. This code prints the 1000th timesteps for X and Y components, but obviously you could do whatever is needed with the arrays.

Python code

!/usr/bin/env python3

import sys
import struct

class Seismogram:

    def __init__(self, filename):
        self.filename = filename
        self.nt = 0
        self.dt = 0.0

    def parseHeader(self, header_str):
        #Find null terminator
        split_point = header_str[0:8].find(b'\x00')
        self.version = header_str[0:split_point].decode()
        if self.version!="12.10":
           print("Error: version does not match expected string '12.10', aborting.")
           sys.exit(2)
        #Find null terminator
        split_point = header_str[8:16].find(b'\x00')
        self.site = header_str[8:8+split_point].decode()
        self.source_id = struct.unpack('i', header_str[24:28])[0]
        self.rupture_id = struct.unpack('i', header_str[28:32])[0]
        self.rup_var_id = struct.unpack('i', header_str[32:36])[0]
        self.dt = struct.unpack('f', header_str[36:40])[0]
        self.nt = struct.unpack('i', header_str[40:44])[0]
        self.comps = struct.unpack('i', header_str[44:48])[0]
        self.x_flag = self.y_flag = self.z_flag = False
        if (self.comps & 1)==1:
            self.x_flag = True
        if (self.comps & 2)==2:
            self.y_flag = True
        if (self.comps & 4)==4:
            self.z_flag = True
        self.det_max_freq = struct.unpack('f', header_str[48:52])[0]
        self.stoch_max_freq = struct.unpack('f', header_str[52:56])[0]

    def readData(self):
        fp_in = open(self.filename, "rb")
        header_str = fp_in.read(56)
        self.parseHeader(header_str)
        data_str = fp_in.read(4*self.nt)
        self.x_data = struct.unpack("%df" % self.nt, data_str)
        data_str = fp_in.read(4*self.nt)
        self.y_data = struct.unpack("%df" % self.nt, data_str)
        fp_in.close()

if len(sys.argv)<2:
    print("Usage: %s <seismogram file>" % sys.argv[0])
    sys.exit(1)

seis_file = sys.argv[1]
s = Seismogram(seis_file)
s.readData()
print("seis_data[X][1000] = %f" % s.x_data[1000])
print("seis_data[Y][1000] = %f" % s.y_data[1000])

C code

#include <stdio.h>
#include <stdlib.h>
#define X 0
#define Y 1

struct seisheader {
 char version[8];
 char site_name[8];
 char padding[8];
 int source_id;
 int rupture_id;
 int rup_var_id;
 float dt;
 int nt;
 int comps;
 float det_max_freq;
 float stoch_max_freq;
};

int main(int argc, char** argv) {
        if (argc<2) {
                printf("Usage: %s <input seismogram>\n", argv[0]);
        }
        char* input_file = argv[1];
        FILE* fp_in = fopen(input_file, "rb");
        struct seisheader shead;
        fread(&shead, sizeof(struct seisheader), 1, fp_in);
        //Array with X and Y seismograms
        //seis_data[0] has X, seis_data[1] has Y
        float* seis_data[2];
        int i;
        for (i=0; i<2; i++) {
                seis_data[i] = malloc(sizeof(float)*shead.nt);
                fread(seis_data[i], sizeof(float), shead.nt, fp_in);
        }
        fclose(fp_in);
        printf("seis_data[X][1000] = %f\n", seis_data[X][1000]);
        printf("seis_data[Y][1000] = %f\n", seis_data[Y][1000]);
        for (i=0; i<2; i++) {
                free(seis_data[i]);
        }
}