Publishing UCERF3-ETAS Event Reports

After successfully running a UCERF3-ETAS simulation and generating our plots, the generated UCERF3-ETAS forecast results can be published as part of a larger SCEC Event page. This page outlines the process by which results are published.

Refer to UCERF3-ETAS Measurements for detailed instructions on how to run simulations and generate plots across HPC systems. The following examples allow us to take our generated results directly from the HPC system on which the computation occurs, although results can be published from any system with an internet connection.

Creating and Updating SCEC Event Pages

SCEC Event Pages detail earthquake events. For a given mainshock, recorded magnitude, time and location, and aftershock sequences are recorded. Identify a given event from its USGS ID and use this to generate an event on the SCEC.org website at https://central.scec.org/earthquakes/eventpage/generate. (See Figure 1).

After generating an event page, it should be populated and available to view from the SCEC Event Pages list. This page is not yet published and isn't available to the public. Navigate to the "View" link to view the event page. (See Figure 2). Under the Table of Contents, you shouldn't see a "UCERF3-ETAS Forecast" yet, but after generating your results and making them available, you will be able to update this event page by selecting the "Regenerate Page with Latest Data" button. (See Figure 3)

Fig 1. SCEC Event Page Generator available at scec.org

Fig 2. SCEC Event Page for M4 earthquake near Malibu, CA

Fig 3. UCERF3-ETAS Forecast as seen on the Event Page

Git Troubleshooting

Pushing your changes upstream requires you to have an added SSH key on Frontera to allow you to make changes with your GitHub account. You must also use a GitHub account that is authorized to push changes directly to the event-reports/master branch.

You can request edit permissions to the repository owner, Akash Bhatthal <bhattha@usc.edu>

Consider the following brief instructions on getting set up with SSH.

1. Check if you already have an SSH key generated.

ls ~/.ssh/id_rsa.pub

If not, then generate one with ssh-keygen. It's possible you may have a non-RSA public key. Just check for any file ending in .pub.

2. Copy the contents of this public SSH key to your clipboard.
You can either select it directly from your Terminal, or copy the file over SSH.

3. Add as an authenticated key on your GitHub account.

Go to GitHub Settings -> SSH and GPG Keys -> New SSH key and paste the public key you copied. It should start with ssh-rsa. Be careful to not use the private key, the public key is from the file ending in .pub.

4. If you cloned the repository with HTTPS instead of SSH, then you need to update your remote accordingly.

git remote set-url origin git@github.com:(your_user_name)/event-reports.git

5. If your account has sufficient permissions, you should be able to push directly upstream to event-reports/master.

Publishing the UCERF3-ETAS Forecast

Generate your UCERF3-ETAS forecast for a given earthquake event following instructions available at UCERF3-ETAS Measurements. Our generated results are pushed to a private GitHub repository (SCECcode/event-reports) that is directly read by the SCEC Event Page. You must be granted permission to view and contribute to this repository to continue publishing UCERF3-ETAS results for use in SCEC Event Pages.

1) Clone this repository to your home directory with the following command:

cd $HOME && git clone git@github.com:SCECcode/event-reports.git

2) Get scripts

We can't directly copy results into this repository. The data is prepared using the u3etas_jar_wrapper shell script. You should already have such scripts downloaded and made available to execute by adding to your PATH. If not, clone the repository.

git clone https://github.com/opensha/ucerf3-etas-launcher.git

3) Ensure that you have selected the correct Event ID and directories. See the following section for an explanation of the parameters used.

1. Set environment variables in bash config

• Ensure you're on an interactive node:
• Update your bash config with:

module load cpu/0.15.4

module load openjdk/11.0.2

export ETAS_JAR_DISABLE_UPDATE=1

ETAS_LAUNCHER=/expanse/lustre/projects/usc143/qwxdev/apps/expanse/rocky8.8/ucerf3-etas/069e27e/ucerf3-etas-launcher

export PATH=$ETAS_LAUNCHER/sbin:$PATH

4) Connect to an interactive node srun --partition=debug --pty --account=usc143 --nodes=1 --ntasks-per-node=4 --mem=16G -t 00:30:00 --wait=0 --export=ALL /bin/bash

5) Assuming you have reserved an interactive node with sufficient memory, ensure that you have sufficient memory for execution on this node. Set ETAS_MEM_GB=32.

• export ETAS_MEM_GB=32

This doesn't have to be in your bash config, just execute directly. Adjust memory needed if you encounter an OutOfMemoryError during execution.

6) Run ComCatReportPageGen The following example has the event-reports repository cloned in our home directory on Expanse. Execution updates the local repository, after which you can git push the changes upstream given sufficient permissions. Recall that such changes don't update an already generated SCEC Event page, in which case the page will have to be regenerated.

u3etas_jar_wrapper.sh org.opensha.commons.data.comcat.plot.ComcatReportPageGen --event-id ci41075584 \

--min-mag 0d --radius 50 --output-parent-dir /home1/10177/bhatthal/event-reports \

--etas-dir $ETAS_SIM_DIR/frontera-comcat-malibu-m3.9-n14-s100000 \

--etas-output-dir /home1/10177/bhatthal/event-reports/ucerf3-etas

You will need to update the above command with the relevant absolute paths on your system. If you encounter issues with the specified outputDir, remove the conflicting "outputDir" value in your simulation's `config.json` and try again.

If you are unable to identify the event-id, try passing with the shortened parameter and ensuring these is no space, i.e. `-eci41075584`.

7) Commit the changes in `event-reports` and push upstream to origin/master. See the Git Troubleshooting section if you're unable to do this.

ComcatReportPageGen Usage

The u3etas_jar_wrapper.sh shell script is used to execute any Java application in the provided OpenSHA Jar. In this case, we're executing the ComcatReportPageGen application.

usage: ComcatReportPageGen [-?] [-d <arg>] -e <arg> [-eod <arg>] [-etas <arg>] [-m <arg>] [-o <arg>] [-opd <arg>] [-r <arg>]

-?,--help                        Display this message
-d,--days-before <arg>           Number of days of events before the mainshock to fetch (default: 3)
-e,--event-id <arg>              ComCat event id, e.g. 'ci39126079'
-eod,--etas-output-dir <arg>     If supplied, ETAS only results will also be written to <path>/<event-id>
-etas,--etas-dir <arg>           Path to a UCERF3-ETAS simulation directory
-m,--min-mag <arg>               Minimum magnitude of events to fetch (default: 0.0)
-o,--output-dir <arg>            Output dirctory. Must supply either this or --output-parent-dir
-opd,--output-parent-dir <arg>   Output parent dirctory. The directory name will be generated automatically from the
                                 event name, date, and magnitude. Must supply either this or --output-dir
-r,--radius <arg>                Search radius around mainshock for aftershocks. Default is the greater of 10.0 km and
                                 twice the Wells & Coppersmith (1994) median rupture length for the mainshock magnitude

In our example, we specify the absolute path to where we cloned the event-reports repository, as well as the absolute path to a UCERF3-ETAS results directory within the event-reports repository. Our results are written into two folders to allow us to filter by either date or Event ID.

If running on Expanse, copy any plots from scratch into your own account as we can't run in someone else's folder. You can create a tarball and copy the whole folder into your account. The same logic should apply to any other HPC system. Ensure you execute the script on an interactive node, not on the head node.

Workflow Automation and Potential Challenges

This process of getting our UCERF3-ETAS forecasts into SCEC Event Pages could be automated for jobs run on Quakeworx.

We face the following challenges in doing so:

• Not all Quakeworx users should have permission to publish results for SCEC Event Pages
• If there's an existing result, does the latest run overwrite? New UI for selecting result to use?
• We would have to start tagging commits to track overwritten events and update the web service to checkout accordingly
• SCEC Event Page regeneration would need to be triggered from Quakeworx.

Handling of permissions for a Quakeworx GitHub account, structural changes for storing and reading results from the event-reports repository, and implementing a "Write to Event Page" boolean field on job submission is all feasible. Implementing a UI to retroactively change the selected forecast and trigger page regeneration would require further investigation into the capabilities of the Quakeworx framework and if there's an API for externally triggering the generator. These changes may improve the user experience and make publishing UCERF3-ETAS forecasts easier without requiring knowledge of a Linux terminal.

Quakeworx

Results can be published even when generated on Quakeworx, not just directly via the command-line. Results are stored inside the Quakeworx development (qwxdev) scratch directory, but each user has a unique ID where their jobs are written to. Below is a list of known Drupaluid IDs for users who frequently run UCERF3-ETAS on Quakeworx.

• Phil Maechling: 6
• Fabio Silva: 7
• Akash Bhatthal: 20

For example, a job Phil ran on Quakeworx labelled "DeepSprings_EQ" would be found at /expanse/lustre/scratch/qwxdev/temp_project/qwx1/users/drupaluid_6/jobs/DeepSprings_EQ/.

In order to run the ComcatReportPageGen, you should copy this event from scratch into your own home directory. Here you can make changes to the config.json configuration as necessary and execute the generator (See step 6) with an updated etas-dir.