SCECpedia - User contributions [en]

Mendocino Event

2024-12-12T19:05:36Z

Fsilva: /* Post-earthquake Quakeworx UCERF3-ETAS Runs */

Running U3_ETAS for this event on Quakeworx.

== USGS Event ID ==
* nc75095651
*[https://earthquake.usgs.gov/earthquakes/eventpage/nc75095651/map?dyfi-responses-10km=true&shakemap-intensity=false Comcat Shakemap for Event]

== Checking for Quakeworx jobs on Expanse ==
*squeue -u qwxdev

<pre>
(base) squeue -u qwxdev
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
35601027 compute tapisjob qwxdev PD 0:00 14 (Priority)
</pre>

== Post-earthquake Quakeworx UCERF3-ETAS Runs ==

{| class="wikitable"
! Submit time (PT)
! Operator
! Complete Time
! Compute Time (HM:MM:SS)
! Makespan (HM:MM:SS)
! System
! RunType
! Queue
! DataSize (GB)
! Core Hours Charged
! Result
! Config
|-
| 11:05:00
| Fabio
| 14:32:00
| 01:11:00
| 03:27:00
| Expanse
| QW
| compute
| 18
| 2181
| [https://g-c662a6.a78b8.36fe.data.globus.org/ucerf3-etas/quakeworx/ferndale/index.html M7.0 Mendocino UCERF3-ETAS Results]
| Default
|-
| 19:05:00
| Phil
| 21:52:00
| 01:29:00
| 02:47:00
| Expanse
| QW
| compute
| 24.1
| 2312
| [https://g-c662a6.a78b8.36fe.data.globus.org/ucerf3-etas/quakeworx/outputs/index.html M7.0 Mendocino UCERF3-ETAS Results]
| Default
|}

== Output directorie amd Globus Transfer ==
An example path on expanse to Quakeworx results are below. When using the Globus connections on Expanse lustre drop the first parts of this path.
*/expanse/lustre/scratch/qwxdev/temp_project/qwx1/users/drupaluid_7/jobs/UCERF3-ETAS-Ferndale
*/expanse/lustre/scratch/qwxdev/temp_project/qwx1/users/drupaluid_6/jobs/Mendocino_ETAS/outputs/tapisjob.out

== Workshop Scaling Tests UCERF3-ETAS Runs ==

{| class="wikitable"
! Submit time (PT)
! Operator
! Complete Time
! Compute Time (HM:MM:SS)
! Makespan (HM:MM:SS)
! System
! RunType
! Number of Catalogs:
! Nodes:
! Tasks Total:
! Queue
! DataSize (GB)
! Core Hours Charged
! Result
! Config
|-
| 00:00:00
| Scott
| 00:45:00
| 00:15:00
| 00:45:00
| Expanse
| QW
| 1000
| 1
| 32
| compute
| XX.XX
| 32
| index.html
| Default
|}

== Link to Results on CARC ==
*[https://g-c662a6.a78b8.36fe.data.globus.org/ucerf3-etas/quakeworx/outputs/index.html M7.0 Mendocino UCERF3-ETAS Results]

== Preparing U3-ETAS for Next Post-Earthquake Simulations ==
#Manage Accounts:
##Quakeworx Accounts:
##Frontera Accounts:
##Expanse Accounts:
##Discovery Accounts:
#Manage Allocations:
##Quakeworx
##Frontera
##Expanse
##Discovery
#Evidence that U3-ETAS is correct:
##Quakeworx is correct:
##Command Line is correct:
#Keeping the Software Current:
##Quakeworx Current
##Keeping Command Line Current:
#Information to help Users specific Parameters:
#Ability to run with/without planner
#Ability to run with/without plotter
#Transfer results to end-users:
#Access to partial results

== Related Entries ==
*[[Event Response]]

CSEP - ETAS Simulation Plan

2024-08-28T17:34:52Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 134
* Completed percent: 100.0%
* Updated: 27-August-2024 5:00PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:lime" | 010
| style="background-color:lime" | 011
| style="background-color:lime" | 012
| style="background-color:lime" | 013
| style="background-color:lime" | 014
| style="background-color:lime" | 015
| style="background-color:lime" | 016
|-
! 12/2008-12/2009
| style="background-color:lime" | 017
| style="background-color:lime" | 018
| style="background-color:lime" | 019
| style="background-color:lime" | 020
| style="background-color:lime" | 021
| style="background-color:lime" | 022
| style="background-color:lime" | 023
| style="background-color:lime" | 024
| style="background-color:lime" | 025
| style="background-color:lime" | 026
| style="background-color:lime" | 027
| style="background-color:lime" | 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:lime" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-27T17:09:48Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 133
* Completed percent: 99.3%
* Updated: 27-August-2024 10:00AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:lime" | 010
| style="background-color:lime" | 011
| style="background-color:lime" | 012
| style="background-color:lime" | 013
| style="background-color:lime" | 014
| style="background-color:lime" | 015
| style="background-color:lime" | 016
|-
! 12/2008-12/2009
| style="background-color:lime" | 017
| style="background-color:lime" | 018
| style="background-color:lime" | 019
| style="background-color:lime" | 020
| style="background-color:lime" | 021
| style="background-color:lime" | 022
| style="background-color:lime" | 023
| style="background-color:lime" | 024
| style="background-color:lime" | 025
| style="background-color:lime" | 026
| style="background-color:lime" | 027
| style="background-color:lime" | 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:yellow" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-27T02:25:51Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 125
* Completed percent: 93.3%
* Updated: 26-August-2024 7:25PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:lime" | 010
| style="background-color:lime" | 011
| style="background-color:lime" | 012
| style="background-color:lime" | 013
| style="background-color:lime" | 014
| style="background-color:lime" | 015
| style="background-color:lime" | 016
|-
! 12/2008-12/2009
| style="background-color:lime" | 017
| style="background-color:lime" | 018
| style="background-color:lime" | 019
| style="background-color:lime" | 020
| style="background-color:lime" | 021
| style="background-color:aqua" | 022
| style="background-color:aqua" | 023
| style="background-color:aqua" | 024
| style="background-color:aqua" | 025
| style="background-color:aqua" | 026
| style="background-color:aqua" | 027
| style="background-color:aqua" | 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:yellow" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-27T02:24:47Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 124
* Completed percent: 92.5%
* Updated: 26-August-2024 10:10AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:lime" | 010
| style="background-color:lime" | 011
| style="background-color:lime" | 012
| style="background-color:lime" | 013
| style="background-color:lime" | 014
| style="background-color:lime" | 015
| style="background-color:lime" | 016
|-
! 12/2008-12/2009
| style="background-color:lime" | 017
| style="background-color:lime" | 018
| style="background-color:lime" | 019
| style="background-color:lime" | 020
| style="background-color:aqua" | 021
| style="background-color:aqua" | 022
| style="background-color:aqua" | 023
| style="background-color:aqua" | 024
| style="background-color:aqua" | 025
| style="background-color:aqua" | 026
| style="background-color:aqua" | 027
| style="background-color:aqua" | 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:yellow" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-26T17:10:23Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 124
* Completed percent: 92.5%
* Updated: 26-August-2024 10:10AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:lime" | 010
| style="background-color:lime" | 011
| style="background-color:lime" | 012
| style="background-color:lime" | 013
| style="background-color:lime" | 014
| style="background-color:lime" | 015
| style="background-color:lime" | 016
|-
! 12/2008-12/2009
| style="background-color:lime" | 017
| style="background-color:lime" | 018
| style="background-color:lime" | 019
| style="background-color:lime" | 020
| style="background-color:aqua" | 021
| style="background-color:aqua" | 022
| style="background-color:aqua" | 023
| style="background-color:aqua" | 024
| style="background-color:aqua" | 025
| style="background-color:aqua" | 026
| style="background-color:aqua" | 027
| style="background-color:aqua" | 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-26T17:10:02Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 114
* Completed percent: 85.1%
* Updated: 26-August-2024 10:10AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:lime" | 010
| style="background-color:lime" | 011
| style="background-color:lime" | 012
| style="background-color:lime" | 013
| style="background-color:lime" | 014
| style="background-color:lime" | 015
| style="background-color:lime" | 016
|-
! 12/2008-12/2009
| style="background-color:lime" | 017
| style="background-color:lime" | 018
| style="background-color:lime" | 019
| style="background-color:lime" | 020
| style="background-color:aqua" | 021
| style="background-color:aqua" | 022
| style="background-color:aqua" | 023
| style="background-color:aqua" | 024
| style="background-color:aqua" | 025
| style="background-color:aqua" | 026
| style="background-color:aqua" | 027
| style="background-color:aqua" | 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-26T17:07:54Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 114
* Completed percent: 85.1%
* Updated: 25-August-2024 9:15AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:lime" | 010
| style="background-color:lime" | 011
| style="background-color:lime" | 012
| style="background-color:lime" | 013
| style="background-color:lime" | 014
| style="background-color:lime" | 015
| style="background-color:lime" | 016
|-
! 12/2008-12/2009
| style="background-color:lime" | 017
| style="background-color:lime" | 018
| style="background-color:lime" | 019
| style="background-color:lime" | 020
| style="background-color:aqua" | 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-25T16:13:43Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 114
* Completed percent: 85.1%
* Updated: 25-August-2024 9:15AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:aqua" | 010
| style="background-color:aqua" | 011
| style="background-color:aqua" | 012
| style="background-color:aqua" | 013
| style="background-color:aqua" | 014
| style="background-color:aqua" | 015
| style="background-color:aqua" | 016
|-
! 12/2008-12/2009
| style="background-color:aqua" | 017
| style="background-color:aqua" | 018
| style="background-color:yellow" | 019
| style="background-color:yellow" | 020
| style="background-color:yellow" | 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-25T16:12:27Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 105
* Completed percent: 78.4%
* Updated: 25-August-2024 9:15AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:lime" | 001
| style="background-color:lime" | 002
| style="background-color:lime" | 003
| style="background-color:lime" | 004
|-
! 12/2007-12/2008
| style="background-color:lime" | 005
| style="background-color:lime" | 006
| style="background-color:lime" | 007
| style="background-color:lime" | 008
| style="background-color:lime" | 009
| style="background-color:aqua" | 010
| style="background-color:aqua" | 011
| style="background-color:aqua" | 012
| style="background-color:aqua" | 013
| style="background-color:aqua" | 014
| style="background-color:aqua" | 015
| style="background-color:aqua" | 016
|-
! 12/2008-12/2009
| style="background-color:aqua" | 017
| style="background-color:aqua" | 018
| style="background-color:yellow" | 019
| style="background-color:yellow" | 020
| style="background-color:yellow" | 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-24T18:37:40Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 105
* Completed percent: 78.4%
* Updated: 24-August-2024 11:45AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:aqua" | 001
| style="background-color:aqua" | 002
| style="background-color:aqua" | 003
| style="background-color:aqua" | 004
|-
! 12/2007-12/2008
| style="background-color:aqua" | 005
| style="background-color:aqua" | 006
| style="background-color:aqua" | 007
| style="background-color:aqua" | 008
| style="background-color:aqua" | 009
| style="background-color:yellow" | 010
| style="background-color:yellow" | 011
| style="background-color:yellow" | 012
| style="background-color:yellow" | 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-24T18:36:41Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 93
* Completed percent: 69.4%
* Updated: 23-August-2024 10:00AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| style="background-color:aqua" | 001
| style="background-color:aqua" | 002
| style="background-color:aqua" | 003
| style="background-color:aqua" | 004
|-
! 12/2007-12/2008
| style="background-color:aqua" | 005
| style="background-color:aqua" | 006
| style="background-color:aqua" | 007
| style="background-color:aqua" | 008
| style="background-color:aqua" | 009
| style="background-color:yellow" | 010
| style="background-color:yellow" | 011
| style="background-color:yellow" | 012
| style="background-color:yellow" | 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-24T18:33:02Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 93
* Completed percent: 69.4%
* Updated: 23-August-2024 10:00AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:lime" | 121
| style="background-color:lime" | 122
| style="background-color:red" | 123
| style="background-color:lime" | 124
|-
! 12/2017-12/2018
| style="background-color:lime" | 125
| style="background-color:lime" | 126
| style="background-color:lime" | 127
| style="background-color:lime" | 128
| style="background-color:lime" | 129
| style="background-color:lime" | 130
| style="background-color:lime" | 131
| style="background-color:lime" | 132
| style="background-color:lime" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-23T17:08:48Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 93
* Completed percent: 69.4%
* Updated: 23-August-2024 10:00AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:aqua" | 121
| style="background-color:aqua" | 122
| style="background-color:red" | 123
| style="background-color:aqua" | 124
|-
! 12/2017-12/2018
| style="background-color:aqua" | 125
| style="background-color:aqua" | 126
| style="background-color:aqua" | 127
| style="background-color:aqua" | 128
| style="background-color:aqua" | 129
| style="background-color:yellow" | 130
| style="background-color:yellow" | 131
| style="background-color:yellow" | 132
| style="background-color:yellow" | 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-23T06:35:10Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 93
* Completed percent: 69.4%
* Updated: 22-August-2024 10:20PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:aqua" | 121
| style="background-color:aqua" | 122
| style="background-color:aqua" | 123
| style="background-color:yellow" | 124
|-
! 12/2017-12/2018
| style="background-color:yellow" | 125
| style="background-color:yellow" | 126
| style="background-color:yellow" | 127
| style="background-color:yellow" | 128
| style="background-color:yellow" | 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-23T06:33:31Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 93
* Completed percent: 69.4%
* Updated: 22-August-2024 10:20PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:aqua" | 121
| style="background-color:yellow" | 122
| style="background-color:yellow" | 123
| style="background-color:yellow" | 124
|-
! 12/2017-12/2018
| style="background-color:yellow" | 125
| style="background-color:yellow" | 126
| style="background-color:yellow" | 127
| style="background-color:yellow" | 128
| style="background-color:yellow" | 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-23T05:30:33Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 93
* Completed percent: 69.4%
* Updated: 22-August-2024 10:20PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:lime" | 120
| style="background-color:aqua" | 121
| style="background-color:yellow" | 122
| style="background-color:yellow" | 123
| style="background-color:yellow" | 124
|-
! 12/2017-12/2018
| style="background-color:yellow" | 125
| style="background-color:yellow" | 126
| style="background-color:yellow" | 127
| style="background-color:yellow" | 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-23T01:40:23Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 92
* Completed percent: 68.7%
* Updated: 22-August-2024 6:45PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:lime" | 117
| style="background-color:lime" | 118
| style="background-color:lime" | 119
| style="background-color:aqua" | 120
| style="background-color:yellow" | 121
| style="background-color:yellow" | 122
| style="background-color:yellow" | 123
| style="background-color:yellow" | 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-22T20:16:45Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 89
* Completed percent: 66.4%
* Updated: 22-August-2024 1:15PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:aqua" | 117
| style="background-color:aqua" | 118
| style="background-color:aqua" | 119
| style="background-color:aqua" | 120
| style="background-color:yellow" | 121
| style="background-color:yellow" | 122
| style="background-color:yellow" | 123
| style="background-color:yellow" | 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-22T20:16:09Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 80
* Completed percent: 59.7%
* Updated: 22-August-2024 1:15PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:lime" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:lime" | 109
| style="background-color:lime" | 110
| style="background-color:lime" | 111
| style="background-color:lime" | 112
|-
! 12/2016-12/2017
| style="background-color:lime" | 113
| style="background-color:lime" | 114
| style="background-color:lime" | 115
| style="background-color:lime" | 116
| style="background-color:aqua" | 117
| style="background-color:aqua" | 118
| style="background-color:aqua" | 119
| style="background-color:aqua" | 120
| style="background-color:yellow" | 121
| style="background-color:yellow" | 122
| style="background-color:yellow" | 123
| style="background-color:yellow" | 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-21T21:12:31Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 80
* Completed percent: 59.7%
* Updated: 21-August-2024 2:15PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:yellow" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:aqua" | 109
| style="background-color:aqua" | 110
| style="background-color:aqua" | 111
| style="background-color:aqua" | 112
|-
! 12/2016-12/2017
| style="background-color:aqua" | 113
| style="background-color:aqua" | 114
| style="background-color:aqua" | 115
| style="background-color:aqua" | 116
| style="background-color:yellow" | 117
| style="background-color:yellow" | 118
| style="background-color:yellow" | 119
| style="background-color:yellow" | 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-21T21:10:50Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 80
* Completed percent: 59.7%
* Updated: 20-August-2024 10:00PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:yellow" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:aqua" | 109
| style="background-color:aqua" | 110
| style="background-color:aqua" | 111
| style="background-color:aqua" | 112
|-
! 12/2016-12/2017
| style="background-color:aqua" | 113
| style="background-color:aqua" | 114
| style="background-color:aqua" | 115
| style="background-color:aqua" | 116
| style="background-color:yellow" | 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-21T15:51:24Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 80
* Completed percent: 59.7%
* Updated: 20-August-2024 10:00PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:aqua" | 109
| style="background-color:aqua" | 110
| style="background-color:aqua" | 111
| style="background-color:aqua" | 112
|-
! 12/2016-12/2017
| style="background-color:aqua" | 113
| style="background-color:yellow" | 114
| style="background-color:yellow" | 115
| style="background-color:yellow" | 116
| style="background-color:yellow" | 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-21T05:19:40Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 80
* Completed percent: 59.7%
* Updated: 20-August-2024 10:00PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:lime" | 105
| style="background-color:lime" | 106
| style="background-color:lime" | 107
| style="background-color:lime" | 108
| style="background-color:yellow" | 109
| style="background-color:yellow" | 110
| style="background-color:yellow" | 111
| style="background-color:yellow" | 112
|-
! 12/2016-12/2017
| style="background-color:yellow" | 113
| style="background-color:yellow" | 114
| style="background-color:yellow" | 115
| style="background-color:yellow" | 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-20T16:39:04Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 76
* Completed percent: 56.7%
* Updated: 20-August-2024 9:30AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:aqua" | 105
| style="background-color:aqua" | 106
| style="background-color:aqua" | 107
| style="background-color:aqua" | 108
| style="background-color:yellow" | 109
| style="background-color:yellow" | 110
| style="background-color:yellow" | 111
| style="background-color:yellow" | 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-20T16:38:27Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 70
* Completed percent: 52.2%
* Updated: 19-August-2024 9:30AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:lime" | 099
| style="background-color:lime" | 100
|-
! 12/2015-12/2016
| style="background-color:lime" | 101
| style="background-color:lime" | 102
| style="background-color:lime" | 103
| style="background-color:lime" | 104
| style="background-color:aqua" | 105
| style="background-color:aqua" | 106
| style="background-color:aqua" | 107
| style="background-color:aqua" | 108
| style="background-color:yellow" | 109
| style="background-color:yellow" | 110
| style="background-color:yellow" | 111
| style="background-color:yellow" | 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-19T18:04:53Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 70
* Completed percent: 52.2%
* Updated: 19-August-2024 9:30AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:aqua" | 099
| style="background-color:aqua" | 100
|-
! 12/2015-12/2016
| style="background-color:aqua" | 101
| style="background-color:aqua" | 102
| style="background-color:aqua" | 103
| style="background-color:aqua" | 104
| style="background-color:yellow" | 105
| style="background-color:yellow" | 106
| style="background-color:yellow" | 107
| style="background-color:yellow" | 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-19T16:28:57Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 70
* Completed percent: 52.2%
* Updated: 19-August-2024 9:30AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:aqua" | 099
| style="background-color:aqua" | 100
|-
! 12/2015-12/2016
| style="background-color:aqua" | 101
| style="background-color:aqua" | 102
| style="background-color:yellow" | 103
| style="background-color:yellow" | 104
| style="background-color:yellow" | 105
| style="background-color:yellow" | 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-19T16:27:49Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 61
* Completed percent: 45.5%
* Updated: 18-August-2024 12:15PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:lime" | 090
| style="background-color:lime" | 091
| style="background-color:lime" | 092
| style="background-color:lime" | 093
| style="background-color:lime" | 094
| style="background-color:lime" | 095
| style="background-color:lime" | 096
| style="background-color:lime" | 097
| style="background-color:lime" | 098
| style="background-color:aqua" | 099
| style="background-color:aqua" | 100
|-
! 12/2015-12/2016
| style="background-color:aqua" | 101
| style="background-color:aqua" | 102
| style="background-color:yellow" | 103
| style="background-color:yellow" | 104
| style="background-color:yellow" | 105
| style="background-color:yellow" | 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-18T19:19:59Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 61
* Completed percent: 45.5%
* Updated: 18-August-2024 12:15PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:aqua" | 090
| style="background-color:aqua" | 091
| style="background-color:aqua" | 092
| style="background-color:aqua" | 093
| style="background-color:aqua" | 094
| style="background-color:aqua" | 095
| style="background-color:aqua" | 096
| style="background-color:aqua" | 097
| style="background-color:aqua" | 098
| style="background-color:yellow" | 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-18T19:19:08Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 55
* Completed percent: 41.0%
* Updated: 17-August-2024 11:50AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:lime" | 084
| style="background-color:lime" | 085
| style="background-color:lime" | 086
| style="background-color:lime" | 087
| style="background-color:lime" | 088
|-
! 12/2014-12/2015
| style="background-color:lime" | 089
| style="background-color:aqua" | 090
| style="background-color:aqua" | 091
| style="background-color:aqua" | 092
| style="background-color:aqua" | 093
| style="background-color:aqua" | 094
| style="background-color:aqua" | 095
| style="background-color:aqua" | 096
| style="background-color:aqua" | 097
| style="background-color:aqua" | 098
| style="background-color:yellow" | 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-17T18:55:17Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 55
* Completed percent: 41.0%
* Updated: 17-August-2024 11:50AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:aqua" | 084
| style="background-color:aqua" | 085
| style="background-color:yellow" | 086
| style="background-color:yellow" | 087
| style="background-color:yellow" | 088
|-
! 12/2014-12/2015
| style="background-color:yellow" | 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-17T18:54:50Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 49
* Completed percent: 36.6%
* Updated: 17-August-2024 11:50AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:lime" | 078
| style="background-color:lime" | 079
| style="background-color:lime" | 080
| style="background-color:lime" | 081
| style="background-color:lime" | 082
| style="background-color:lime" | 083
| style="background-color:aqua" | 084
| style="background-color:aqua" | 085
| style="background-color:yellow" | 086
| style="background-color:yellow" | 087
| style="background-color:yellow" | 088
|-
! 12/2014-12/2015
| style="background-color:yellow" | 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-16T17:00:50Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 49
* Completed percent: 36.6%
* Updated: 16-August-2024 10:00AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:aqua" | 078
| style="background-color:aqua" | 079
| style="background-color:aqua" | 080
| style="background-color:aqua" | 081
| style="background-color:yellow" | 082
| style="background-color:yellow" | 083
| style="background-color:yellow" | 084
| style="background-color:yellow" | 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-16T17:00:01Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 48
* Completed percent: 35.9%
* Updated: 15-August-2024 12:45PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:lime" | 077
| style="background-color:aqua" | 078
| style="background-color:aqua" | 079
| style="background-color:aqua" | 080
| style="background-color:aqua" | 081
| style="background-color:yellow" | 082
| style="background-color:yellow" | 083
| style="background-color:yellow" | 084
| style="background-color:yellow" | 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-15T19:48:06Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 48
* Completed percent: 35.9%
* Updated: 15-August-2024 12:45PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:aqua" | 077
| style="background-color:yellow" | 078
| style="background-color:yellow" | 079
| style="background-color:yellow" | 080
| style="background-color:yellow" | 081
| 082
| 083
| 084
| 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-15T19:47:26Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 42
* Completed percent: 31.3%
* Updated: 14-August-2024 5:45PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:lime" | 071
| style="background-color:lime" | 072
| style="background-color:lime" | 073
| style="background-color:lime" | 074
| style="background-color:lime" | 075
| style="background-color:lime" | 076
|-
! 12/2013-12/2014
| style="background-color:aqua" | 077
| style="background-color:yellow" | 078
| style="background-color:yellow" | 079
| style="background-color:yellow" | 080
| style="background-color:yellow" | 081
| 082
| 083
| 084
| 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-15T00:45:40Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 42
* Completed percent: 31.3%
* Updated: 14-August-2024 5:45PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:aqua" | 071
| style="background-color:aqua" | 072
| style="background-color:aqua" | 073
| style="background-color:yellow" | 074
| style="background-color:yellow" | 075
| style="background-color:yellow" | 076
|-
! 12/2013-12/2014
| style="background-color:yellow" | 077
| 078
| 079
| 080
| 081
| 082
| 083
| 084
| 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-15T00:44:46Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 42
* Completed percent: 31.3%
* Updated: 14-August-2024 5:45PM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:lime" | 070
| style="background-color:aqua" | 071
| style="background-color:aqua" | 072
| style="background-color:yellow" | 073
| style="background-color:yellow" | 074
| style="background-color:yellow" | 075
| style="background-color:yellow" | 076
|-
! 12/2013-12/2014
| style="background-color:yellow" | 077
| 078
| 079
| 080
| 081
| 082
| 083
| 084
| 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-14T17:52:13Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 41
* Completed percent: 30.6%
* Updated: 14-August-2024 10:50AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:aqua" | 070
| style="background-color:aqua" | 071
| style="background-color:aqua" | 072
| style="background-color:yellow" | 073
| style="background-color:yellow" | 074
| style="background-color:yellow" | 075
| style="background-color:yellow" | 076
|-
! 12/2013-12/2014
| 077
| 078
| 079
| 080
| 081
| 082
| 083
| 084
| 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-14T17:51:51Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 41
* Completed percent: 30.6%
* Updated: 14-August-2024 10:00AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:lime" | 069
| style="background-color:aqua" | 070
| style="background-color:aqua" | 071
| style="background-color:aqua" | 072
| style="background-color:yellow" | 073
| style="background-color:yellow" | 074
| style="background-color:yellow" | 075
| style="background-color:yellow" | 076
|-
! 12/2013-12/2014
| 077
| 078
| 079
| 080
| 081
| 082
| 083
| 084
| 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-14T17:09:21Z

Fsilva: /* ETAS Production Run Status */

This page summarizes the performance study ran on Stampede 3 in order to calculate the simulation requirements for computing 1 day ETAS forecasts from 2007 to 2018.

== Stampede 3 Installation and Configuration ==

* Java - jdk-21.0.1+12
* FastMPJ

The Slurm file header used to run the simulations is shown below:

<pre>
#SBATCH -t 6:00:00
#SBATCH -N 7
#SBATCH -n 336
#SBATCH -p spr
#SBATCH -A DS-Cybershake
</pre>

where the "-n" parameter is calculated as 48 * number of nodes (N). Additionally, the following parameters were used in the Slurm script

<pre>
MEM_GIGS=110
THREADS=20
FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
#CLEAN_OPTION="--clean"
export FMPJ_HOME=/work2/02404/fsilva/stampede3/FastMPJ
</pre>

== Performance Results ==

To measure the Stampede 3 performance running the one day ETAS forecasts, we ran the same simulation scenario using different numbers of nodes. To compute these results, we used the same random seed and date. We also deleted the cached results stored in the scratch filesystem after each simulation in order to force the recalculation of the entire run. The command-line used to generate the run was:

<pre>
u3etas_comcat_config_builder.sh --end-time 1717484400000 --num-simulations 100000 --duration-years 0.002737851 --include-spontaneous --historical-catalog --start-after-historical --etas-k-cov 1.5 --random-seed 123456789 --hpc-site TACC_FRONTERA --nodes 35 --hours 24 --queue normal --output-dir $ETAS_SIM_DIR/2024_06_04-ComcatPlusHistorical-Start20240604_1day_100000Simulations_Statewide_PointSources_kCOV1p5_Spontaneous_HistCatalog --binary-output
</pre>

The results are as follows:

{| border="1" cellpadding="5"
|-
! Number of Nodes
! Runtime (min)
! Service Units (SUs) Used
|-
! 7
! 169
! 19.7
|-
! 14
! 84
! 19.6
|-
! 28
! 43
! 20.1
|-
! 56
! 24
! 22.4
|}

In the table above, service units was computed by dividing the runtime by 60 and multiplying by the number of nodes used. We observe that as the numbers of nodes used increased, runtime decreased linearly while the allocation usage remained mostly flat.

== Requirements for the Complete ETAS Simulation Runs ==

Using the performance results obtained above, we can calculate the requirements for the full simulation run by multiplying the required SUs for a single run by the total number of runs:

* Start Date = 1 August 2007
* End Date = 30 August 2018
* Total Number of Days = 4045

For the total required service units, we multiple the total number of 1 day forecasts (4045) by the number of service units used for each run (20):

* SUs needed = 20 * 4045 = 80900 SUs

The storage requirements per 1 day forecast is as follows:

* Binary results (results_*.bin files) ~ 100M
* Complete output folder (with logs) ~ 272M

If we multiply the numbers above by the total number of 1-day forecasts, we have

* Total storage for data = 100M * 4045 = 405G
* Total storage (including logs) = 272M * 4045 = 1.1T

== ETAS Production Run Setup ==

For the UCERF3-ETAS production run setup on Stampede 3, we split the simulation period into 134 30-day bundles using UTC times. We also used the following parameters for each bundle:

* Number of nodes: 40
* Node type: Skylake
* Wall time: 24 hours
* Memory: 110 GB
* Threads: 20
* Options: TEMP_OPTION, SCRATCH_OPTION

Slurm header:

<pre>
#SBATCH -t 24:00:00
#SBATCH -N 40
#SBATCH -n 1920
#SBATCH -p skx
</pre>

== ETAS Production Run Status ==

* Total batches: 134
* Completed batches: 40
* Completed percent: 29.9%
* Updated: 14-August-2024 10:00AM PDT

The table below shows the status for each individual batch:

{| class="wikitable" style="text-align: center; border="1"
|-
! Period
!
!
!
!
!
!
!
!
!
!
!
!
|-
! 12/2006-12/2007
|
|
|
|
|
|
|
| style="background-color:lime" | 000
| 001
| 002
| 003
| 004
|-
! 12/2007-12/2008
| 005
| 006
| 007
| 008
| 009
| 010
| 011
| 012
| 013
| 014
| 015
| 016
|-
! 12/2008-12/2009
| 017
| 018
| 019
| 020
| 021
| 022
| 023
| 024
| 025
| 026
| 027
| 028
|-
! 12/2009-12/2010
| style="background-color:lime" | 029
| style="background-color:lime" | 030
| style="background-color:lime" | 031
| style="background-color:lime" | 032
| style="background-color:lime" | 033
| style="background-color:lime" | 034
| style="background-color:lime" | 035
| style="background-color:lime" | 036
| style="background-color:lime" | 037
| style="background-color:lime" | 038
| style="background-color:lime" | 039
| style="background-color:lime" | 040
|-
! 12/2010-12/2011
| style="background-color:lime" | 041
| style="background-color:lime" | 042
| style="background-color:lime" | 043
| style="background-color:lime" | 044
| style="background-color:lime" | 045
| style="background-color:lime" | 046
| style="background-color:lime" | 047
| style="background-color:lime" | 048
| style="background-color:lime" | 049
| style="background-color:lime" | 050
| style="background-color:lime" | 051
| style="background-color:lime" | 052
|-
! 12/2011-12/2012
| style="background-color:lime" | 053
| style="background-color:lime" | 054
| style="background-color:lime" | 055
| style="background-color:lime" | 056
| style="background-color:lime" | 057
| style="background-color:lime" | 058
| style="background-color:lime" | 059
| style="background-color:lime" | 060
| style="background-color:lime" | 061
| style="background-color:lime" | 062
| style="background-color:red" | 063
| style="background-color:lime" | 064
|-
! 12/2012-12/2013
| style="background-color:lime" | 065
| style="background-color:lime" | 066
| style="background-color:lime" | 067
| style="background-color:lime" | 068
| style="background-color:aqua" | 069
| style="background-color:aqua" | 070
| style="background-color:aqua" | 071
| style="background-color:aqua" | 072
| style="background-color:yellow" | 073
| style="background-color:yellow" | 074
| style="background-color:yellow" | 075
| style="background-color:yellow" | 076
|-
! 12/2013-12/2014
| 077
| 078
| 079
| 080
| 081
| 082
| 083
| 084
| 085
| 086
| 087
| 088
|-
! 12/2014-12/2015
| 089
| 090
| 091
| 092
| 093
| 094
| 095
| 096
| 097
| 098
| 099
| 100
|-
! 12/2015-12/2016
| 101
| 102
| 103
| 104
| 105
| 106
| 107
| 108
| 109
| 110
| 111
| 112
|-
! 12/2016-12/2017
| 113
| 114
| 115
| 116
| 117
| 118
| 119
| 120
| 121
| 122
| 123
| 124
|-
! 12/2017-12/2018
| 125
| 126
| 127
| 128
| 129
| 130
| 131
| 132
| 133
|
|
|
|}

 Completed 

 Running 

 Queued 

 Failed

CSEP - ETAS Simulation Plan

2024-08-14T00:46:34Z

Fsilva: /* ETAS Production Run Status */

CSEP - ETAS Simulation Plan

2024-08-13T18:02:44Z

Fsilva: /* ETAS Production Run Status */

CSEP - ETAS Simulation Plan

2024-08-13T18:01:29Z

Fsilva: /* ETAS Production Run Status */

CSEP - ETAS Simulation Plan

2024-08-13T17:16:35Z

Fsilva: /* ETAS Production Run Status */

CSEP - ETAS Simulation Plan

2024-08-13T15:13:02Z

Fsilva: /* ETAS Production Run Status */

CSEP - ETAS Simulation Plan

2024-08-12T17:08:30Z

Fsilva: /* ETAS Production Run Status */

CSEP - ETAS Simulation Plan

2024-08-12T16:38:16Z

Fsilva: /* ETAS Production Run Status */

CSEP - ETAS Simulation Plan

2024-08-12T16:15:36Z

Fsilva: /* ETAS Production Run Status */