UCERF3-ETAS Measurements - Revision history

Bhatthal: /* Docker */ Fix bug in Docker instructions

2026-01-05T16:45:16Z

Docker: Fix bug in Docker instructions

Bhatthal: /* Frontera */ Fix mpjrun wrapper command for Frontera

2025-03-17T16:36:37Z

Frontera: Fix mpjrun wrapper command for Frontera

Bhatthal: Remove conclusions section. We can't definitively claim which systems to use without context. Can't effectively compare system cost.

2025-01-07T17:40:27Z

Remove conclusions section. We can't definitively claim which systems to use without context. Can't effectively compare system cost.

Bhatthal: /* Performance Results */ Add Expanse run n=14, s=100k, 8GB/thread.

2024-11-14T19:36:12Z

Performance Results: Add Expanse run n=14, s=100k, 8GB/thread.

Bhatthal: /* Performance Results */ Add Fixed Cores (threads) = 350.

2024-10-21T16:10:36Z

Performance Results: Add Fixed Cores (threads) = 350.

Bhatthal: /* Performance Results */ Add Stampede3 weighted charges

2024-10-05T17:25:42Z

Performance Results: Add Stampede3 weighted charges

Bhatthal: /* Conclusions */

2024-10-03T16:44:06Z

Conclusions

Bhatthal: Create Conclusions section (under development)

2024-10-03T16:38:12Z

Create Conclusions section (under development)

Bhatthal: Remove TODO items and page incomplete notice

2024-09-26T16:38:59Z

Remove TODO items and page incomplete notice

Bhatthal: /* Performance Results */ Add ICX 100k

2024-09-26T16:37:31Z

Performance Results: Add ICX 100k

@@ Line 192: / Line 192: @@
 u3etas_comcat_event_config_builder.sh --event-id ci38457511 --mag-complete 3.5 --radius 25 --num-simulations $NUM_SIM --days-before 7 --max-point-src-mag 6 --finite-surf-shakemap --finite-surf-shakemap-min-mag 4.5 --nodes 1 --hours 24 --output-dir target/docker-comcat-ridgecrest-m7.1-n1-s$NUM_SIM
-u3etas_launcher.sh $HOME/target/docker-comcat-ridgecrest-m7.1-n${NUM_SIM}/config.json | tee target/docker-comcat-ridgecrest-m7.1-n1-s${NUM_SIM}/u3etas_launcher.log
+u3etas_launcher.sh $HOME/target/docker-comcat-ridgecrest-m7.1-n1-s${NUM_SIM}/config.json | tee target/docker-comcat-ridgecrest-m7.1-n1-s${NUM_SIM}/u3etas_launcher.log
 u3etas_plot_generator.sh $HOME/target/docker-comcat-ridgecrest-m7.1-n1-s${NUM_SIM}/config.json

@@ Line 503: / Line 503: @@
 date
 echo "RUNNING MPJ"
-mpjrun_errdetect_wrapper.sh -machinefile $PBS_NODEFILE -np $NP -dev niodev -Djava.library.path=$FMPJ_HOME/lib -Xmx${MEM_GIGS}G -cp $JAR_FILE -class scratch.UCERF3.erf.ETAS.launcher.MPJ_ETAS_Launcher --min-dispatch $MIN_DISPATCH --max-dispatch $MAX_DISPATCH --threads $THREADS $TEMP_OPTION $SCRATCH_OPTION $CLEAN_OPTION --end-time `scontrol show job $SLURM_JOB_ID | egrep --only-matching 'EndTime=[^ ]+' | cut -c 9-` $ETAS_CONF_JSON
+mpjrun_errdetect_wrapper.sh $PBS_NODEFILE -dev hybdev -Djava.library.path=$MPJ_HOME/lib -Xmx${MEM_GIGS}G -cp $JAR_FILE scratch.UCERF3.erf.ETAS.launcher.MPJ_ETAS_Launcher --min-dispatch $MIN_DISPATCH --max-dispatch $MAX_DISPATCH --threads $THREADS $TEMP_OPTION $SCRATCH_OPTION $CLEAN_OPTION --end-time `scontrol show job $SLURM_JOB_ID | egrep --only-matching 'EndTime=[^ ]+' | cut -c 9-` $ETAS_CONF_JSON
 ret=$?
 date

← Older revision		Revision as of 17:40, 7 January 2025
Line 649:		Line 649:
	exit $ret		exit $ret
	</pre>		</pre>
−
−	~~== Conclusions ==~~
−	~~'''NOTE: This section is still under development'''~~
−
−	~~After running 100k catalog simulations across various HPC systems with as comparable configurations as resources permit, I can recommend the following configurations.~~
−
−	~~# Best Value - System with greatest performance over cost: '''TODO'''~~
−	~~# Cheapest - Overall cheapest system to run in terms of service units charged: '''TODO'''~~
−	~~# Best Performance - Fastest runtime regardless of cost: '''Expanse (100k catalogs, 32 nodes, 41.8 minutes)~~

@@ Line 104: / Line 104: @@
 |-
 | 14 || 128 || 200 || 40 || 560 || 5 || Y || 100,000 || 90.5 || 2703
 |-
 | 14 || 128 || 224 || 14 || 196 || 16 || Y || 10,000 || 15.6 || 467

@@ Line 82: / Line 82: @@
 |-
 | 1 || 128 || 200 || 40 || 40 || 5 || Y || 10,000 || 67.1 || 143
 |-
 | 14 || 128 || 50 || 10 || 140 || 5 ||  || 10 || 1.8 || 54
@@ Line 140: / Line 142: @@
 |-
 | 14 || 56 || 160 || 20 || 280 || 8 || Y || 100,000 || 103.8 || 1356
 |}
@@ Line 160: / Line 164: @@
 |-
 | 14 || SPR || 112 || 104 || 13 || 182 || 8 || Y || 100,000 || 164.5 || 4299 || 38.4 || 2 || 76.8
 |}

@@ Line 147: / Line 147: @@
 |+ Stampede3 Measurements
 |-
-! Number of Nodes !! Queue !! Cores / Node !! RAM / Node (GB) !! Threads / Node !! Total Threads !! RAM / Thread !! Scratch Enabled !! Number of Catalogs !! Runtime (min) !! Core Hours
+! Number of Nodes !! Queue !! Cores / Node !! RAM / Node (GB) !! Threads / Node !! Total Threads !! RAM / Thread !! Scratch Enabled !! Number of Catalogs !! Runtime (min) !! Core Hours !! Node Hours !! Charge Rate (SU/Node Hour) !! Charge (SU)
 |-
-| 14 || ICX || 80 || 200 || 25 || 350 || 8 || Y || 10,000 || 10.6 || 198
+| 14 || ICX || 80 || 200 || 25 || 350 || 8 || Y || 10,000 || 10.6 || 198 || 2.48 || 1.5 || 3.7
 |-
-| 14 || ICX || 80 || 200 || 25 || 350 || 8 || Y || 100,000 || 83.5 || 1559
+| 14 || ICX || 80 || 200 || 25 || 350 || 8 || Y || 100,000 || 83.5 || 1559 || 19.5 || 1.5 || 29.3
 |-
-| 14 || SKX || 48 || 144 || 18 || 252 || 8 || Y || 10,000 || 13.9 || 156
+| 14 || SKX || 48 || 144 || 18 || 252 || 8 || Y || 10,000 || 13.9 || 156 || 3.25 || 1 || 3.3
 |-
-| 14 || SKX || 48 || 144 || 18 || 252 || 8 || Y || 100,000 || 111.1 || 1244
+| 14 || SKX || 48 || 144 || 18 || 252 || 8 || Y || 100,000 || 111.1 || 1244 || 25.9 || 1 || 25.9
 |-
-| 14 || SPR || 112 || 104 || 13 || 182 || 8 || Y || 10,000 || 18.9 || 494
+| 14 || SPR || 112 || 104 || 13 || 182 || 8 || Y || 10,000 || 18.9 || 494 || 4.41 || 2 || 8.8
 |-
-| 14 || SPR || 112 || 104 || 13 || 182 || 8 || Y || 100,000 || 164.5 || 4299
+| 14 || SPR || 112 || 104 || 13 || 182 || 8 || Y || 100,000 || 164.5 || 4299 || 38.4 || 2 || 76.8
 |}
 == Installation and Configuration ==

@@ Line 644: / Line 644: @@
 After running 100k catalog simulations across various HPC systems with as comparable configurations as resources permit, I can recommend the following configurations.
-# Value - System with greatest performance over cost: '''TODO'''
+# Best Value - System with greatest performance over cost: '''TODO'''
-# Lowest Cost - Overall cheapest system to run in terms of service units charged: '''TODO'''
+# Cheapest - Overall cheapest system to run in terms of service units charged: '''TODO'''
-# Greatest Performance - Fastest runtime regardless of cost: '''Expanse (100k catalogs, 32 nodes, 41.8 minutes)
+# Best Performance - Fastest runtime regardless of cost: '''Expanse (100k catalogs, 32 nodes, 41.8 minutes)

← Older revision		Revision as of 16:38, 3 October 2024
Line 638:		Line 638:
	exit $ret		exit $ret
	</pre>		</pre>
		+
		+	== Conclusions ==
		+	'''NOTE: This section is still under development'''
		+
		+	After running 100k catalog simulations across various HPC systems with as comparable configurations as resources permit, I can recommend the following configurations.
		+
		+	# Value - System with greatest performance over cost: '''TODO'''
		+	# Lowest Cost - Overall cheapest system to run in terms of service units charged: '''TODO'''
		+	# Greatest Performance - Fastest runtime regardless of cost: '''Expanse (100k catalogs, 32 nodes, 41.8 minutes)

@@ Line 1: / Line 1: @@
 This page summarizes the performance study of UCERF3-ETAS ran locally in Docker and on [https://www.carc.usc.edu/user-guides/hpc-systems/discovery CARC Discovery], [https://www.sdsc.edu/services/hpc/expanse/ SDSC Expanse], [https://tacc.utexas.edu/systems/frontera/ TACC Frontera], and [https://tacc.utexas.edu/systems/stampede3/ TACC Stampede3]. This study allows for evaluation of resource requirements in single-node and multiple-node simulations of the [https://earthquake.usgs.gov/earthquakes/eventpage/ci38457511/executive Ridgecrest M7.1 ETAS forecast (ci38457511)].
@@ Line 165: / Line 162: @@
 |}
 == Installation and Configuration ==