DirectSynth - Revision history

Scottcal: /* Communication */

2015-09-10T17:57:42Z

Communication

Scottcal at 15:59, 10 September 2015

2015-09-10T15:59:28Z

Scottcal: /* Communication */

2015-04-21T19:18:46Z

Communication

Scottcal at 19:18, 21 April 2015

2015-04-21T19:18:03Z

Scottcal at 18:54, 21 April 2015

2015-04-21T18:54:57Z

Scottcal at 18:54, 21 April 2015

2015-04-21T18:54:35Z

Scottcal: /* Communication patterns */

2015-04-21T18:09:46Z

Communication patterns

Scottcal at 17:53, 21 April 2015

2015-04-21T17:53:57Z

Scottcal at 16:07, 21 April 2015

2015-04-21T16:07:18Z

Scottcal: /* Details */

2015-04-21T15:48:46Z

Details

@@ Line 67: / Line 67: @@
 * Broadcast of SGT points to SGT handler mapping, from master: 4 KB
 * Sending of SGT point headers, from master to individual SGT handlers: 911 KB/SGT handler x 1151 SGT handlers = 1 GB
-* Each worker processes 16 tasks, which requires data from the SGT handler to the worker:  16 tasks * 38220 SGT points/task * 188 KB/SGT point = 110 GB/worker x 2559 workers = <b>274 TB</b>.
+* Each worker processes 17.1 tasks, which requires data from the SGT handler to the worker:  17.1 tasks * 137220 SGT points/task * 188 KB/SGT point = 420 GB/worker x 2559 workers = <b>997 TB</b>.
 * Output is sent from worker to master: 64 KB/rupture variation x 12 rupture variations/task x 16 tasks/worker = 12 MB/worker x 2559 workers = 30 GB.
-Walltime is about 10 hours, so this requires <b>average throughput of 7.8 GB/sec</b>.
+Walltime is about 10 hours, so this requires <b>average throughput of 28.4 GB/sec</b>.
 Essentially, we have a bundling factor of 12; on average, each SGT we transfer is used to calculate 12 rupture variations.  If we permit more memory to be used for rupture variation storage and less for SGTs, we can increase the bundling factor but also increase the number of SGT requests.
@@ Line 76: / Line 76: @@
 If we decrease the SGT request size to 512 MB, this would create the following changes:
-* Average number of rupture variations per task increases to 37.
+* Average number of rupture variations per task increases to 30.
-* Number of tasks decreases to 17,100.
+* Number of tasks decreases to 28,800.
-* Each worker processes 6.9 tasks, which requires data from the SGT handler to the worker:  6.9 tasks * 38220 SGT points/task * 188 KB/SGT point = 47.3 GB/worker x 2559 workers = 118 TB; throughput of 3.4 GB/sec.
+* Each worker processes 11.3 tasks, which requires data from the SGT handler to the worker:  11.3 tasks * 130310 SGT points/task * 188 KB/SGT point = 262.8 GB/worker x 2559 workers = 657 TB; throughput of 18.7 GB/sec.

@@ Line 27: / Line 27: @@
 * Task Manager (process <num SGT handlers>)
 *# Read in rupture list which we are synthesizing seismograms for
-*# For each rupture in the list, determine how many tasks are needed to calculate all the rupture variations for that task.  The limiting factor is memory per worker process, since (size of rupture variation)*num_rupture_variations + 1 GB sgt buffer < 1.8 GB.  If the rupture is in the checkpoint file, it's already been calculated and isn't included from the task list.
+*# For each rupture in the list, determine how many tasks are needed to calculate all the rupture variations for that task.  The limiting factor is memory per worker process, since (size of rupture variation)*num_rupture_variations + SGT buffer < 1.8 GB.  If the rupture is in the checkpoint file, it's already been calculated and isn't included from the task list.
 *# While there are still tasks in the task list:
 *## Handler worker requests for more work by sending a task description

@@ Line 67: / Line 67: @@
 * Broadcast of SGT points to SGT handler mapping, from master: 4 KB
 * Sending of SGT point headers, from master to individual SGT handlers: 911 KB/SGT handler x 1151 SGT handlers = 1 GB
-* Each worker processes 16 tasks, which requires data from the SGT handler to the worker:  16 tasks * 38220 SGT points/task * 188 KB/SGT point = 110 GB/worker x 2559 workers = 274 TB.
+* Each worker processes 16 tasks, which requires data from the SGT handler to the worker:  16 tasks * 38220 SGT points/task * 188 KB/SGT point = 110 GB/worker x 2559 workers = <b>274 TB</b>.
 * Output is sent from worker to master: 64 KB/rupture variation x 12 rupture variations/task x 16 tasks/worker = 12 MB/worker x 2559 workers = 30 GB.
-Walltime is about 10 hours, so this requires throughput of 7.8 GB/sec.
+Walltime is about 10 hours, so this requires <b>average throughput of 7.8 GB/sec</b>.
 Essentially, we have a bundling factor of 12; on average, each SGT we transfer is used to calculate 12 rupture variations.  If we permit more memory to be used for rupture variation storage and less for SGTs, we can increase the bundling factor but also increase the number of SGT requests.

@@ Line 49: / Line 49: @@
 By far, the majority of the communication is between the SGT handlers and the workers.  For an average 1 Hz CyberShake run, here are communication estimates.
-Assumptions:
+==== Assumptions ====
 * 1 Hz CyberShake run
 * 750 GB SGTs per component (8.39M points)
@@ Line 62: / Line 62: @@
 * Average of 500 SGTs per request.
-Communication
+==== Communication ====
-* Broadcast of SGT header, from master: 480 MB
+* Broadcast of SGT header, from master: 192 MB
 * Broadcast of SGT points to SGT handler mapping, from master: 4 KB
 * Sending of SGT point headers, from master to individual SGT handlers: 911 KB/SGT handler x 1151 SGT handlers = 1 GB
@@ Line 71: / Line 71: @@
 Walltime is about 10 hours, so this requires throughput of 7.8 GB/sec.

← Older revision		Revision as of 18:54, 21 April 2015
Line 70:		Line 70:
	* Output is sent from worker to master: 64 KB/rupture variation x 12 rupture variations/task x 16 tasks/worker = 12 MB/worker x 2559 workers = 30 GB.		* Output is sent from worker to master: 64 KB/rupture variation x 12 rupture variations/task x 16 tasks/worker = 12 MB/worker x 2559 workers = 30 GB.

−	Walltime is about 10 hours, so this requires throughput of 7.9 GB/sec.	+	Walltime is about 10 hours, so this requires throughput of 7.8 GB/sec.
−
−	~~== Improvement over previous post-processing ==~~
−
−
−	~~== Limitations ==~~

@@ Line 12: / Line 12: @@
 *# Reads in header information, broadcasts to everyone.
 *# Determines which SGT points go to which SGT handlers, broadcasts to everyone.
 *# While there is still work to do:
 *## Gathers data from workers and writes to files
@@ Line 19: / Line 20: @@
 *# Receive header information from master.
 *# Receive mapping of SGT points to handlers.
 *# Together, MPI-read in SGT files, with each handler reading its assigned section.
 *# While there is still work to do:
@@ Line 54: / Line 56: @@
 * 7000 ruptures
 * Average of 69 rupture variations per rupture
-* Average of 41,500 tasks
+* Average of 41,500 tasks = 12 rupture variations/task.
 Communication
-# Broadcast of SGT header from master: 480 MB
+* Broadcast of SGT header, from master: 480 MB
-# Broadcast of SGT points to SGT handler mapping:
+* Broadcast of SGT points to SGT handler mapping, from master: 4 KB
 == Improvement over previous post-processing ==

@@ Line 43: / Line 43: @@
 *## Send data to Master for writing to filesystem.
-== Communication patterns ==
+=== Communication patterns ===
 == Improvement over previous post-processing ==

← Older revision		Revision as of 16:07, 21 April 2015
Line 30:		Line 30:

	* Workers (process <num SGT handlers + 1> - end)		* Workers (process <num SGT handlers + 1> - end)
		+	*# Receive mapping of SGT points to handlers.
		+	*# While there is still work to do:
		+	*## Request and receive task from Task Manager.
		+	*## Perform task:
		+	*### Generate rupture variations
		+	*### Determine list of SGTs which are required
		+	*### Divide list into individual requests, not to exceed 1 GB (each request goes to 1 SGT Handler)
		+	*### For each request:
		+	*#### Request and obtain SGTs and header information from the correct SGT Handler
		+	*#### Perform processing and convolution of SGTs with each rupture variation
		+	*### Perform PSA and RotD calculations
		+	*## Send data to Master for writing to filesystem.

	== Communication patterns ==		== Communication patterns ==