Parallel Broadband - Revision history

Maechlin: /* Overview */

2012-03-13T03:19:40Z

Overview

← Older revision		Revision as of 03:19, 13 March 2012
Line 9:		Line 9:
	# goodness of fit calculations		# goodness of fit calculations
	# plotting		# plotting


	Rupture generation reads in a few input files and outputs a single source rupture description. Seismogram synthesis and site effects run on a list of N stations. Goodness of fit and plotting require information from the previously generated seismograms of all stations.		Rupture generation reads in a few input files and outputs a single source rupture description. Seismogram synthesis and site effects run on a list of N stations. Goodness of fit and plotting require information from the previously generated seismograms of all stations.


	There are three levels in which the platform can be parallelized:		There are three levels in which the platform can be parallelized:
Line 22:		Line 20:

	The approach described here is the second option. Seismogram synthesis and site effects are broken out by station and those modules are run in parallel. A simulation involving 100 stations may be run on up to 100 cores, giving an approximately 100x speed-up in execution speed.		The approach described here is the second option. Seismogram synthesis and site effects are broken out by station and those modules are run in parallel. A simulation involving 100 stations may be run on up to 100 cores, giving an approximately 100x speed-up in execution speed.


	== Parallel Broadband ==		== Parallel Broadband ==

Patrices at 19:03, 16 February 2012

2012-02-16T19:03:56Z

← Older revision		Revision as of 19:03, 16 February 2012
Line 203:		Line 203:
	The batch planning process saves metadata in the ./batch/<batch_name> directory. Three files are saved for the batch:		The batch planning process saves metadata in the ./batch/<batch_name> directory. Three files are saved for the batch:

	* <batch_name>.bxml : Batch workflow description (which can be executed by bbp_batch_sim.py with the -b option)		* <batch_name>.bxml : Batch workflow description (which can be executed by bbp_batch_sim.py with the -x option)
	* index.txt : Index table that maps the simulation ID of each simulation to all variable assignments from your option template		* index.txt : Index table that maps the simulation ID of each simulation to all variable assignments from your option template
	* options.txt : Original parameterized option template		* options.txt : Original parameterized option template

Patrices at 19:03, 16 February 2012

2012-02-16T19:03:31Z

← Older revision		Revision as of 19:03, 16 February 2012
Line 50:		Line 50:
	* Running a parallel workflow remotely:		* Running a parallel workflow remotely:
	** BBP installation (with all of its dependencies) on both local and remote system		** BBP installation (with all of its dependencies) on both local and remote system
	** GRAM support on remote system		** GRAM support on local and remote system
	** GridFTP support on both local and remote systems		** GridFTP support on both local and remote systems
	** Job manager on remote system		** Job manager on remote system
Line 160:		Line 160:
	y		y
	y		y
			n
	n		n
	</pre>		</pre>
Line 199:		Line 200:

	The option template file is a parameterized option file similar to what run_bbp_2G.py accepts, except lines in the file may be replaced with variables that expand out into one or more options. See [[#Running a Parallel Workflow Remotely]] for an example. The batch name parameter is just a user-defined label to assign to the batch. The -g option specifies to generate the batch only, and not execute it. A previously generated batch can be executed with the -x option.		The option template file is a parameterized option file similar to what run_bbp_2G.py accepts, except lines in the file may be replaced with variables that expand out into one or more options. See [[#Running a Parallel Workflow Remotely]] for an example. The batch name parameter is just a user-defined label to assign to the batch. The -g option specifies to generate the batch only, and not execute it. A previously generated batch can be executed with the -x option.

			The batch planning process saves metadata in the ./batch/<batch_name> directory. Three files are saved for the batch:

			* <batch_name>.bxml : Batch workflow description (which can be executed by bbp_batch_sim.py with the -b option)
			* index.txt : Index table that maps the simulation ID of each simulation to all variable assignments from your option template
			* options.txt : Original parameterized option template

Patrices at 00:27, 19 January 2012

2012-01-19T00:27:23Z

← Older revision		Revision as of 00:27, 19 January 2012
Line 45:		Line 45:

	* Running a parallel workflow locally:		* Running a parallel workflow locally:
	** BBP installation		** BBP installation (with all of its dependencies)
	** Optional job scheduler		** Optional job manager/scheduler

	* Running a parallel workflow remotely:		* Running a parallel workflow remotely:
	** BBP installation on both local and remote system		** BBP installation (with all of its dependencies) on both local and remote system
	** GRAM support on remote system		** GRAM support on remote system
	** GridFTP support on both local and remote systems		** GridFTP support on both local and remote systems
	** Job ~~scheduler~~ on remote system		** Job manager on remote system


Line 198:		Line 198:


	The option template file is a parameterized option file similar to what run_bbp_2G.py accepts, except lines in the file may be replaced with variables that expand out into one or more options. See [[#Running a Parallel Workflow Remotely]] for an example. The batch name parameter is just a user-~~define~~ label to assign to the batch. The -g option specifies to generate the batch only, and not execute it. A previously generated batch can be executed with the -x option.		The option template file is a parameterized option file similar to what run_bbp_2G.py accepts, except lines in the file may be replaced with variables that expand out into one or more options. See [[#Running a Parallel Workflow Remotely]] for an example. The batch name parameter is just a user-defined label to assign to the batch. The -g option specifies to generate the batch only, and not execute it. A previously generated batch can be executed with the -x option.

Patrices at 23:16, 18 January 2012

2012-01-18T23:16:32Z

← Older revision		Revision as of 23:16, 18 January 2012
Line 39:		Line 39:


	So, Parallel Broadband can be run on a quad-core desktop computer and use all four cores. It can also run as a ~~PBS/Torque~~ job submitted to USC HPCC, and use as many cores as there are stations in the simulation.		So, Parallel Broadband can be run on a quad-core desktop computer and use all four cores. It can also run as a job submitted to USC HPCC, and use as many cores as there are stations in the simulation.

Patrices at 23:13, 18 January 2012

2012-01-18T23:13:47Z

← Older revision		Revision as of 23:13, 18 January 2012
Line 46:		Line 46:
	* Running a parallel workflow locally:		* Running a parallel workflow locally:
	** BBP installation		** BBP installation
			** Optional job scheduler

	* Running a parallel workflow remotely:		* Running a parallel workflow remotely:
Line 51:		Line 52:
	** GRAM support on remote system		** GRAM support on remote system
	** GridFTP support on both local and remote systems		** GridFTP support on both local and remote systems
			** Job scheduler on remote system

Patrices at 22:39, 18 January 2012

2012-01-18T22:39:39Z

← Older revision		Revision as of 22:39, 18 January 2012
Line 182:		Line 182:
	</pre>		</pre>

	The program is blocking in that it will end only when the remote job is finished and the simulation data has been retrieved.		The -k option specifies the host in self.HOSTS (./comps/install_cfg.py) in which to send the job. The -b option assigns a string label to the batch. The -o option provides the option template file. The remaining options specify the job parameters. The program is blocking in that it will end only when the remote job is finished and the simulation data has been retrieved.


Line 189:		Line 189:
	The Broadband platform provides a batching interface that allows multiple simulations to be created and run with a single step. This batching mechanism interacts with the parallel interface as can be seen in [[#Running a Parallel Workflow Remotely]] but it can also be used independently.		The Broadband platform provides a batching interface that allows multiple simulations to be created and run with a single step. This batching mechanism interacts with the parallel interface as can be seen in [[#Running a Parallel Workflow Remotely]] but it can also be used independently.

	To create a batch of simulations that can be run together as a unit, use the bbp_batch_sim.py utility.		To create a batch of simulations that can be run together as a unit, use the bbp_batch_sim.py utility:

	<pre>		<pre>

Patrices at 22:35, 18 January 2012

2012-01-18T22:35:27Z

@@ Line 40: / Line 40: @@
 So, Parallel Broadband can be run on a quad-core desktop computer and use all four cores. It can also run as a PBS/Torque job submitted to USC HPCC, and use as many cores as there are stations in the simulation.
@@ Line 132: / Line 143: @@
-Input files are generally a rupture description and a station list for each simulation that the user wishes to run. The option template file is a parameterized option file similar to what run_bbp_2G.py accepts, except lines in the file may be replaced with variables that expand out into one or more options. In this way, many simulations can be defined. The following is an example option template file for a hanging wall simulation:
+Input files are generally a rupture description and a station list for each simulation that the user wishes to run. The option template file is a parameterized option file similar to what run_bbp_2G.py accepts, except lines in the file may be replaced with variables that expand out into one or more options. In this way, many simulations can be defined. The following is an example option template file for a hanging wall simulation, rob_template.txt:
 <pre>
@@ Line 164: / Line 175: @@
 Keep in mind that the paths shown here are all on the local system. When this option file is sent to the remote HPC system, all of these local paths are changed by BBP to point to the unpacked input files that it transfered over with GridFTP.
 == Simulation Batching ==
-The Broadband platform provides a batching interface that allows multiple simulations to be created and run with a single step. This batching mechanism interacts with the parallel interface as can be seen in [[#Running a Parallel Workflow Remotely]]
+The Broadband platform provides a batching interface that allows multiple simulations to be created and run with a single step. This batching mechanism interacts with the parallel interface as can be seen in [[#Running a Parallel Workflow Remotely]] but it can also be used independently.

Patrices at 22:18, 18 January 2012

2012-01-18T22:18:37Z

@@ Line 44: / Line 44: @@
 === Installation ===
-Parallel BBP is fully integrated with the original Broadband platform and shares its installation process. The parallel functionality may require an optional configuration step depending on how the system is run. When a parallel simulation is run as a job on an HPC host, the user may need to manually setup the default environment on the compute nodes. Specifically, the user may need to configure environment variables (PATH and PYTHONPATH, for example) so that the correct version of Python is used. The shell script ./comps/setup_bbp_env.sh is the place to declare those custom environment settings.
+Parallel BBP is fully integrated with the original Broadband platform and shares its installation process. The parallel functionality may require optional configuration steps depending on how the system is run.
 If run on a stand-alone computer, customizing that shell script is unnecessary.
@@ Line 84: / Line 121: @@
 === Running a Parallel Workflow Remotely ===
-TBD
+One or more simulations may be run remotely on an HPC system with GRAM by using the ./comps/run_bbp_2g_gram.py script. This is an advanced capability. In this mode, the user specifies an option template file (containing all the answers to a run_bbp_2G.py interactive session), lists their input files, and specifies some job parameters for the remote job scheduler (walltime, number of cores, etc). The system then performs these steps:

Patrices at 22:01, 18 January 2012

2012-01-18T22:01:09Z

← Older revision		Revision as of 22:01, 18 January 2012
Line 26:		Line 26:
	== Parallel Broadband ==		== Parallel Broadband ==

	A new software layer has been added on top of the existing BBP software stack that allows a user to parallelize an existing serial workflow and execute it either on multiple cores within a single host or multiple nodes on a HPC resource. The user begins by creating a simulation workflow description in the traditional way (using ~~run_bbp~~.py) and saving it to disk. Then, the new parallel interface is invoked on that workflow description. The serial workflow is parsed, parallelized, and optionally executed.		A new software layer has been added on top of the existing BBP software stack that allows a user to parallelize an existing serial workflow and execute it either on multiple cores within a single host or multiple nodes on a HPC resource. The user begins by creating a simulation workflow description in the traditional way (using run_bbp_2G.py) and saving it to disk. Then, the new parallel interface is invoked on that workflow description. The serial workflow is parsed, parallelized, and optionally executed.


Line 44:		Line 44:
	=== Installation ===		=== Installation ===

	Parallel BBP is fully integrated with the original Broadband platform and shares its installation process. The parallel functionality may require an optional configuration step depending on how the system is run. When run on an HPC host, ~~ssh is used to login~~ to compute nodes ~~from a MoM node~~. ~~The~~ user may need to ~~setup~~ environment variables (PATH and PYTHONPATH, for example) ~~immediately following~~ that ~~login~~. The shell script ./comps/setup_bbp_env.sh is the place to declare those environment settings.		Parallel BBP is fully integrated with the original Broadband platform and shares its installation process. The parallel functionality may require an optional configuration step depending on how the system is run. When a parallel simulation is run as a job on an HPC host, the user may need to manually setup the default environment on the compute nodes. Specifically, the user may need to configure environment variables (PATH and PYTHONPATH, for example) so that the correct version of Python is used. The shell script ./comps/setup_bbp_env.sh is the place to declare those custom environment settings.

	If run on a stand-alone computer, customizing that shell script is unnecessary.		If run on a stand-alone computer, customizing that shell script is unnecessary.


	=== ~~Converting Serial Workflow to~~ Parallel Workflow ===		=== Running a Parallel Workflow Locally ===

	~~TBD~~		On the machine you wish to run the simulation, generate a serial workflow description using run_bbp_2G.py with the "-g" option.

			<pre>
			$ ./run_bbp_2G.py -g
			</pre>

	=== Running ~~the~~ Parallel Workflow===		You will be prompted interactively for the codebases and inputs to be used in your simulation unless you provide a response file with the "-o" option. Take note of the path to the XML file that is saved at the end.


			Then, convert the serial workflow to a parallel workflow:
			<pre>
			$ ./bbp_s2p.py -g -s ../xml/1234567.xml
			</pre>

			This command takes the serial workflow "../xml/1234567.xml", converts it to a parallel workflow, and saves it to "../parallel/1234567.pxml"


			At this point you may execute the parallel workflow. On a stand-alone machine with four processors, execute the following:

			<pre>
			$ ./bbp_s2p.py -x ../parallel/1234567.pxml -c 4 -n localhost
			</pre>


			In a PBS script on USC HPCC as a four core job, execute the following:

			<pre>
			$ ./bbp_s2p.py -x ../parallel/1234567.pxml -c 4 -n $PBS_NODEFILE
			</pre>


			=== Running a Parallel Workflow Remotely ===

	TBD		TBD