CyberShake DAX Generator - Revision history

Scottcal: /* Java classes */

2018-02-28T20:16:55Z

Java classes

Scottcal: /* Call Graphs */

2018-02-28T20:15:33Z

Call Graphs

Scottcal: /* Stochastic post-processing */

2018-02-28T20:08:02Z

Stochastic post-processing

Scottcal: /* Call Graphs */

2018-02-28T20:07:34Z

Call Graphs

Scottcal: /* Call Graphs */

2018-02-28T19:35:49Z

Call Graphs

Scottcal: /* Call Graphs */

2018-02-28T19:34:59Z

Call Graphs

Scottcal: /* SGT Generation */

2018-02-28T19:34:20Z

SGT Generation

Scottcal at 19:33, 28 February 2018

2018-02-28T19:33:45Z

Scottcal at 19:33, 28 February 2018

2018-02-28T19:33:35Z

Scottcal: /* SGT Generation */

2018-02-28T19:19:04Z

SGT Generation

@@ Line 19: / Line 19: @@
 and helper classes:
-*CyberShake_Workflow_Container.java: This was designed for when we had multiple post-processing workflows per run.  It's still used to combine a pre PP subworkflow with a PP subworkflow, a DB subworkflow, and a post subworkflow.
+*CyberShake_Workflow_Container.java: This is used to hold the sub-workflows which are part of a PP workflow.  It's needed to set up dependencies correctly when running an integrated workflow, or if there are multiple post-processing workflows per run (this case is not used anymore).
 *DBConnect.java: Contains code for accessing databases.
 *PP_DAXParameters.java: Container class to track all science and technical parameters for a post-processing workflow which impact either which codes are called, or what to pass in the command-line to tasks.

← Older revision		Revision as of 20:15, 28 February 2018
Line 94:		Line 94:
	****...		****...
	****CyberShake_DB_DAXGen.'''createDBReportJob()'''		****CyberShake_DB_DAXGen.'''createDBReportJob()'''
		+
		+	=== Full integrated workflow ===
		+
		+	*'''CyberShake_Integrated_DAXGen.main()''': calls functions to create workflows, sets up dependencies between sub-workflows, writes to files
		+	**CyberShake_Integrated_DAXGen.'''parseCommandLine()''': parse the command-line, create RunIDQuery object, separate arguments into SGT and PP args
		+	**'''CyberShake_SGT_DAXGen.subMain()''': Create SGT workflow
		+	**<Same call graph as in SGT workflow, above>
		+	**'''CyberShake_Integrated_DAXGen.createJobIDJob()''': sets JobID in the database between SGT and PP, since the last SGT workflow job removes the Job ID
		+	**'''CyberShake_PP_DAXGen.subMain()''': Create PP workflow
		+	**<Same call graph as in PP workflow, above>

@@ Line 87: / Line 87: @@
 ******CyberShake_Sub_Stoch_DAXGen.'''createLocalVMJob()'''
 ******CyberShake_Sub_Stoch_DAXGen.'''createHFSynthJob()'''
-...
+******...
 ******CyberShake_Sub_Stoch_DAXGen.'''createLFSiteResponse()'''
 ***'''CyberShake_DB_DAXGen.makeDAX()''': based on parameters, creates jobs for database insertion, checking, and data product generation

@@ Line 48: / Line 48: @@
 *****CyberShake_SGT_DAXGen.'''addPreCVM()'''
 *****CyberShake_SGT_DAXGen.'''addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs.  At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
-******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies
+******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies, initializes RunIDQuery object
 *******CyberShake_AWP_SGT_DAXGen.'''addVMeshSingle()'''
 *******CyberShake_AWP_SGT_DAXGen.'''addPreSGT()'''
@@ Line 70: / Line 72: @@
 ******CyberShake_DB_DAXGen.'''createDBReportJob()'''
 ****'''CyberShake_PP_DAXGen.makePostDax()''': creates Post DAX with Update job

@@ Line 57: / Line 57: @@
 *'''CyberShake_PP_DAXGen.main()''': Sets up dependencies between DAXes created by subMain(), writes DAXes to files
-**'''CyberShake_PP_DAXGen.subMain()''': initializes PP_DAXParameters and CyberShake_PP_DAXGen objects
+**CyberShake_PP_DAXGen.'''subMain()''': initializes PP_DAXParameters and CyberShake_PP_DAXGen objects
-***'''CyberShake_PP_DAXGen.parseCommandLine()''': parses command-line arguments, populates PP_DAXParameters and RunIDQuery objects
+***CyberShake_PP_DAXGen.'''parseCommandLine()''': parses command-line arguments, populates PP_DAXParameters and RunIDQuery objects
-***'''CyberShake_PP_DAXGen.makeDAX()''': initializes RunIDQuery and CyberShake_Workflow_Container objects, and depending on parameters, calls functions to create DirectSynth, extraction, and synthesis jobs
+***CyberShake_PP_DAXGen.'''makeDAX()''': initializes RunIDQuery and CyberShake_Workflow_Container objects, and depending on parameters, calls functions to create DirectSynth, extraction, and synthesis jobs
-****'''CyberShake_PP_DAXGen.putFreqInDB()''': populates database with Max_Frequency value
+****CyberShake_PP_DAXGen.'''putFreqInDB()''': populates database with Max_Frequency value
-****'''CyberShake_PP_DAXGen.makePreDax()''': creates Pre DAX with Update job, MD5sum check jobs, set PP host job
+****CyberShake_PP_DAXGen.'''makePreDax()''': creates Pre DAX with Update job, MD5sum check jobs, set PP host job
-****'''CyberShake_PP_DAXGen.addDirectSynth()'''
+****CyberShake_PP_DAXGen.'''addDirectSynth()'''
-****'''CyberShake_PP_DAXGen.genDBProductsDAX()''': creates CyberShake_DB_DAXGen object, initializes with values from PP_DAXParameters
+****CyberShake_PP_DAXGen.'''genDBProductsDAX()''': creates CyberShake_DB_DAXGen object, initializes with values from PP_DAXParameters
 *****'''CyberShake_DB_DAXGen.makeDAX()''': based on parameters, creates jobs for database insertion, checking, and data product generation
-******'''CyberShake_DB_DAXGen.createDBInsertionJob()'''
+******CyberShake_DB_DAXGen.'''createDBInsertionJob()'''
-******'''CyberShake_DB_DAXGen.createDBCheckJob()'''
+******CyberShake_DB_DAXGen.'''createDBCheckJob()'''
 ******...
-******'''CyberShake_DB_DAXGen.createDBReportJob()'''
+******CyberShake_DB_DAXGen.'''createDBReportJob()'''
 ****'''CyberShake_PP_DAXGen.makePostDax()''': creates Post DAX with Update job

@@ Line 41: / Line 41: @@
 === SGT Generation ===
-*CyberShake_SGT_DAXGen.'''main()''': after workflows are created in subMain(), writes DAX object to file
+*'''CyberShake_SGT_DAXGen.main()''': after workflows are created in subMain(), writes DAX object to file
-**'''CyberShake_SGT_DAXGen.subMain()'''
+**CyberShake_SGT_DAXGen.'''subMain()'''
-***'''CyberShake_SGT_DAXGen.parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects
+***CyberShake_SGT_DAXGen.'''parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects
-***'''CyberShake_SGT_DAXGen.makeWorkflows()''': initializes the CyberShake_SGT_DAXGen object
+***CyberShake_SGT_DAXGen.'''makeWorkflows()''': initializes the CyberShake_SGT_DAXGen object
-****'''CyberShake_SGT_DAXGen.makeDAX()''': based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies
+****CyberShake_SGT_DAXGen.'''makeDAX()''': based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies
-*****'''CyberShake_SGT_DAXGen.addPreCVM()'''
+*****CyberShake_SGT_DAXGen.'''addPreCVM()'''
-*****'''CyberShake_SGT_DAXGen.addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs.  At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
+*****CyberShake_SGT_DAXGen.'''addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs.  At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
 ******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies
-*******'''CyberShake_AWP_SGT_DAXGen.addVMeshSingle()'''
+*******CyberShake_AWP_SGT_DAXGen.'''addVMeshSingle()'''
-*******'''CyberShake_AWP_SGT_DAXGen.addPreSGT()'''
+*******CyberShake_AWP_SGT_DAXGen.'''addPreSGT()'''
 *******...
-*******'''CyberShake_AWP_SGT_DAXGen.addUpdate()'''
+*******CyberShake_AWP_SGT_DAXGen.'''addUpdate()'''
 === Post-processing ===

@@ Line 40: / Line 40: @@
 Rough call graph:
-*CyberShake_SGT_DAXGen.main()
+*'''CyberShake_SGT_DAXGen.main()'''
-**CyberShake_SGT_DAXGen.subMain()
+**'''CyberShake_SGT_DAXGen.subMain()'''
-***CyberShake_SGT_DAXGen.parseCommandLine(): parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects
+***'''CyberShake_SGT_DAXGen.parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects
-***CyberShake_SGT_DAXGen.makeWorkflows(): initializes the CyberShake_SGT_DAXGen object
+***'''CyberShake_SGT_DAXGen.makeWorkflows()''': initializes the CyberShake_SGT_DAXGen object
-****CyberShake_SGT_DAXGen.makeDAX(): based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies
+****'''CyberShake_SGT_DAXGen.makeDAX()''': based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies
-*****CyberShake_SGT_DAXGen.addPreCVM()
+*****'''CyberShake_SGT_DAXGen.addPreCVM()'''
-*****CyberShake_SGT_DAXGen.addGenSGTDAX(): job which creates a workflow which contains the rest of the SGT jobs.  At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
+*****'''CyberShake_SGT_DAXGen.addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs.  At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
-******CyberShake_AWP_SGT_DAXGen.main(): parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies
+******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies
-*******CyberShake_AWP_SGT_DAXGen.addVMeshSingle()
+*******'''CyberShake_AWP_SGT_DAXGen.addVMeshSingle()'''
-*******CyberShake_AWP_SGT_DAXGen.addPreSGT()
+*******'''CyberShake_AWP_SGT_DAXGen.addPreSGT()'''
 *******...
-*******CyberShake_AWP_SGT_DAXGen.addUpdate()
+*******'''CyberShake_AWP_SGT_DAXGen.addUpdate()'''