CyberShake DAX Generator - Revision history

Scottcal: /* Java classes */

2018-02-28T20:16:55Z

Java classes

← Older revision		Revision as of 20:16, 28 February 2018
Line 19:		Line 19:
	and helper classes:		and helper classes:

	*CyberShake_Workflow_Container.java: This ~~was designed for when we had multiple post~~-~~processing~~ workflows ~~per run~~. It's ~~still used~~ to ~~combine a pre PP subworkflow with a PP subworkflow~~, ~~a DB subworkflow, and a~~ post ~~subworkflow~~.		*CyberShake_Workflow_Container.java: This is used to hold the sub-workflows which are part of a PP workflow. It's needed to set up dependencies correctly when running an integrated workflow, or if there are multiple post-processing workflows per run (this case is not used anymore).
	*DBConnect.java: Contains code for accessing databases.		*DBConnect.java: Contains code for accessing databases.
	*PP_DAXParameters.java: Container class to track all science and technical parameters for a post-processing workflow which impact either which codes are called, or what to pass in the command-line to tasks.		*PP_DAXParameters.java: Container class to track all science and technical parameters for a post-processing workflow which impact either which codes are called, or what to pass in the command-line to tasks.

Scottcal: /* Call Graphs */

2018-02-28T20:15:33Z

Call Graphs

← Older revision		Revision as of 20:15, 28 February 2018
Line 94:		Line 94:
	****...		****...
	****CyberShake_DB_DAXGen.'''createDBReportJob()'''		****CyberShake_DB_DAXGen.'''createDBReportJob()'''

			=== Full integrated workflow ===

			*'''CyberShake_Integrated_DAXGen.main()''': calls functions to create workflows, sets up dependencies between sub-workflows, writes to files
			**CyberShake_Integrated_DAXGen.'''parseCommandLine()''': parse the command-line, create RunIDQuery object, separate arguments into SGT and PP args
			**'''CyberShake_SGT_DAXGen.subMain()''': Create SGT workflow
			**<Same call graph as in SGT workflow, above>
			**'''CyberShake_Integrated_DAXGen.createJobIDJob()''': sets JobID in the database between SGT and PP, since the last SGT workflow job removes the Job ID
			**'''CyberShake_PP_DAXGen.subMain()''': Create PP workflow
			**<Same call graph as in PP workflow, above>

Scottcal: /* Stochastic post-processing */

2018-02-28T20:08:02Z

Stochastic post-processing

← Older revision		Revision as of 20:08, 28 February 2018
Line 87:		Line 87:
	******CyberShake_Sub_Stoch_DAXGen.'''createLocalVMJob()'''		******CyberShake_Sub_Stoch_DAXGen.'''createLocalVMJob()'''
	******CyberShake_Sub_Stoch_DAXGen.'''createHFSynthJob()'''		******CyberShake_Sub_Stoch_DAXGen.'''createHFSynthJob()'''
	...		******...
	******CyberShake_Sub_Stoch_DAXGen.'''createLFSiteResponse()'''		******CyberShake_Sub_Stoch_DAXGen.'''createLFSiteResponse()'''
	***'''CyberShake_DB_DAXGen.makeDAX()''': based on parameters, creates jobs for database insertion, checking, and data product generation		***'''CyberShake_DB_DAXGen.makeDAX()''': based on parameters, creates jobs for database insertion, checking, and data product generation

Scottcal: /* Call Graphs */

2018-02-28T20:07:34Z

Call Graphs

@@ Line 48: / Line 48: @@
 *****CyberShake_SGT_DAXGen.'''addPreCVM()'''
 *****CyberShake_SGT_DAXGen.'''addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs.  At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
-******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies
+******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies, initializes RunIDQuery object
 *******CyberShake_AWP_SGT_DAXGen.'''addVMeshSingle()'''
 *******CyberShake_AWP_SGT_DAXGen.'''addPreSGT()'''
@@ Line 70: / Line 72: @@
 ******CyberShake_DB_DAXGen.'''createDBReportJob()'''
 ****'''CyberShake_PP_DAXGen.makePostDax()''': creates Post DAX with Update job

Scottcal: /* Call Graphs */

2018-02-28T19:35:49Z

Call Graphs

← Older revision		Revision as of 19:35, 28 February 2018
Line 57:		Line 57:

	*'''CyberShake_PP_DAXGen.main()''': Sets up dependencies between DAXes created by subMain(), writes DAXes to files		*'''CyberShake_PP_DAXGen.main()''': Sets up dependencies between DAXes created by subMain(), writes DAXes to files
	**'''~~CyberShake_PP_DAXGen.~~subMain()''': initializes PP_DAXParameters and CyberShake_PP_DAXGen objects		**CyberShake_PP_DAXGen.'''subMain()''': initializes PP_DAXParameters and CyberShake_PP_DAXGen objects
	***'''~~CyberShake_PP_DAXGen.~~parseCommandLine()''': parses command-line arguments, populates PP_DAXParameters and RunIDQuery objects		***CyberShake_PP_DAXGen.'''parseCommandLine()''': parses command-line arguments, populates PP_DAXParameters and RunIDQuery objects
	***'''~~CyberShake_PP_DAXGen.~~makeDAX()''': initializes RunIDQuery and CyberShake_Workflow_Container objects, and depending on parameters, calls functions to create DirectSynth, extraction, and synthesis jobs		***CyberShake_PP_DAXGen.'''makeDAX()''': initializes RunIDQuery and CyberShake_Workflow_Container objects, and depending on parameters, calls functions to create DirectSynth, extraction, and synthesis jobs
	****'''~~CyberShake_PP_DAXGen.~~putFreqInDB()''': populates database with Max_Frequency value		****CyberShake_PP_DAXGen.'''putFreqInDB()''': populates database with Max_Frequency value
	****'''~~CyberShake_PP_DAXGen.~~makePreDax()''': creates Pre DAX with Update job, MD5sum check jobs, set PP host job		****CyberShake_PP_DAXGen.'''makePreDax()''': creates Pre DAX with Update job, MD5sum check jobs, set PP host job
	****'''~~CyberShake_PP_DAXGen.~~addDirectSynth()'''		****CyberShake_PP_DAXGen.'''addDirectSynth()'''
	****'''~~CyberShake_PP_DAXGen.~~genDBProductsDAX()''': creates CyberShake_DB_DAXGen object, initializes with values from PP_DAXParameters		****CyberShake_PP_DAXGen.'''genDBProductsDAX()''': creates CyberShake_DB_DAXGen object, initializes with values from PP_DAXParameters
	*****'''CyberShake_DB_DAXGen.makeDAX()''': based on parameters, creates jobs for database insertion, checking, and data product generation		*****'''CyberShake_DB_DAXGen.makeDAX()''': based on parameters, creates jobs for database insertion, checking, and data product generation
	******'''~~CyberShake_DB_DAXGen.~~createDBInsertionJob()'''		******CyberShake_DB_DAXGen.'''createDBInsertionJob()'''
	******'''~~CyberShake_DB_DAXGen.~~createDBCheckJob()'''		******CyberShake_DB_DAXGen.'''createDBCheckJob()'''
	******...		******...
	******'''~~CyberShake_DB_DAXGen.~~createDBReportJob()'''		******CyberShake_DB_DAXGen.'''createDBReportJob()'''
	****'''CyberShake_PP_DAXGen.makePostDax()''': creates Post DAX with Update job		****'''CyberShake_PP_DAXGen.makePostDax()''': creates Post DAX with Update job

Scottcal: /* Call Graphs */

2018-02-28T19:34:59Z

Call Graphs

← Older revision		Revision as of 19:34, 28 February 2018
Line 41:		Line 41:
	=== SGT Generation ===		=== SGT Generation ===

	*~~CyberShake_SGT_DAXGen.~~'''main()''': after workflows are created in subMain(), writes DAX object to file		*'''CyberShake_SGT_DAXGen.main()''': after workflows are created in subMain(), writes DAX object to file
	**'''~~CyberShake_SGT_DAXGen.~~subMain()'''		**CyberShake_SGT_DAXGen.'''subMain()'''
	***'''~~CyberShake_SGT_DAXGen.~~parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects		***CyberShake_SGT_DAXGen.'''parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects
	***'''~~CyberShake_SGT_DAXGen.~~makeWorkflows()''': initializes the CyberShake_SGT_DAXGen object		***CyberShake_SGT_DAXGen.'''makeWorkflows()''': initializes the CyberShake_SGT_DAXGen object
	****'''~~CyberShake_SGT_DAXGen.~~makeDAX()''': based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies		****CyberShake_SGT_DAXGen.'''makeDAX()''': based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies
	*****'''~~CyberShake_SGT_DAXGen.~~addPreCVM()'''		*****CyberShake_SGT_DAXGen.'''addPreCVM()'''
	*****'''~~CyberShake_SGT_DAXGen.~~addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs. At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.		*****CyberShake_SGT_DAXGen.'''addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs. At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
	******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies		******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies
	*******'''~~CyberShake_AWP_SGT_DAXGen.~~addVMeshSingle()'''		*******CyberShake_AWP_SGT_DAXGen.'''addVMeshSingle()'''
	*******'''~~CyberShake_AWP_SGT_DAXGen.~~addPreSGT()'''		*******CyberShake_AWP_SGT_DAXGen.'''addPreSGT()'''
	*******...		*******...
	*******'''~~CyberShake_AWP_SGT_DAXGen.~~addUpdate()'''		*******CyberShake_AWP_SGT_DAXGen.'''addUpdate()'''

	=== Post-processing ===		=== Post-processing ===

Scottcal: /* SGT Generation */

2018-02-28T19:34:20Z

SGT Generation

← Older revision		Revision as of 19:34, 28 February 2018
Line 41:		Line 41:
	=== SGT Generation ===		=== SGT Generation ===

	*'''~~CyberShake_SGT_DAXGen.~~main()''': after workflows are created in subMain(), writes DAX object to file		*CyberShake_SGT_DAXGen.'''main()''': after workflows are created in subMain(), writes DAX object to file
	**'''CyberShake_SGT_DAXGen.subMain()'''		**'''CyberShake_SGT_DAXGen.subMain()'''
	***'''CyberShake_SGT_DAXGen.parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects		***'''CyberShake_SGT_DAXGen.parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects

Scottcal at 19:33, 28 February 2018

2018-02-28T19:33:45Z

← Older revision		Revision as of 19:33, 28 February 2018
Line 54:		Line 54:
	*******'''CyberShake_AWP_SGT_DAXGen.addUpdate()'''		*******'''CyberShake_AWP_SGT_DAXGen.addUpdate()'''

	== Post-processing ===		=== Post-processing ===

	*'''CyberShake_PP_DAXGen.main()''': Sets up dependencies between DAXes created by subMain(), writes DAXes to files		*'''CyberShake_PP_DAXGen.main()''': Sets up dependencies between DAXes created by subMain(), writes DAXes to files

Scottcal at 19:33, 28 February 2018

2018-02-28T19:33:35Z

@@ Line 37: / Line 37: @@
 |}
-== SGT Generation ==
+== Call Graphs ==
-Rough call graph:
+=== SGT Generation ===
 *'''CyberShake_SGT_DAXGen.main()'''
 **'''CyberShake_SGT_DAXGen.subMain()'''
 ***'''CyberShake_SGT_DAXGen.parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects
@@ Line 52: / Line 53: @@
 *******...
 *******'''CyberShake_AWP_SGT_DAXGen.addUpdate()'''

Scottcal: /* SGT Generation */

2018-02-28T19:19:04Z

SGT Generation

← Older revision		Revision as of 19:19, 28 February 2018
Line 40:		Line 40:

	Rough call graph:		Rough call graph:
	*CyberShake_SGT_DAXGen.main()		*'''CyberShake_SGT_DAXGen.main()'''
	**CyberShake_SGT_DAXGen.subMain()		**'''CyberShake_SGT_DAXGen.subMain()'''
	***CyberShake_SGT_DAXGen.parseCommandLine(): parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects		***'''CyberShake_SGT_DAXGen.parseCommandLine()''': parses command-line arguments, populates SGT_DAXParameters and RunIDQuery objects
	***CyberShake_SGT_DAXGen.makeWorkflows(): initializes the CyberShake_SGT_DAXGen object		***'''CyberShake_SGT_DAXGen.makeWorkflows()''': initializes the CyberShake_SGT_DAXGen object
	****CyberShake_SGT_DAXGen.makeDAX(): based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies		****'''CyberShake_SGT_DAXGen.makeDAX()''': based on parameters, calls appropriate sequence of functions to create individual jobs with dependencies
	*****CyberShake_SGT_DAXGen.addPreCVM()		*****'''CyberShake_SGT_DAXGen.addPreCVM()'''
	*****CyberShake_SGT_DAXGen.addGenSGTDAX(): job which creates a workflow which contains the rest of the SGT jobs. At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.		*****'''CyberShake_SGT_DAXGen.addGenSGTDAX()''': job which creates a workflow which contains the rest of the SGT jobs. At runtime, CyberShake_AWP_SGT_DAXGen.main() is called.
	******CyberShake_AWP_SGT_DAXGen.main(): parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies		******'''CyberShake_AWP_SGT_DAXGen.main()''': parses command-line options, calls appropriate sequence of functions to create individual jobs with dependencies
	*******CyberShake_AWP_SGT_DAXGen.addVMeshSingle()		*******'''CyberShake_AWP_SGT_DAXGen.addVMeshSingle()'''
	*******CyberShake_AWP_SGT_DAXGen.addPreSGT()		*******'''CyberShake_AWP_SGT_DAXGen.addPreSGT()'''
	*******...		*******...
	*******CyberShake_AWP_SGT_DAXGen.addUpdate()		*******'''CyberShake_AWP_SGT_DAXGen.addUpdate()'''