CyberShake Testing - Revision history

Maechlin at 23:42, 21 February 2011

2011-02-21T23:42:17Z

Maechlin at 23:32, 21 February 2011

2011-02-21T23:32:11Z

Maechlin at 19:13, 20 February 2011

2011-02-20T19:13:59Z

Maechlin at 19:13, 20 February 2011

2011-02-20T19:13:36Z

Maechlin at 07:58, 20 February 2011

2011-02-20T07:58:46Z

Maechlin at 07:57, 20 February 2011

2011-02-20T07:57:53Z

Maechlin at 07:55, 20 February 2011

2011-02-20T07:55:09Z

Maechlin: Created page with 'Computational scale and complexity of the SCEC CyberShake system requires automated and repeatable system-level testing capabilities. The CyberShake testing must be capable of en…'

2011-02-20T07:50:41Z

Created page with 'Computational scale and complexity of the SCEC CyberShake system requires automated and repeatable system-level testing capabilities. The CyberShake testing must be capable of en…'

New page

Computational scale and complexity of the SCEC CyberShake system requires automated and repeatable system-level testing capabilities. The CyberShake testing must be capable of end-to-end testing, showing that all elements, inputs, earth models, computational codes, and data processing and reduction codes all work together.

The CyberShake Testing system combines a distributed workflow-based HPC software testing harness together with a database of reference problems and expected solutions.

@@ Line 25: / Line 25: @@
 CyberShake 2.0 development requires a workflow-based system capable of automating multiple CyberShake HPC calculations.
-In 2011, Wikipedia describes Test Harness in the following terms.
+An alternative definition of a test harness is software constructed to facilitate integration testing.
-<pre>
-"In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the Test execution engine and the Test script repository.
+In 2011, Wikipedia describes Test Harness in the following terms. "In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the Test execution engine and the Test script repository.
 Test harnesses allow for the automation of tests. They can call functions with supplied parameters and print out and compare the results to the desired value. The test harness is a hook to the developed code, which can be tested using an automation framework.
@@ Line 34: / Line 34: @@
 The typical objectives of a test harness are to:
 *[http://en.wikipedia.org/wiki/Test_harness WikiPedia Test Harness]

@@ Line 21: / Line 21: @@
 #List of Sites
-== CyberShake Test Harness ==
+== CyberShake Automated Testing Framework  ==
-We required a workflow-based system capable of automating multiple CyberShake HPC calculations. This is modeled on the virtual data processing model of Pegasus.
+CyberShake 2.0 development requires a workflow-based system capable of automating multiple CyberShake HPC calculations.
 == CyberShake Test Oracle ==

@@ Line 35: / Line 35: @@
 == Related Entries ==
-[[CyberShake]]
+*[[CyberShake]]
-[[CyberShake Workplan]]
+*[[CyberShake Workplan]]
 == See Also ==
 [[Main Page]]

@@ Line 3: / Line 3: @@
 The CyberShake Testing system combines a distributed workflow-based HPC software testing harness together with a database of reference problems and expected solutions.
-== CyberShake Test Harness ==
+== CyberShake Testing Requirements ==
+#Must support broad range of HPC codes including calculations too large to run on SCEC computers
-We required a workflow-based system capable of automating multiple CyberShake HPC calculations. This is modeled on the virtual data processing model of Pegasus.
+#Must perform end-to-end calculations, capable of performing a series of calculations
+#Must support multiple test evaluations including file-based comparisons, integer and floating-point with tolerance comparisons, and relational database entry comparisons.
-== CyberShake Test Oracle ==
+#Must be modular, capable of testing performance of alternative codes on equivalent calculations
+#Must support and help manage a test repository that contains reference problems and reference results
-We require a reference database that describes specific test problems, describes the input files, and output files, and defines a list of expected results.
+#Must support provisioning of, and job submission to, multiple HPC resource providers including USC HPCC, TeraGrid, and possibly DOE computer resources
+#Must provide well-define metadata description of every test result that describes the code under test, the input parameters used, the reference results used in comparisons, and final test results
 == Required Evaluation Tests ==
 #Rupture Generator
@@ Line 19: / Line 20: @@
 #Ten moderate earthquake distributed around California
 #List of Sites
 == References ==

@@ Line 22: / Line 22: @@
 == References ==
-#Maechling, P., E. Deelman, Y. Cui (2009), Implementing Software Acceptance Tests as Scientific Workflows, Proceedings of the IEEE International Parallel & Distributed Processing Symposium 2009, Las Vegas Nevada, July, 2009
+# Maechling, P., Deelman, E., Cui, Y. (2009), Implementing Software Acceptance Tests as Scientific Workflows, Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications, Hamid R. Arabnia (Ed.): PDPTA 2009, Las Vegas, Nevada, USA, July 13-17, 2009, 2 Volumes. CSREA Press 2009, ISBN 1-60132-123-6, pp. 317-323
 == Related Entries ==

← Older revision		Revision as of 07:57, 20 February 2011
Line 21:		Line 21:

	== References ==		== References ==
		+
		+	#Maechling, P., E. Deelman, Y. Cui (2009), Implementing Software Acceptance Tests as Scientific Workflows, Proceedings of the IEEE International Parallel & Distributed Processing Symposium 2009, Las Vegas Nevada, July, 2009

	== Related Entries ==		== Related Entries ==

← Older revision		Revision as of 07:55, 20 February 2011
Line 2:		Line 2:

	The CyberShake Testing system combines a distributed workflow-based HPC software testing harness together with a database of reference problems and expected solutions.		The CyberShake Testing system combines a distributed workflow-based HPC software testing harness together with a database of reference problems and expected solutions.
		+
		+	== CyberShake Test Harness ==
		+
		+	We required a workflow-based system capable of automating multiple CyberShake HPC calculations. This is modeled on the virtual data processing model of Pegasus.
		+
		+	== CyberShake Test Oracle ==
		+
		+	We require a reference database that describes specific test problems, describes the input files, and output files, and defines a list of expected results.
		+
		+	== Required Evaluation Tests ==
		+	#Rupture Generator
		+	#SGT Calculation
		+	#Mesh Maker
		+	#Distance Calculation
		+	#Site-Rupture Set Determination
		+	#Ten moderate earthquake distributed around California
		+	#List of Sites
		+
		+	== References ==
		+
		+	== Related Entries ==
		+
		+	[[CyberShake]]
		+	[[CyberShake Workplan]]
		+
		+	== See Also ==
		+
		+	[[Main Page]]