CyberShake Testing
Computational scale and complexity of the SCEC CyberShake system requires automated and repeatable system-level testing capabilities. The CyberShake testing must be capable of end-to-end testing, showing that all elements, inputs, earth models, computational codes, and data processing and reduction codes all work together.
The CyberShake Testing system combines a distributed workflow-based HPC software testing harness together with a database of reference problems and expected solutions.
Contents
CyberShake Testing Requirements
- Must support broad range of HPC codes including calculations too large to run on SCEC computers
- Must perform end-to-end calculations, capable of performing a series of calculations
- Must support multiple test evaluations including file-based comparisons, integer and floating-point with tolerance comparisons, and relational database entry comparisons.
- Must be modular, capable of testing performance of alternative codes on equivalent calculations
- Must support and help manage a test repository that contains reference problems and reference results
- Must support provisioning of, and job submission to, multiple HPC resource providers including USC HPCC, TeraGrid, and possibly DOE computer resources
- Must provide well-define metadata description of every test result that describes the code under test, the input parameters used, the reference results used in comparisons, and final test results
Required Evaluation Tests
- Rupture Generator
- SGT Calculation
- Mesh Maker
- Distance Calculation
- Site-Rupture Set Determination
- Ten moderate earthquake distributed around California
- List of Sites
CyberShake Test Harness
We required a workflow-based system capable of automating multiple CyberShake HPC calculations. This is modeled on the virtual data processing model of Pegasus.
CyberShake Test Oracle
We require a reference database that describes specific test problems, describes the input files, and output files, and defines a list of expected results.
References
- Maechling, P., Deelman, E., Cui, Y. (2009), Implementing Software Acceptance Tests as Scientific Workflows, Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications, Hamid R. Arabnia (Ed.): PDPTA 2009, Las Vegas, Nevada, USA, July 13-17, 2009, 2 Volumes. CSREA Press 2009, ISBN 1-60132-123-6, pp. 317-323
