CyberShake Workflow Framework
From SCECpedia
Jump to navigationJump to searchThis page provides documentation on the workflow framework that we use to execute CyberShake.
As of 2018, we use shock.usc.edu as our workflow submit host for CyberShake, but this setup could be replicated on any machine
Contents
Overview
The workflow framework consists of three phases:
- Create. In this phase, we use the Pegasus API to create an "abstract description" of the workflow, so named because it does not have the specific paths or runtime configuration information for any system. It is a general description of the tasks to execute in the workflow, the input and output files, and the dependencies between them. This description is called a DAX and is in XML format.
- Plan. In this phase, we use the Pegasus planner to take our abstract description and convert it into a "concrete description" for execution on one or more specific computational resources. At this stage, paths to executables and specific configuration information, such as the correct certificate to use, what account to charge, etc. are added into the workflow. This description is called a DAG and consists of multiple files in a format expected by HTCondor.
- Run. In this phase, we use Pegasus to hand our DAG off to HTCondor, which supervises the execution of the workflow. HTCondor maintains a queue of jobs. Jobs with all their dependencies satisfied are eligible to run, and are sent to their execution system and checked on completion for success or failure.
