CSEP Workflows

SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.

CSEP System Design Considerations

Maintain processing consistent with 2007-2017.
Distribute processing across multiple servers
Support deployment onto cloud servers
Improve process scheduling removing dependence on cron for scheduling
Add automatic re-try for recoverable processing errors
Implement Data transfers without using rsync
Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
Document/remove use of SVN for older catalogs.
Support prospective and retrospective processing
Support external testing centers (NZ, Europe,…) as we currently do.
Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
Develop new system minimizing the amount of existing data required by new system.
Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
Review each forecast under test, and confirm each is needed before running in a new system.