CSEP Workflows

SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.

CSEP System Design Considerations

1. Maintain processing consistent with 2007-2017.
2. Distribute processing across multiple servers
3. Support deployment onto cloud servers
4. Improve process scheduling removing dependence on cron for scheduling
5. Add automatic re-try for recoverable processing errors
6. Implement Data transfers without using rsync
7. Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
8. Document/remove use of SVN for older catalogs.
9. Support prospective and retrospective processing
10. Support external testing centers (NZ, Europe,…) as we currently do.
11. Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
12. Develop new system minimizing the amount of existing data required by new system.
13. Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
14. Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
15. Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
16. Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
17. Review each forecast under test, and confirm each is needed before running in a new system.

CSEP Overview

• CSEP System Design Description
• CSEP Software Design Description

Related Entries

• CSEP
• CSEP Home Page
• CME Project