Difference between revisions of "CSEP Workflows"
From SCECpedia
Jump to navigationJump to search| Line 20: | Line 20: | ||
# Review each forecast under test, and confirm each is needed before running in a new system. | # Review each forecast under test, and confirm each is needed before running in a new system. | ||
| − | == CSEP | + | == Draft CSEP System Notes and Plans == |
| + | * [http://hypocenter.usc.edu/research/CSEP/CSEP_Review_1May2017.pptx CSEP System Status Review - May 2017] | ||
| + | |||
| + | == Early CSEP Design and Architecture Presentations == | ||
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description] | * [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description] | ||
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description] | * [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description] | ||
Latest revision as of 19:38, 10 August 2017
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.
Contents
CSEP System Design Considerations
- Maintain processing consistent with 2007-2017.
- Distribute processing across multiple servers
- Support deployment onto cloud servers
- Improve process scheduling removing dependence on cron for scheduling
- Add automatic re-try for recoverable processing errors
- Implement Data transfers without using rsync
- Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
- Document/remove use of SVN for older catalogs.
- Support prospective and retrospective processing
- Support external testing centers (NZ, Europe,…) as we currently do.
- Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
- Develop new system minimizing the amount of existing data required by new system.
- Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
- Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
- Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
- Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
- Review each forecast under test, and confirm each is needed before running in a new system.
