Difference between revisions of "CSEP Workflows"
From SCECpedia
Jump to navigationJump to search| (2 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. | SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. | ||
| − | == CSEP | + | == CSEP System Design Considerations == |
| + | # Maintain processing consistent with 2007-2017. | ||
| + | # Distribute processing across multiple servers | ||
| + | # Support deployment onto cloud servers | ||
| + | # Improve process scheduling removing dependence on cron for scheduling | ||
| + | # Add automatic re-try for recoverable processing errors | ||
| + | # Implement Data transfers without using rsync | ||
| + | # Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals | ||
| + | # Document/remove use of SVN for older catalogs. | ||
| + | # Support prospective and retrospective processing | ||
| + | # Support external testing centers (NZ, Europe,…) as we currently do. | ||
| + | # Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations. | ||
| + | # Develop new system minimizing the amount of existing data required by new system. | ||
| + | # Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid. | ||
| + | # Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems. | ||
| + | # Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts. | ||
| + | # Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs | ||
| + | # Review each forecast under test, and confirm each is needed before running in a new system. | ||
| + | |||
| + | == Draft CSEP System Notes and Plans == | ||
| + | * [http://hypocenter.usc.edu/research/CSEP/CSEP_Review_1May2017.pptx CSEP System Status Review - May 2017] | ||
| + | |||
| + | == Early CSEP Design and Architecture Presentations == | ||
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description] | * [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description] | ||
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description] | * [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description] | ||
Latest revision as of 19:38, 10 August 2017
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.
Contents
CSEP System Design Considerations
- Maintain processing consistent with 2007-2017.
- Distribute processing across multiple servers
- Support deployment onto cloud servers
- Improve process scheduling removing dependence on cron for scheduling
- Add automatic re-try for recoverable processing errors
- Implement Data transfers without using rsync
- Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
- Document/remove use of SVN for older catalogs.
- Support prospective and retrospective processing
- Support external testing centers (NZ, Europe,…) as we currently do.
- Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
- Develop new system minimizing the amount of existing data required by new system.
- Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
- Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
- Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
- Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
- Review each forecast under test, and confirm each is needed before running in a new system.
