Difference between revisions of "CSEP Workflows"
From SCECpedia
Jump to navigationJump to search| Line 1: | Line 1: | ||
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. | SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. | ||
| + | |||
| + | == CSEP System Design Considerations == | ||
| + | # Maintain processing consistent with 2007-2017. | ||
| + | # Distribute processing across multiple servers | ||
| + | # Support deployment onto cloud servers | ||
| + | # Improve process scheduling removing dependence on cron for scheduling | ||
| + | # Add automatic re-try for recoverable processing errors | ||
| + | # Implement Data transfers without using rsync | ||
| + | # Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals | ||
| + | # Document/remove use of SVN for older catalogs. | ||
| + | # Support prospective and retrospective processing | ||
| + | # Support external testing centers (NZ, Europe,…) as we currently do. | ||
| + | # Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations. | ||
| + | # Develop new system minimizing the amount of existing data required by new system. | ||
| + | # Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid. | ||
| + | # Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems. | ||
| + | # Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts. | ||
| + | # Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs | ||
| + | # Review each forecast under test, and confirm each is needed before running in a new system. | ||
| + | |||
| + | |||
| + | |||
== CSEP Overview == | == CSEP Overview == | ||
Revision as of 04:48, 11 May 2017
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.
CSEP System Design Considerations
- Maintain processing consistent with 2007-2017.
- Distribute processing across multiple servers
- Support deployment onto cloud servers
- Improve process scheduling removing dependence on cron for scheduling
- Add automatic re-try for recoverable processing errors
- Implement Data transfers without using rsync
- Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
- Document/remove use of SVN for older catalogs.
- Support prospective and retrospective processing
- Support external testing centers (NZ, Europe,…) as we currently do.
- Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
- Develop new system minimizing the amount of existing data required by new system.
- Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
- Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
- Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
- Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
- Review each forecast under test, and confirm each is needed before running in a new system.
