CSEP Workflows: Difference between revisions
From SCECpedia
Jump to navigationJump to search
Created page with "SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. == CSEP Overview == * [http://hypocenter.usc.edu/r..." |
No edit summary |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. | SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. | ||
== CSEP | == CSEP System Design Considerations == | ||
# Maintain processing consistent with 2007-2017. | |||
# Distribute processing across multiple servers | |||
# Support deployment onto cloud servers | |||
# Improve process scheduling removing dependence on cron for scheduling | |||
# Add automatic re-try for recoverable processing errors | |||
# Implement Data transfers without using rsync | |||
# Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals | |||
# Document/remove use of SVN for older catalogs. | |||
# Support prospective and retrospective processing | |||
# Support external testing centers (NZ, Europe,…) as we currently do. | |||
# Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations. | |||
# Develop new system minimizing the amount of existing data required by new system. | |||
# Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid. | |||
# Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems. | |||
# Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts. | |||
# Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs | |||
# Review each forecast under test, and confirm each is needed before running in a new system. | |||
== Draft CSEP System Notes and Plans == | |||
* [http://hypocenter.usc.edu/research/CSEP/CSEP_Review_1May2017.pptx CSEP System Status Review - May 2017] | |||
== Early CSEP Design and Architecture Presentations == | |||
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description] | * [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description] | ||
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description] | * [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description] | ||
== Related Entries == | == Related Entries == | ||
* [[CSEP]] | *[[CSEP]] | ||
* [http://cseptesting.org CSEP Home Page] | *[http://cseptesting.org CSEP Home Page] | ||
*[[CME Project]] | *[[CME Project]] | ||
Latest revision as of 19:38, 10 August 2017
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.
CSEP System Design Considerations
- Maintain processing consistent with 2007-2017.
- Distribute processing across multiple servers
- Support deployment onto cloud servers
- Improve process scheduling removing dependence on cron for scheduling
- Add automatic re-try for recoverable processing errors
- Implement Data transfers without using rsync
- Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
- Document/remove use of SVN for older catalogs.
- Support prospective and retrospective processing
- Support external testing centers (NZ, Europe,…) as we currently do.
- Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
- Develop new system minimizing the amount of existing data required by new system.
- Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
- Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
- Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
- Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
- Review each forecast under test, and confirm each is needed before running in a new system.
