Difference between revisions of "CSEP Workflows"

From SCECpedia
Jump to navigationJump to search
(Created page with "SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system. == CSEP Overview == * [http://hypocenter.usc.edu/r...")
 
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.
 
SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.
  
== CSEP Overview ==
+
== CSEP System Design Considerations ==
 +
# Maintain processing consistent with 2007-2017.
 +
# Distribute processing across multiple servers
 +
# Support deployment onto cloud servers
 +
# Improve process scheduling removing dependence on cron for scheduling
 +
# Add automatic re-try for recoverable processing errors
 +
# Implement Data transfers without using rsync
 +
# Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
 +
# Document/remove use of SVN for older catalogs.
 +
# Support prospective and retrospective processing
 +
# Support external testing centers (NZ, Europe,…) as we currently do.
 +
# Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
 +
# Develop new system minimizing the amount of existing data required by new system.
 +
# Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
 +
# Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
 +
# Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
 +
# Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
 +
# Review each forecast under test, and confirm each is needed before running in a new system.
 +
 
 +
== Draft CSEP System Notes and Plans ==
 +
* [http://hypocenter.usc.edu/research/CSEP/CSEP_Review_1May2017.pptx CSEP System Status Review - May 2017]
 +
 
 +
== Early CSEP Design and Architecture Presentations ==
 
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description]
 
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_System_Design_Principles_Overview_111707.pdf CSEP System Design Description]
 
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description]
 
* [http://hypocenter.usc.edu/research/CSEPfiles/CSEP_Software_Arch_Overview_111707.pdf CSEP Software Design Description]
  
 
== Related Entries ==
 
== Related Entries ==
* [[CSEP]]
+
*[[CSEP]]
* [http://cseptesting.org CSEP Home Page]
+
*[http://cseptesting.org CSEP Home Page]
 
*[[CME Project]]
 
*[[CME Project]]
*[[Main_page]]
 

Latest revision as of 19:38, 10 August 2017

SCEC CSEP processing system has a single server limitation. There are several new features needed for the next CSEP system.

CSEP System Design Considerations

  1. Maintain processing consistent with 2007-2017.
  2. Distribute processing across multiple servers
  3. Support deployment onto cloud servers
  4. Improve process scheduling removing dependence on cron for scheduling
  5. Add automatic re-try for recoverable processing errors
  6. Implement Data transfers without using rsync
  7. Identify common datasets, and maximize re-use of catalog data to reduce unnecessary data retrievals
  8. Document/remove use of SVN for older catalogs.
  9. Support prospective and retrospective processing
  10. Support external testing centers (NZ, Europe,…) as we currently do.
  11. Preserve existing data archives as static data set. Review for completeness, and re-process any expected, but missing forecasts and evaluations.
  12. Develop new system minimizing the amount of existing data required by new system.
  13. Improved processing status reporting, identifying processing failure, ensuring all required data products are produced and are valid.
  14. Improved resolution on system problem reports, identifying not only the forecast group, but identifying the specific forecast, or evaluation, or processing (e.g. data transfer) that has problems.
  15. Explicit definition of dependencies between processing dispatcher scripts, forecast groups, and forecasts.
  16. Improved availability (discovery) and access (distribution) of data products including forecast groups, forecasts, evaluations, catalogs, logs
  17. Review each forecast under test, and confirm each is needed before running in a new system.

Draft CSEP System Notes and Plans

Early CSEP Design and Architecture Presentations

Related Entries