Difference between revisions of "GetFile"

From SCECpedia
Jump to navigationJump to search
(Rewrite GetFile entry in terms of OpenSHA problem)
(list of all supported hazard models and GetFile server endpoints)
 
(2 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
== Proposed Solution ==
 
== Proposed Solution ==
 
GetFile is a more robust solution to hosting hazard models for use in OpenSHA. It will be used to download and validate the UCERF3 and NSHM23 models. It may see use in other models and several projects across SCEC that need to download and validate files, such as UCVM. Scientific models can be stored on USC CARC and downloaded via the GetFile framework. GetFile provides a sophisticated feature-set for data validation, rolling back to older model snapshots, and enabling automatic updates of the GetFile framework for seamless deployment of new features and bug fixes.
 
GetFile is a more robust solution to hosting hazard models for use in OpenSHA. It will be used to download and validate the UCERF3 and NSHM23 models. It may see use in other models and several projects across SCEC that need to download and validate files, such as UCVM. Scientific models can be stored on USC CARC and downloaded via the GetFile framework. GetFile provides a sophisticated feature-set for data validation, rolling back to older model snapshots, and enabling automatic updates of the GetFile framework for seamless deployment of new features and bug fixes.
 +
 +
== Supported Models ==
 +
OpenSHA hazard models are available for download using GetFile to try in sequential order:
 +
# CARC /project2/scec_608/public/getfile and
 +
# CARC /project/scec_608/public/getfile.
 +
See links to access files via the Globus Connect Service at [[Hypocenter_Replacement]].
 +
Note that /project is deprecated and may not be available soon. The first location GetFile is able to make a connection and validate metadata with MD5 checksums will be used for the duration of the GetFile lifetime for all subsequent downloads.
 +
 +
The following list of ERF models are supported as of writing (Nov 16 2025) in the OpenSHA v25.12.0-alpha with GetFile v25.11.0:
 +
* branch_avgs_combined (v24.11.0) 
 +
* cached_dep100.0_depMean_rakeMean (v24.11.0) 
 +
* cached_FM3_1_dep100.0_depMean_rakeMean (v24.11.0) 
 +
* cached_FM3_2_dep100.0_depMean_rakeMean (v24.11.0) 
 +
* FM3_1_ABM_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_1_branch_averaged_full_modules (v24.11.0) 
 +
* FM3_1_branch_averaged_with_logic_tree (v24.11.0) 
 +
* FM3_1_branch_averaged (v24.11.0) 
 +
* FM3_1_GEOL_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_1_mean_ucerf3_sol (v24.11.0) 
 +
* FM3_1_NEOK_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_1_SpatSeisU2_branch_averaged_full_modules (v24.11.0) 
 +
* FM3_1_SpatSeisU2_branch_averaged (v24.11.0) 
 +
* FM3_1_SpatSeisU3_branch_averaged_full_modules (v24.11.0) 
 +
* FM3_1_SpatSeisU3_branch_averaged (v24.11.0) 
 +
* FM3_1_ZENGBB_Shaw09Mod_DsrTap_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_1_ZENGBB_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_2_ABM_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_2_branch_averaged_full_modules (v24.11.0) 
 +
* FM3_2_branch_averaged_with_logic_tree (v24.11.0) 
 +
* FM3_2_branch_averaged (v24.11.0) 
 +
* FM3_2_GEOL_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_2_mean_ucerf3_sol (v24.11.0) 
 +
* FM3_2_NEOK_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_2_SpatSeisU2_branch_averaged_full_modules (v24.11.0) 
 +
* FM3_2_SpatSeisU2_branch_averaged (v24.11.0) 
 +
* FM3_2_SpatSeisU3_branch_averaged_full_modules (v24.11.0) 
 +
* FM3_2_SpatSeisU3_branch_averaged (v24.11.0) 
 +
* FM3_2_ZENGBB_Shaw09Mod_DsrTap_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* FM3_2_ZENGBB_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0) 
 +
* full_logic_tree_with_gridded (v24.11.0) 
 +
* full_logic_tree (v24.11.0) 
 +
* mean_ucerf3_sol_with_mappings (v24.11.0) 
 +
* mean_ucerf3_sol (v24.11.0) 
 +
* rake_basis (v24.11.0) 
 +
* full_ucerf3_compound_sol (v24.12.0)
 +
* WUS_branch_averaged_gridded_simplified (v25.1.0)
 +
  
 
== Docs and Code ==
 
== Docs and Code ==
Line 12: Line 59:
  
 
Demo applications using the GetFile library: https://github.com/abhatthal/getfile-demo
 
Demo applications using the GetFile library: https://github.com/abhatthal/getfile-demo
 +
 +
Video demonstration of downloading files with GetFile in HazardCurveApplication. [[File:Getfile-opensha-demo.mp4]]
 +
 +
This video demo verifies the following behavior for file versions and network status.
 +
# no existing files / download multiple
 +
# already up to date / don’t download
 +
# no files + no wifi / fails with error message
 +
# no files + wifi outage / fails with error message
 +
# outdated files / update only outdated
 +
# outdated files + no wifi / silently use outdated files

Latest revision as of 17:50, 16 November 2025

OpenSHA Problem

Fault section data and fault system rupture sets are loaded by OpenSHA to compute earthquake rupture forecasts. The geospatial data for several models is stored directly with the OpenSHA code that operates on it. Models for seismic hazard analysis under the OpenSHA framework are becoming progressively larger. Unfortunately, there are file size constraints of 100MB on GitHub, which can't fit the new 2023 US National Seismic Hazard Model (NSHM23).

Current Solution

Smaller models can continue to be hosted on GitHub with the OpenSHA code, but UCERF3 has been moved to a server on USC campus. Currently, OpenSHA downloads the model from the ASB "cheesegrater" server. This solution is not scalable and has the potential to partially download or otherwise download a corrupted version of the UCERF3 model. These older servers are going to be decommissioned soon and we need to transition to a better long-term solution.

Proposed Solution

GetFile is a more robust solution to hosting hazard models for use in OpenSHA. It will be used to download and validate the UCERF3 and NSHM23 models. It may see use in other models and several projects across SCEC that need to download and validate files, such as UCVM. Scientific models can be stored on USC CARC and downloaded via the GetFile framework. GetFile provides a sophisticated feature-set for data validation, rolling back to older model snapshots, and enabling automatic updates of the GetFile framework for seamless deployment of new features and bug fixes.

Supported Models

OpenSHA hazard models are available for download using GetFile to try in sequential order:

  1. CARC /project2/scec_608/public/getfile and
  2. CARC /project/scec_608/public/getfile.

See links to access files via the Globus Connect Service at Hypocenter_Replacement. Note that /project is deprecated and may not be available soon. The first location GetFile is able to make a connection and validate metadata with MD5 checksums will be used for the duration of the GetFile lifetime for all subsequent downloads.

The following list of ERF models are supported as of writing (Nov 16 2025) in the OpenSHA v25.12.0-alpha with GetFile v25.11.0:

  • branch_avgs_combined (v24.11.0)
  • cached_dep100.0_depMean_rakeMean (v24.11.0)
  • cached_FM3_1_dep100.0_depMean_rakeMean (v24.11.0)
  • cached_FM3_2_dep100.0_depMean_rakeMean (v24.11.0)
  • FM3_1_ABM_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_1_branch_averaged_full_modules (v24.11.0)
  • FM3_1_branch_averaged_with_logic_tree (v24.11.0)
  • FM3_1_branch_averaged (v24.11.0)
  • FM3_1_GEOL_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_1_mean_ucerf3_sol (v24.11.0)
  • FM3_1_NEOK_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_1_SpatSeisU2_branch_averaged_full_modules (v24.11.0)
  • FM3_1_SpatSeisU2_branch_averaged (v24.11.0)
  • FM3_1_SpatSeisU3_branch_averaged_full_modules (v24.11.0)
  • FM3_1_SpatSeisU3_branch_averaged (v24.11.0)
  • FM3_1_ZENGBB_Shaw09Mod_DsrTap_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_1_ZENGBB_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_2_ABM_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_2_branch_averaged_full_modules (v24.11.0)
  • FM3_2_branch_averaged_with_logic_tree (v24.11.0)
  • FM3_2_branch_averaged (v24.11.0)
  • FM3_2_GEOL_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_2_mean_ucerf3_sol (v24.11.0)
  • FM3_2_NEOK_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_2_SpatSeisU2_branch_averaged_full_modules (v24.11.0)
  • FM3_2_SpatSeisU2_branch_averaged (v24.11.0)
  • FM3_2_SpatSeisU3_branch_averaged_full_modules (v24.11.0)
  • FM3_2_SpatSeisU3_branch_averaged (v24.11.0)
  • FM3_2_ZENGBB_Shaw09Mod_DsrTap_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • FM3_2_ZENGBB_Shaw09Mod_DsrUni_CharConst_M5Rate7.9_MMaxOff7.6_NoFix_SpatSeisU3 (v24.11.0)
  • full_logic_tree_with_gridded (v24.11.0)
  • full_logic_tree (v24.11.0)
  • mean_ucerf3_sol_with_mappings (v24.11.0)
  • mean_ucerf3_sol (v24.11.0)
  • rake_basis (v24.11.0)
  • full_ucerf3_compound_sol (v24.12.0)
  • WUS_branch_averaged_gridded_simplified (v25.1.0)


Docs and Code

The source code and detailed usage and setup documentation: https://github.com/abhatthal/getfile

Demo applications using the GetFile library: https://github.com/abhatthal/getfile-demo

Video demonstration of downloading files with GetFile in HazardCurveApplication. File:Getfile-opensha-demo.mp4

This video demo verifies the following behavior for file versions and network status.

  1. no existing files / download multiple
  2. already up to date / don’t download
  3. no files + no wifi / fails with error message
  4. no files + wifi outage / fails with error message
  5. outdated files / update only outdated
  6. outdated files + no wifi / silently use outdated files