Difference between revisions of "Moment-carc configuration"
From SCECpedia
Jump to navigationJump to search| Line 125: | Line 125: | ||
! Table !! Data !! Size (index + data) | ! Table !! Data !! Size (index + data) | ||
|- | |- | ||
| − | + | | AR_Hazard_Curve_Points || all || 41.5 GB | |
|- | |- | ||
| − | + | | AR_Hazard_Curves || all || 1.3 GB | |
|- | |- | ||
| − | + | | AR_Hazard_Datasets || all || <1 MB | |
|- | |- | ||
| − | + | | Atten_Rel_Metadata || all || <1 MB | |
|- | |- | ||
| − | + | | Atten_Rels || all || <1 MB | |
|- | |- | ||
| − | + | | CyberShake_Runs || all || <1 GB | |
|- | |- | ||
| − | + | | CyberShake_Site_Regions || all || <1 GB | |
|- | |- | ||
| − | + | | CyberShake_Site_Ruptures || all || 9.6 GB | |
|- | |- | ||
| − | + | | CyberShake_Site_Types || all || <1 MB | |
|- | |- | ||
| − | + | | CyberShake_Sites || all || <1 MB | |
|- | |- | ||
| − | + | | ERF_IDs || all || <1 MB | |
|- | |- | ||
| − | + | | ERF_Metadata || all || <1 MB | |
|- | |- | ||
| − | + | | ERF_Probability_Models || all || <1 MB | |
|- | |- | ||
| − | + | | Hazard_Curve_Points || all || <1 GB | |
|- | |- | ||
| − | + | | Hazard_Curves || all || <1 GB | |
|- | |- | ||
| − | + | | Hazard_Curves_old || none | |
|- | |- | ||
| − | + | | Hazard_Datasets || all || <1 MB | |
|- | |- | ||
| − | + | | IM_Types || all || <1 MB | |
|- | |- | ||
| − | + | | Mesh_Vsitop_Metadata || all || <1 MB | |
|- | |- | ||
| − | + | | PeakAmplitudes | |
|- | |- | ||
| − | + | | Points || none | |
|- | |- | ||
| − | + | | Rup_Var_Seeds || all || <1 GB | |
|- | |- | ||
| − | + | | Rupture_Variation_Probability_Modifier || all || <1 MB | |
|- | |- | ||
| − | + | | Rupture_Variation_Scenario_IDs || all || <1 MB | |
|- | |- | ||
| − | + | | Rupture_Variation_Scenario_Metadata || all || <1 MB | |
|- | |- | ||
| − | + | | Rupture_Variations || all || 2.4 GB | |
|- | |- | ||
| − | + | | Ruptures || all || 2.2 GB | |
|- | |- | ||
| − | + | | SGT_Variation_IDs || all || <1 MB | |
|- | |- | ||
| − | + | | SGT_Variation_Metadata || all || <1 MB | |
|- | |- | ||
| − | + | | Studies || all || <1 MB | |
|- | |- | ||
| − | + | | Time_Spans || all || <1 MB | |
|- | |- | ||
| − | + | | Velocity_Model_Metadata || all || <1 MB | |
|- | |- | ||
| − | + | | Velocity_Models || all || <1 MB | |
|- | |- | ||
| − | + | | Vref_eff_Metadata || all || <1 MB | |
|} | |} | ||
Latest revision as of 05:02, 3 May 2024
This page documents the configuration requirements for moment-carc, the CyberShake production database hosted by CARC.
Contents
Functionality
moment-carc should support the following:
- Relational database to host CyberShake input data, run status, run metadata, and store commonly used intensity measures and hazard curves.
- Different stages of the workflow are able to access this database, including at workflow creation, during some jobs, and data product creation.
Hardware
- 2 TB of local storage for the database (SSD, if possible).
- Current system has 64 GB RAM
Software (installed by CARC)
- Latest production version of MariaDB (11.1.2 as of 10/18/23)
Software (installed by SCEC)
Database configuration
We would like to use the following settings in the mariadb-server.cnf file:
innodb_file_per_table = 1 innodb_buffer_pool_size = 48G (should be 16 GB less than the available system memory) innodb_buffer_pool_instances = 8 query_cache_type = 0 tmp_table_size = 32M max_heap_table_size = 32M innodb_log_file_size = 2G
In terms of users, The functionality we want is read-only access for the user 'cybershk_ro', and read/write access for the user 'cybershk'. In our current setup, entries for these users in the USER_PRIVILEGES table are:
| GRANTEE | TABLE_CATALOG | PRIVILEGE_TYPE | IS_GRANTABLE | |
|---|---|---|---|---|
| 'cybershk'@'localhost' | def | INDEX | NO | |
| 'cybershk'@'localhost' | def | CREATE TABLESPACE | NO | |
| 'cybershk'@'localhost' | def | ALTER | NO | |
| 'cybershk'@'localhost' | def | DELETE HISTORY | NO | |
| 'cybershk'@'localhost' | def | SHOW DATABASES | NO | |
| 'cybershk'@'localhost' | def | SUPER | NO | |
| 'cybershk'@'localhost' | def | CREATE TEMPORARY TABLES | NO | |
| 'cybershk'@'localhost' | def | LOCK TABLES | NO | |
| 'cybershk'@'localhost' | def | SELECT | NO | |
| 'cybershk'@'localhost' | def | EXECUTE | NO | |
| 'cybershk'@'localhost' | def | INSERT | NO | |
| 'cybershk'@'localhost' | def | REPLICATION SLAVE | NO | |
| 'cybershk'@'localhost' | def | UPDATE | NO | |
| 'cybershk'@'localhost' | def | REPLICATION CLIENT | NO | |
| 'cybershk'@'localhost' | def | DELETE | NO | |
| 'cybershk'@'localhost' | def | CREATE VIEW | NO | |
| 'cybershk'@'localhost' | def | CREATE | NO | |
| 'cybershk'@'localhost' | def | SHOW VIEW | NO | |
| 'cybershk'@'localhost' | def | DROP | NO | |
| 'cybershk'@'localhost' | def | CREATE ROUTINE | NO | |
| 'cybershk'@'localhost' | def | RELOAD | NO | |
| 'cybershk'@'localhost' | def | ALTER ROUTINE | NO | |
| 'cybershk'@'localhost' | def | SHUTDOWN | NO | |
| 'cybershk'@'localhost' | def | CREATE USER | NO | |
| 'cybershk'@'localhost' | def | PROCESS | NO | |
| 'cybershk'@'localhost' | def | EVENT | NO | |
| 'cybershk'@'localhost' | def | FILE | NO | |
| 'cybershk'@'localhost' | def | TRIGGER | NO | |
| 'cybershk'@'localhost' | def | REFERENCES | NO | |
| 'cybershk'@'%.usc.edu' | def | USAGE | NO | |
| 'cybershk'@'%' | def | SHOW DATABASES | NO | |
| 'cybershk'@'%' | def | DELETE HISTORY | NO | |
| 'cybershk_ro'@'localhost' | def | USAGE | NO | |
| 'cybershk_ro'@'%' | def | USAGE | NO |
Network configuration
To support access to the MariaDB server, we would like to enable access to the MySQL port (3306 by default) for external users.
We would like the following systems to be whitelisted:
- USC VPN
- opensha.usc.edu
- Summit compute nodes
For SSH access, we would like support for logging in from endeavour.
Initial database setup
We will copy the following data from the database at moment.usc.edu to moment-carc:
| Table | Data | Size (index + data) |
|---|---|---|
| AR_Hazard_Curve_Points | all | 41.5 GB |
| AR_Hazard_Curves | all | 1.3 GB |
| AR_Hazard_Datasets | all | <1 MB |
| Atten_Rel_Metadata | all | <1 MB |
| Atten_Rels | all | <1 MB |
| CyberShake_Runs | all | <1 GB |
| CyberShake_Site_Regions | all | <1 GB |
| CyberShake_Site_Ruptures | all | 9.6 GB |
| CyberShake_Site_Types | all | <1 MB |
| CyberShake_Sites | all | <1 MB |
| ERF_IDs | all | <1 MB |
| ERF_Metadata | all | <1 MB |
| ERF_Probability_Models | all | <1 MB |
| Hazard_Curve_Points | all | <1 GB |
| Hazard_Curves | all | <1 GB |
| Hazard_Curves_old | none | |
| Hazard_Datasets | all | <1 MB |
| IM_Types | all | <1 MB |
| Mesh_Vsitop_Metadata | all | <1 MB |
| PeakAmplitudes | ||
| Points | none | |
| Rup_Var_Seeds | all | <1 GB |
| Rupture_Variation_Probability_Modifier | all | <1 MB |
| Rupture_Variation_Scenario_IDs | all | <1 MB |
| Rupture_Variation_Scenario_Metadata | all | <1 MB |
| Rupture_Variations | all | 2.4 GB |
| Ruptures | all | 2.2 GB |
| SGT_Variation_IDs | all | <1 MB |
| SGT_Variation_Metadata | all | <1 MB |
| Studies | all | <1 MB |
| Time_Spans | all | <1 MB |
| Velocity_Model_Metadata | all | <1 MB |
| Velocity_Models | all | <1 MB |
| Vref_eff_Metadata | all | <1 MB |
