Difference between revisions of "Moment-carc configuration"

From SCECpedia
Jump to navigationJump to search
 
(13 intermediate revisions by the same user not shown)
Line 6: Line 6:
  
 
*Relational database to host CyberShake input data, run status, run metadata, and store commonly used intensity measures and hazard curves.
 
*Relational database to host CyberShake input data, run status, run metadata, and store commonly used intensity measures and hazard curves.
 +
*Different stages of the workflow are able to access this database, including at workflow creation, during some jobs, and data product creation.
  
 
== Hardware ==
 
== Hardware ==
  
−
*2 TB of local storage for the database (SSD, if possible).  Current database size on moment is ~1.5 TB, but some of that should be migrated to focal, as the intent is that focal serves as the access point for completed studies.
+
*2 TB of local storage for the database (SSD, if possible).
−
*At least 64 GB RAM (current moment has 128 GB)
+
*Current system has 64 GB RAM
  
 
== Software (installed by CARC) ==
 
== Software (installed by CARC) ==
  
−
*Latest production version of MariaDB (10.6.7 as of 4/27/22)
+
*Latest production version of MariaDB (11.1.2 as of 10/18/23)
  
 
== Software (installed by SCEC) ==
 
== Software (installed by SCEC) ==
Line 24: Line 25:
 
<pre>
 
<pre>
 
innodb_file_per_table = 1
 
innodb_file_per_table = 1
−
innodb_buffer_pool_size = 112G (should be 16 GB less than the available system memory)
+
innodb_buffer_pool_size = 48G (should be 16 GB less than the available system memory)
 
innodb_buffer_pool_instances = 8
 
innodb_buffer_pool_instances = 8
 
query_cache_type = 0
 
query_cache_type = 0
Line 32: Line 33:
 
</pre>
 
</pre>
  
−
In terms of users, The functionality we want is read-only access for the user 'cybershk_ro' who can authenticate from anywhere, and read/write access for the user 'cybershk', from anywhere.  In our current setup, entries for these users in the USER_PRIVILEGES table are:
+
In terms of users, The functionality we want is read-only access for the user 'cybershk_ro', and read/write access for the user 'cybershk'.  In our current setup, entries for these users in the USER_PRIVILEGES table are:
  
 
{|
 
{|
Line 109: Line 110:
  
 
To support access to the MariaDB server, we would like to enable access to the MySQL port (3306 by default) for external users.
 
To support access to the MariaDB server, we would like to enable access to the MySQL port (3306 by default) for external users.
 +
 +
We would like the following systems to be whitelisted:
 +
*USC VPN
 +
*opensha.usc.edu
 +
*Summit compute nodes
  
 
For SSH access, we would like support for logging in from endeavour.
 
For SSH access, we would like support for logging in from endeavour.
 +
 +
== Initial database setup ==
 +
 +
We will copy the following data from the database at moment.usc.edu to moment-carc:
 +
 +
{| border="1" cellpadding="3"
 +
! Table !! Data !! Size (index + data)
 +
|-
 +
| AR_Hazard_Curve_Points || all || 41.5 GB
 +
|-
 +
| AR_Hazard_Curves || all || 1.3 GB
 +
|-
 +
| AR_Hazard_Datasets || all || <1 MB
 +
|-
 +
| Atten_Rel_Metadata || all || <1 MB
 +
|-
 +
| Atten_Rels || all || <1 MB
 +
|-
 +
| CyberShake_Runs || all || <1 GB
 +
|-
 +
| CyberShake_Site_Regions || all || <1 GB
 +
|-
 +
| CyberShake_Site_Ruptures || all || 9.6 GB
 +
|-
 +
| CyberShake_Site_Types || all || <1 MB
 +
|-
 +
| CyberShake_Sites || all || <1 MB
 +
|-
 +
| ERF_IDs || all || <1 MB
 +
|-
 +
| ERF_Metadata || all || <1 MB
 +
|-
 +
| ERF_Probability_Models || all || <1 MB
 +
|-
 +
| Hazard_Curve_Points || all || <1 GB
 +
|-
 +
| Hazard_Curves || all || <1 GB
 +
|-
 +
| Hazard_Curves_old || none
 +
|-
 +
| Hazard_Datasets || all || <1 MB
 +
|-
 +
| IM_Types || all || <1 MB
 +
|-
 +
| Mesh_Vsitop_Metadata || all || <1 MB
 +
|-
 +
| PeakAmplitudes
 +
|-
 +
| Points || none
 +
|-
 +
| Rup_Var_Seeds || all || <1 GB
 +
|-
 +
| Rupture_Variation_Probability_Modifier || all || <1 MB
 +
|-
 +
| Rupture_Variation_Scenario_IDs || all || <1 MB
 +
|-
 +
| Rupture_Variation_Scenario_Metadata || all || <1 MB
 +
|-
 +
| Rupture_Variations || all || 2.4 GB
 +
|-
 +
| Ruptures || all || 2.2 GB
 +
|-
 +
| SGT_Variation_IDs || all || <1 MB
 +
|-
 +
| SGT_Variation_Metadata || all || <1 MB
 +
|-
 +
| Studies || all || <1 MB
 +
|-
 +
| Time_Spans || all || <1 MB
 +
|-
 +
| Velocity_Model_Metadata || all || <1 MB
 +
|-
 +
| Velocity_Models || all || <1 MB
 +
|-
 +
| Vref_eff_Metadata || all || <1 MB
 +
|}

Latest revision as of 05:02, 3 May 2024

This page documents the configuration requirements for moment-carc, the CyberShake production database hosted by CARC.

Functionality

moment-carc should support the following:

  • Relational database to host CyberShake input data, run status, run metadata, and store commonly used intensity measures and hazard curves.
  • Different stages of the workflow are able to access this database, including at workflow creation, during some jobs, and data product creation.

Hardware

  • 2 TB of local storage for the database (SSD, if possible).
  • Current system has 64 GB RAM

Software (installed by CARC)

  • Latest production version of MariaDB (11.1.2 as of 10/18/23)

Software (installed by SCEC)

Database configuration

We would like to use the following settings in the mariadb-server.cnf file: 

innodb_file_per_table = 1
innodb_buffer_pool_size = 48G (should be 16 GB less than the available system memory)
innodb_buffer_pool_instances = 8
query_cache_type = 0
tmp_table_size = 32M
max_heap_table_size = 32M
innodb_log_file_size = 2G

In terms of users, The functionality we want is read-only access for the user 'cybershk_ro', and read/write access for the user 'cybershk'. In our current setup, entries for these users in the USER_PRIVILEGES table are:

GRANTEE TABLE_CATALOG PRIVILEGE_TYPE IS_GRANTABLE
'cybershk'@'localhost' def INDEX NO
'cybershk'@'localhost' def CREATE TABLESPACE NO
'cybershk'@'localhost' def ALTER NO
'cybershk'@'localhost' def DELETE HISTORY NO
'cybershk'@'localhost' def SHOW DATABASES NO
'cybershk'@'localhost' def SUPER NO
'cybershk'@'localhost' def CREATE TEMPORARY TABLES NO
'cybershk'@'localhost' def LOCK TABLES NO
'cybershk'@'localhost' def SELECT NO
'cybershk'@'localhost' def EXECUTE NO
'cybershk'@'localhost' def INSERT NO
'cybershk'@'localhost' def REPLICATION SLAVE NO
'cybershk'@'localhost' def UPDATE NO
'cybershk'@'localhost' def REPLICATION CLIENT NO
'cybershk'@'localhost' def DELETE NO
'cybershk'@'localhost' def CREATE VIEW NO
'cybershk'@'localhost' def CREATE NO
'cybershk'@'localhost' def SHOW VIEW NO
'cybershk'@'localhost' def DROP NO
'cybershk'@'localhost' def CREATE ROUTINE NO
'cybershk'@'localhost' def RELOAD NO
'cybershk'@'localhost' def ALTER ROUTINE NO
'cybershk'@'localhost' def SHUTDOWN NO
'cybershk'@'localhost' def CREATE USER NO
'cybershk'@'localhost' def PROCESS NO
'cybershk'@'localhost' def EVENT NO
'cybershk'@'localhost' def FILE NO
'cybershk'@'localhost' def TRIGGER NO
'cybershk'@'localhost' def REFERENCES NO
'cybershk'@'%.usc.edu' def USAGE NO
'cybershk'@'%' def SHOW DATABASES NO
'cybershk'@'%' def DELETE HISTORY NO
'cybershk_ro'@'localhost' def USAGE NO
'cybershk_ro'@'%' def USAGE NO

Network configuration

To support access to the MariaDB server, we would like to enable access to the MySQL port (3306 by default) for external users.

We would like the following systems to be whitelisted:

  • USC VPN
  • opensha.usc.edu
  • Summit compute nodes

For SSH access, we would like support for logging in from endeavour.

Initial database setup

We will copy the following data from the database at moment.usc.edu to moment-carc:

Table Data Size (index + data)
AR_Hazard_Curve_Points all 41.5 GB
AR_Hazard_Curves all 1.3 GB
AR_Hazard_Datasets all <1 MB
Atten_Rel_Metadata all <1 MB
Atten_Rels all <1 MB
CyberShake_Runs all <1 GB
CyberShake_Site_Regions all <1 GB
CyberShake_Site_Ruptures all 9.6 GB
CyberShake_Site_Types all <1 MB
CyberShake_Sites all <1 MB
ERF_IDs all <1 MB
ERF_Metadata all <1 MB
ERF_Probability_Models all <1 MB
Hazard_Curve_Points all <1 GB
Hazard_Curves all <1 GB
Hazard_Curves_old none
Hazard_Datasets all <1 MB
IM_Types all <1 MB
Mesh_Vsitop_Metadata all <1 MB
PeakAmplitudes
Points none
Rup_Var_Seeds all <1 GB
Rupture_Variation_Probability_Modifier all <1 MB
Rupture_Variation_Scenario_IDs all <1 MB
Rupture_Variation_Scenario_Metadata all <1 MB
Rupture_Variations all 2.4 GB
Ruptures all 2.2 GB
SGT_Variation_IDs all <1 MB
SGT_Variation_Metadata all <1 MB
Studies all <1 MB
Time_Spans all <1 MB
Velocity_Model_Metadata all <1 MB
Velocity_Models all <1 MB
Vref_eff_Metadata all <1 MB
Retrieved from "https://strike.scec.org/scecwiki/index.php?title=Moment-carc_configuration&oldid=28599"