Difference between revisions of "CSEP Gitlab"
| Line 2: | Line 2: | ||
== CSEP CI/CD System Overview Diagram == | == CSEP CI/CD System Overview Diagram == | ||
| − | [[Image:Csep gitlab roadmap.pdf| | + | [[Image:Csep gitlab roadmap.pdf|250px| "Gitlab Server and Runner"]] |
*[[Media:Csep gitlab roadmap.pptx | Gitlab Server Configuration as ppt file]] | *[[Media:Csep gitlab roadmap.pptx | Gitlab Server Configuration as ppt file]] | ||
*[[Media:Csep gitlab roadmap.pdf | Gitlab Server Configuration as pdf file]] | *[[Media:Csep gitlab roadmap.pdf | Gitlab Server Configuration as pdf file]] | ||
| − | |||
| − | |||
| − | |||
== Data Usage Estimates == | == Data Usage Estimates == | ||
| Line 41: | Line 38: | ||
*[https://aws.amazon.com/ec2/pricing/ EC2 Pricing] | *[https://aws.amazon.com/ec2/pricing/ EC2 Pricing] | ||
*[https://calculator.aws/ Total Cost Estimator) | *[https://calculator.aws/ Total Cost Estimator) | ||
| + | *[https://docs.gitlab.com/ee/install/aws/ AWS Gitlab Example Deployment] | ||
== Gitlab Basic System Requirements == | == Gitlab Basic System Requirements == | ||
Revision as of 21:46, 12 May 2020
SCEC's CSEP project is coordinating an earthquake and ground motion forecasting project with national and international partners. Project members want to use the Gitlab system to support the development of the CSEP open-source scientific software development community, the code development, distributed version control, and automated software testing environment. Specifications describing the expected system requirements and preliminary data creation and transfer estimates are below.
Contents
CSEP CI/CD System Overview Diagram
Data Usage Estimates
Estimates for data usages are based on combining the two use cases for the GitLab repository (1) Code storage and (2) Storage of Large data files and experiment results. These estimates are based on data from the current CSEP project, such as current project members, current storage in current and old repositories, and rough estimates of expected data usage for past and future experiments.
- amount generated internally by the system per year, most of which can be left on AWS storage: 5TB/year
- amount of data transferred into AWS: 1TB/year
- amount of data transferred out of AWS: 2.5TB/year
Data Estimates Breakdown
Project Info:
CSEP Project Members: 20
CSEP1 Repo Size (in Gb): 1.2
Storage per experiment (in Tb): 1
(1) Code Storage
Total data stored: 25 Gb (High-end estimate including models, codes and benchmark data sets)
Data transferred: [50-500] Gb
(2) Data/Catalog Storage
Total Storage: 250 Gb
Data transferred: [100 Gb - 1 Tb]
(3) Experiment Results
Total Storage: 4 Tb
Data transferred: [100 Gb - 1 Tb]
Overall Estimates:
Total stored: ~4.5 Tb
Transferred: [250 Gb - 2.5 Tb]
AWS Estimation Webpage
- EC2 Pricing
- [https://calculator.aws/ Total Cost Estimator)
- AWS Gitlab Example Deployment
Gitlab Basic System Requirements
- Recent Linux Distro (Ubuntu,Centos...)
- Generate SSH keys
- Configure SMTP Server
- 8GB RAM is the recommended minimum memory size for all installations and supports up to 100 users. 16GB RAM supports up to 500 users
- Databases: PostgreSQL
- Redis/Sidekiq: stores all user sessions and background task queue processes the background jobs with a multithreaded process
- Prometheus and it’s exporters
- Avoid installing GitLab Runner on the same machine where Gitlab is installed.
- GitLab needs JavaScript enabled in browsers to support features such as Issue Boards.
CSEP Computer and Storage Inventory
- CSEP_Computers
- CSEP_Hardware_Inventory
- Github Docs
- SCEC CSEP Storage Summaries:
- SCEC Storage Summary: Usage Report
- Sorted by Username: Sorted Usage Report
Overview and Gitlab Installation
- Gitlab integration software.
- Gitlab Software Options
- GitLab and Runner Installation Information
- GitLab Installation using Docker
- GitLab Runner Installation with Docker
