SCECpedia - User contributions [en]

CSEP Gitlab

2020-06-25T18:42:02Z

Wsavran: /* AWS Estimate */

SCEC's CSEP project is coordinating an earthquake and ground motion forecasting project with national and international partners. Project members want to use the Gitlab system to support the development of the CSEP open-source scientific software development community, the code development, distributed version control, and automated software testing environment. Specifications describing the expected system requirements and preliminary data creation and transfer estimates are below.

== CSEP CI/CD System Overview Diagram ==
[[Image:Csep gitlab roadmap.pdf|250px| "Gitlab Server and Runner"]]

*[[Media:Csep gitlab roadmap.pptx | Gitlab Server Configuration as ppt file]]
*[[Media:Csep gitlab roadmap.pdf | Gitlab Server Configuration as pdf file]]

== Data Usage Estimates ==
Estimates for data usages are based on combining the two use cases for the GitLab repository (1) Code storage and (2) Storage of Large data files and experiment results. These estimates are based on data from the current CSEP project, such as current project members, current storage in current and old repositories, and rough estimates of expected data usage for past and future experiments.
# amount generated internally by the system per year, most of which can be left on AWS storage: 5TB/year
# amount of data transferred into AWS: 1TB/year
# amount of data transferred out of AWS: 2.5TB/year

== Data Estimates Breakdown ==
Project Info: 
CSEP Project Members: 20 
CSEP1 Repo Size (in Gb): 1.2 
Storage per experiment (in Gb): 10 

(1) Code Storage
Total data stored: 25 Gb (High-end estimate including models, codes and benchmark data sets) 
Data transferred: [50-500] Gb 

(2) Data/Catalog Storage
Total Storage: 250 Gb 
Data transferred: [100 Gb - 1 Tb] 

(3) Experiment Results
Total Storage: 4 Tb 
Data transferred: [100 Gb - 1 Tb] 

Overall Estimates:
Total stored: ~4.5 Tb 
Transferred: [250 Gb - 2.5 Tb] 

== AWS Estimation Webpage ==
*[https://aws.amazon.com/ec2/pricing/ EC2 Pricing]
*[https://calculator.aws/ Total Cost Estimator]
*[https://docs.gitlab.com/ee/install/aws/ AWS Gitlab Example Deployment]

== Gitlab Basic System Requirements ==
* Recent Linux Distro (Ubuntu,Centos...)
* Generate SSH keys
* Configure SMTP Server
* 8GB RAM is the recommended minimum memory size for all installations and supports up to 100 users. 16GB RAM supports up to 500 users
* Databases: PostgreSQL
* Redis/Sidekiq: stores all user sessions and background task queue processes the background jobs with a multithreaded process
* Prometheus and it’s exporters
* Avoid installing GitLab Runner on the same machine where Gitlab is installed.
* GitLab needs JavaScript enabled in browsers to support features such as Issue Boards.

== CSEP Computer and Storage Inventory ==
*[[CSEP_Computers]]
*[[CSEP_Hardware_Inventory]]
*[[Media:Computer Hardware and Storage Systems · SCECcode CSEP Wiki.pdf | Github Docs]]
*SCEC CSEP Storage Summaries:
** SCEC Storage Summary: [[Media:Disk Usage Jan2020.txt | Usage Report]]
** Sorted by Username: [[Media:Sorted_Storage_Jan2020.txt | Sorted Usage Report]]

== Overview and Gitlab Installation ==
*[https://about.gitlab.com/ Gitlab] integration software.
*[https://about.gitlab.com/pricing/#self-managed Gitlab Software Options]
*[https://about.gitlab.com/install/ GitLab and Runner Installation Information]
*[https://docs.gitlab.com/omnibus/docker/ GitLab Installation using Docker]
*[https://docs.gitlab.com/runner/install/docker.html GitLab Runner Installation with Docker]

== AWS Estimate ==

Here is follow-up information for you to look over to get a better idea of the setup, general information, and pricing of the services.
# '''Running GitLab on AWS:'''
#* GitLab has fantastic documentation to get you setup and running. Using their Omnibus package or Marketplace listings are quick and easy ways to get started. GitLab has different Marketplace listings based on which license is used, the community version is linked.
#** https://about.gitlab.com/install/#ubuntu
#** https://aws.amazon.com/marketplace/pp/B071RFCJZK
#* It does require setup and maintenance on your end, it’s not a fully managed service. You would provision the GitLab instance and the Runner instance and complete their espective software install and setup.
# '''Pricing GitLab on AWS:'''
#:'''Factors to consider:'''
#* Storage:
#** S3 (Simple Storage Service) is a great service for storing your data. It is highly scalable, durable, and available object storage where you don’t need to provision anything. It’s significantly cheaper than holding the data in volumes attached to your instances.
#** The EBS (Elastic Block Store) volumes are charged based on their provisioned size and are resizable, so you can start off smaller and resize if needed.
#* Compute:
#** EC2 instance savings plans can greatly reduce your compute cost, ranging from around 30% to 60% compute savings over the on-demand instances shown in the price estimates. It varies based on the commitment time (1 or 3 years) and payment options (no upfront to full upfront).
#** GitLab has a guide on autoscaling Runners on AWS: https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/
#:'''Pricing estimates:'''
#* The wiki setup estimate: https://calculator.aws/#/estimate?id=a540551f0239aef761c25ec52c9b87f5c0571563
#** $8500/year
#** This has two instances, the GitLab instance with 2vCPU and 8GB of RAM, and the Runner instance with 4vCPU and 16GB of RAM. Each has 4TB of hard drive storage. This estimate does not include backups.
#** The lion’s share of the cost from this estimate comes from the 8TB of provisioned storage attached to the instances, the majority of which can be held in S3.
#* Moving storage to S3 estimate: https://calculator.aws/#/estimate?id=636f64ec1dff1f2f2e8aacfe50793480e3dc2160
#** $5700/year
#** The same setup as above but with the majority of storage (5TB) offloaded into S3. The GitLab instance has 50GB of SSD storage and the Runner instance has 250GB of SSD storage, each with daily backups. Storage performance for the instances will be better due to using SSDs instead of hard drives.
#* Including with EC2 Instance Savings Plans: https://calculator.aws/#/estimate?id=7f6a08fbe7e1feba574cd8daf15145773f6fcb27
#** $4500/year
#** Same as the previous estimate but with EC2 one year savings plans paid upfront.
# '''Serverless Managed CI/CD Solution, CodePipeline:'''
#* Fully managed CI/CD pipeline solution where you would just manage the users.
#** Source Control : AWS CodeCommit is a private Git service that can securely store your source code, binaries, and application assets.
#** Build: AWS CodeBuild allows you to build and test your application in preconfigured or custom environments.
#** Deploy: AWS CodeDeploy automates software deployments to your AWS or on-premises servers.
#* Usage based pricing, there are no servers to provision or maintain.
#* Free tier eligible so you can test out the services for free.
#* Natively integrates with other AWS services and control plane.
#* More information about CodePipeline: https://aws.amazon.com/codepipeline/
#:'''Pricing estimates:'''
#* $2500/year
#* I’ve attached a spreadsheet with modifiable red input numbers to get an idea of how much this solution will cost.
#* The 5TB of storage on S3 with 2.5TB of data transfer out per year ($1821/year) would be in addition to CodePipeline costs. For example, if the spreadsheet outputs $650/year, the total with S3 data storage, transfer, and CI/CD would be $2471/year.

[[Media:CodePipeline_Cost_Estimation.xlsx|Configurable Excel Spreadsheet]]

== Related Entries ==
*[[CSEP_Working_Group]]
*[[CSEP_Project]]

CSEP Gitlab

2020-06-25T18:40:26Z

Wsavran: /* AWS Estimate */

SCEC's CSEP project is coordinating an earthquake and ground motion forecasting project with national and international partners. Project members want to use the Gitlab system to support the development of the CSEP open-source scientific software development community, the code development, distributed version control, and automated software testing environment. Specifications describing the expected system requirements and preliminary data creation and transfer estimates are below.

== CSEP CI/CD System Overview Diagram ==
[[Image:Csep gitlab roadmap.pdf|250px| "Gitlab Server and Runner"]]

*[[Media:Csep gitlab roadmap.pptx | Gitlab Server Configuration as ppt file]]
*[[Media:Csep gitlab roadmap.pdf | Gitlab Server Configuration as pdf file]]

== Data Usage Estimates ==
Estimates for data usages are based on combining the two use cases for the GitLab repository (1) Code storage and (2) Storage of Large data files and experiment results. These estimates are based on data from the current CSEP project, such as current project members, current storage in current and old repositories, and rough estimates of expected data usage for past and future experiments.
# amount generated internally by the system per year, most of which can be left on AWS storage: 5TB/year
# amount of data transferred into AWS: 1TB/year
# amount of data transferred out of AWS: 2.5TB/year

== Data Estimates Breakdown ==
Project Info: 
CSEP Project Members: 20 
CSEP1 Repo Size (in Gb): 1.2 
Storage per experiment (in Gb): 10 

(1) Code Storage
Total data stored: 25 Gb (High-end estimate including models, codes and benchmark data sets) 
Data transferred: [50-500] Gb 

(2) Data/Catalog Storage
Total Storage: 250 Gb 
Data transferred: [100 Gb - 1 Tb] 

(3) Experiment Results
Total Storage: 4 Tb 
Data transferred: [100 Gb - 1 Tb] 

Overall Estimates:
Total stored: ~4.5 Tb 
Transferred: [250 Gb - 2.5 Tb] 

== AWS Estimation Webpage ==
*[https://aws.amazon.com/ec2/pricing/ EC2 Pricing]
*[https://calculator.aws/ Total Cost Estimator]
*[https://docs.gitlab.com/ee/install/aws/ AWS Gitlab Example Deployment]

== Gitlab Basic System Requirements ==
* Recent Linux Distro (Ubuntu,Centos...)
* Generate SSH keys
* Configure SMTP Server
* 8GB RAM is the recommended minimum memory size for all installations and supports up to 100 users. 16GB RAM supports up to 500 users
* Databases: PostgreSQL
* Redis/Sidekiq: stores all user sessions and background task queue processes the background jobs with a multithreaded process
* Prometheus and it’s exporters
* Avoid installing GitLab Runner on the same machine where Gitlab is installed.
* GitLab needs JavaScript enabled in browsers to support features such as Issue Boards.

== CSEP Computer and Storage Inventory ==
*[[CSEP_Computers]]
*[[CSEP_Hardware_Inventory]]
*[[Media:Computer Hardware and Storage Systems · SCECcode CSEP Wiki.pdf | Github Docs]]
*SCEC CSEP Storage Summaries:
** SCEC Storage Summary: [[Media:Disk Usage Jan2020.txt | Usage Report]]
** Sorted by Username: [[Media:Sorted_Storage_Jan2020.txt | Sorted Usage Report]]

== Overview and Gitlab Installation ==
*[https://about.gitlab.com/ Gitlab] integration software.
*[https://about.gitlab.com/pricing/#self-managed Gitlab Software Options]
*[https://about.gitlab.com/install/ GitLab and Runner Installation Information]
*[https://docs.gitlab.com/omnibus/docker/ GitLab Installation using Docker]
*[https://docs.gitlab.com/runner/install/docker.html GitLab Runner Installation with Docker]

== AWS Estimate ==

Here is follow-up information for you to look over to get a better idea of the setup, general information, and pricing of the services.
# '''Running GitLab on AWS:'''
#* GitLab has fantastic documentation to get you setup and running. Using their Omnibus package or Marketplace listings are quick and easy ways to get started. GitLab has different Marketplace listings based on which license is used, the community version is linked.
#** https://about.gitlab.com/install/#ubuntu
#** https://aws.amazon.com/marketplace/pp/B071RFCJZK
#* It does require setup and maintenance on your end, it’s not a fully managed service. You would provision the GitLab instance and the Runner instance and complete their espective software install and setup.
# '''Pricing GitLab on AWS:'''
#:'''Factors to consider:'''
#* Storage:
#** S3 (Simple Storage Service) is a great service for storing your data. It is highly scalable, durable, and available object storage where you don’t need to provision anything. It’s significantly cheaper than holding the data in volumes attached to your instances.
#** The EBS (Elastic Block Store) volumes are charged based on their provisioned size and are resizable, so you can start off smaller and resize if needed.
#* Compute:
#** EC2 instance savings plans can greatly reduce your compute cost, ranging from around 30% to 60% compute savings over the on-demand instances shown in the price estimates. It varies based on the commitment time (1 or 3 years) and payment options (no upfront to full upfront).
#** GitLab has a guide on autoscaling Runners on AWS: https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/
#:'''Pricing estimates:'''
#* The wiki setup estimate: https://calculator.aws/#/estimate?id=a540551f0239aef761c25ec52c9b87f5c0571563
#** $8500/year
#** This has two instances, the GitLab instance with 2vCPU and 8GB of RAM, and the Runner instance with 4vCPU and 16GB of RAM. Each has 4TB of hard drive storage. This estimate does not include backups.
#** The lion’s share of the cost from this estimate comes from the 8TB of provisioned storage attached to the instances, the majority of which can be held in S3.
#* Moving storage to S3 estimate: https://calculator.aws/#/estimate?id=636f64ec1dff1f2f2e8aacfe50793480e3dc2160
#** $5700/year
#** The same setup as above but with the majority of storage (5TB) offloaded into S3. The GitLab instance has 50GB of SSD storage and the Runner instance has 250GB of SSD storage, each with daily backups. Storage performance for the instances will be better due to using SSDs instead of hard drives.
#* Including with EC2 Instance Savings Plans: https://calculator.aws/#/estimate?id=7f6a08fbe7e1feba574cd8daf15145773f6fcb27
#** $4500/year
#** Same as the previous estimate but with EC2 one year savings plans paid upfront.
# '''Serverless Managed CI/CD Solution, CodePipeline:'''
#* Fully managed CI/CD pipeline solution where you would just manage the users.
#** Source Control : AWS CodeCommit is a private Git service that can securely store your source code, binaries, and application assets.
#** Build: AWS CodeBuild allows you to build and test your application in preconfigured or custom environments.
#** Deploy: AWS CodeDeploy automates software deployments to your AWS or on-premises servers.
#* Usage based pricing, there are no servers to provision or maintain.
#* Free tier eligible so you can test out the services for free.
#* Natively integrates with other AWS services and control plane.
#* More information about CodePipeline: https://aws.amazon.com/codepipeline/
#:'''Pricing estimates:'''
#* $2500/year
#* I’ve attached a spreadsheet with modifiable red input numbers to get an idea of how much this solution will cost.
#* The 5TB of storage on S3 with 2.5TB of data transfer out per year ($1821/year) would be in addition to CodePipeline costs. For example, if the spreadsheet outputs $650/year, the total with S3 data storage, transfer, and CI/CD would be $2471/year.

[[Media:CodePipeline_Cost_Estimation.xlsx]]

== Related Entries ==
*[[CSEP_Working_Group]]
*[[CSEP_Project]]

Wsavran: /* Summary */

__TOC__

== Summary ==
The general philosophy is that the file type should be human readable and easy to use for both researchers and developers. We should support multiple readers to ingest various catalog formats into the PyCSEP codebase.

The catalog will initially be defined as an ASCII .csv format where each row corresponds to a single event. This file will have the following headers: 
<code>
longitude, latitude, '''M''', time_string format="%Y-%m-%d %H:%M:%S.%f", depth, [catalog_id]
</code>

<code>'''longitude'''</code>: longitude in decimal degrees
<code>'''latitude'''</code>: latitude in decimal degrees
<code>'''M'''</code>: magnitude arbitrary units
<code>'''time string (UTC)'''</code>: year-month-day hour-minute-second.fraction_second. strptime format="%Y-%m-%d %H:%M:%S.%f": example: 1985-01-01 00:00:00.0
<code>'''depth'''</code>: hypocenter depth in (km)
<code>'''catalog_id'''</code>: indicates type of catalog
* observed: -1
* simulated: [0, n_cat-1]

This format can be easily extended to additional file types, including more advanced storage types like HDF5 or databases. Modelers can chose an arbitrary number of catalogs to store within a file. Events will be mapped to catalogs through the ''catalog_id'' field.

== Example Catalog ==
<code>
lon,lat,M,epoch_time,depth,catalog_id,event_id
</code> 
-117.43017,35.616665,4.73,1562383355630,9.35,-1
-117.7365,35.891,4.64,1562383368300,9.1,-1
-117.617836,35.803165,4.84,1562383430720,11.44,-1
-117.67083,35.86067,4.61,1562383527970,10.32,-1
-117.52167,35.55517,4.68,1562386024570,5.64,-1
-117.58083,35.777668,3.85,1562386249040,10.41,-1
-117.69067,35.902332,3.83,1562386262210,7.27,-1
-117.72283,35.887333,3.87,1562386338850,9.17,-1
-117.6145,35.584667,4.8,1562386387080,9.73,-1
-117.402664,35.604168,3.84,1562386460800,6.72,-1
-117.49067,35.6795,3.74,1562386471370,0.91,-1
-117.68483,35.910168,5.44,1562386735790,7.41,-1
-117.6175,35.785,4.49,1562386755470,10.6,-1
-117.717,35.885,3.83,1562386794940,4.9,-1

CSEP 2 CATALOG FORMAT

2020-05-26T17:50:12Z

Wsavran: /* Summary */

__TOC__

== Summary ==
The general philosophy is that the file type should be human readable and easy to use for both researchers and developers. We should support multiple readers to ingest various catalog formats into the PyCSEP codebase.

The catalog will initially be defined as an ASCII .csv format where each row corresponds to a single event. This file will have the following headers: 
<code>
longitude, latitude, '''M''', time_string format="%Y-%m-%d %H:%M:%S.%f", depth, [catalog_id]
</code>

<code>'''longitude'''</code>: longitude in decimal degrees
<code>'''latitude'''</code>: latitude in decimal degrees
<code>'''M'''</code>: magnitude arbitrary units
<code>'''time string (UTC)'''</code>: year-month-day hour-minute-second.fraction_second. strptime format="%Y-%m-%d %H:%M:%S.%f": example: 1985-01-01 00:00:00.0
<code>'''depth'''</code>: hypocenter depth in (km)
<code>'''catalog_id'''</code>: indicates type of catalog
* observed: -1
* simulated: [0, n_cat-1]

This format can be easily extended to additional file types, including more advanced storage types like HDF5 or databases. Modelers can chose an arbitrary number of catalogs to store within a file as events will be mapped to various catalogs through the catalog_id field.

== Example Catalog ==
<code>
lon,lat,M,epoch_time,depth,catalog_id,event_id
</code> 
-117.43017,35.616665,4.73,1562383355630,9.35,-1
-117.7365,35.891,4.64,1562383368300,9.1,-1
-117.617836,35.803165,4.84,1562383430720,11.44,-1
-117.67083,35.86067,4.61,1562383527970,10.32,-1
-117.52167,35.55517,4.68,1562386024570,5.64,-1
-117.58083,35.777668,3.85,1562386249040,10.41,-1
-117.69067,35.902332,3.83,1562386262210,7.27,-1
-117.72283,35.887333,3.87,1562386338850,9.17,-1
-117.6145,35.584667,4.8,1562386387080,9.73,-1
-117.402664,35.604168,3.84,1562386460800,6.72,-1
-117.49067,35.6795,3.74,1562386471370,0.91,-1
-117.68483,35.910168,5.44,1562386735790,7.41,-1
-117.6175,35.785,4.49,1562386755470,10.6,-1
-117.717,35.885,3.83,1562386794940,4.9,-1