Difference between revisions of "Containers for CyberShake"

From SCECpedia
Jump to navigationJump to search
Line 7: Line 7:
 
== Setting up a serial container ==
 
== Setting up a serial container ==
  
Explain the steps involved in building and running serial code in a container
+
(Explains the steps involved in building and running serial code in a container)
 +
 
 +
Image Prework:
 +
Convert Sandbox Director to image file:
 +
sudo singularity build new-sif.sif myUbuntu
 +
imageName sandbox directory
 +
 
 +
User Work:
 +
Install Dependencies
 +
sudo apt-get update && sudo apt-get install -y \
 +
    build-essential \
 +
    uuid-dev \
 +
    libgpgme-dev \
 +
    squashfs-tools \
 +
    libseccomp-dev \
 +
    wget \
 +
    pkg-config \
 +
    git \
 +
    cryptsetup-bin
 +
 
 +
Download Go
 +
export VERSION=1.13.5 OS=linux ARCH=amd64 && \
 +
    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
 +
    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
 +
    rm go$VERSION.$OS-$ARCH.tar.gz
 +
 
 +
Set Up Go
 +
echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
 +
    echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
 +
    source ~/.bashrc
 +
 
 +
Install Singularity
 +
export VERSION=3.5.2 && # adjust this as necessary \
 +
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
 +
    tar -xzf singularity-${VERSION}.tar.gz && \
 +
    cd singularity
 +
 
 +
Check if Singularity Works
 +
git clone https://github.com/sylabs/singularity.git && \
 +
    cd singularity && \
 +
    git checkout v3.5.2
 +
 
 +
 
 +
Get Image
 +
singularity pull <source>
 +
 
 +
Execute Command in from Outside Container
 +
Singularity Commands
 +
singularity exec myPythonContainer.sif cat /etc/lsb-release
 +
imageName command
 +
 
 +
singularity exec myPythonContainer.sif python3 helloWorld.py
 +
imageName command
 +
 
 +
Find Size:
 +
singularity cache list
  
 
== Containers on Frontera ==
 
== Containers on Frontera ==

Revision as of 20:32, 6 August 2020

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

Selection of Containers

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, IntelMPI to name a few. Shifter is highly reliant on MPICH ABI, which would require site-specific MPI libraries to be copied to the container at run time.

Setting up a serial container

(Explains the steps involved in building and running serial code in a container)

Image Prework: Convert Sandbox Director to image file: sudo singularity build new-sif.sif myUbuntu imageName sandbox directory

User Work: Install Dependencies sudo apt-get update && sudo apt-get install -y \

   build-essential \
   uuid-dev \
   libgpgme-dev \
   squashfs-tools \
   libseccomp-dev \
   wget \
   pkg-config \
   git \
   cryptsetup-bin

Download Go export VERSION=1.13.5 OS=linux ARCH=amd64 && \

   wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
   sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
   rm go$VERSION.$OS-$ARCH.tar.gz

Set Up Go echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \

   echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
   source ~/.bashrc

Install Singularity export VERSION=3.5.2 && # adjust this as necessary \

   wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
   tar -xzf singularity-${VERSION}.tar.gz && \
   cd singularity

Check if Singularity Works git clone https://github.com/sylabs/singularity.git && \

   cd singularity && \
   git checkout v3.5.2


Get Image singularity pull <source>

Execute Command in from Outside Container Singularity Commands singularity exec myPythonContainer.sif cat /etc/lsb-release imageName command

singularity exec myPythonContainer.sif python3 helloWorld.py imageName command

Find Size: singularity cache list

Containers on Frontera

Serial Containers

Explain how you got serial containers running on Frontera.

MPI Containers

Explain how you got MPI containers running on Frontera.