SCECpedia - User contributions [en]

User:Llocsin

2020-08-14T01:22:11Z

Llocsin: /* SVN to Git Plan */

==SVN to Git Plan==
Convert SVN repository to a Git Repository.
====Steps:====
#Download svn2git tool: https://github.com/nirvdrum/svn2git
<pre>git clone <remote></pre>
#Pull desired SVN repository
##<pre>svn checkout <remote></pre>
#Create Text Document of Authors to link to Git Accounts
##create authors.txt
##add in all authors with the following format:
###jcoglan = James Coglan <jcoglan@never-you-mind.com>
###stnick = Santa Claus <nicholas@lapland.com> (Recommended to link everyone’s account but if you can’t, it’s not required to link everyone’s account, but you must provide an email, if only a made up one.)
#Convert the repositories with the git commands:
##$ svn2git http://source.usc.edu/svn/mesh_partitioner/ --trunk trunk --tags tags --nobranches --authors ~/authors.txt (Note: --trunk trunk (the trunk argument is the name of the trunk repository-tags tags (tags is the name of your repository of tags))
#Push to the desired git remote. (In this example we use github)
##Add Remote
###git remote add <remote-name> <remote-url>##git remote origin git@github.com:SCECcode/CyberShake.git
##Commit
###git add .
###git commit -m “Initial Commit of Converted SVN to Git Repository Code”
##Use tags command
###git push --tags

Rebase History
If the hosted Subversion repository’s history possesses commits not yet in the local Git repository, the dcommit operation will be rejected until the commits are acquired with this command:

<pre>git svn rebase</pre>

===Resources:===
#svn2git tool (from the guide): https://github.com/nirvdrum/svn2git
#Official Git Documentation:
#Guide: https://viastudio.com/migrate-svn-git/: https://github.github.com/training-kit/downloads/subversion-migration/
#Github Importer Tool (alternate): https://docs.github.com/en/github/importing-your-projects-to-github/about-github-importer

Install Singularity
To Do:

Basic Singularity Commands
Pull - pulls a container image from a remote source
$ sudo singularity pull <remote source>
<remote source>
Singularity Container Services
$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION
Singularity Hub
$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION (Note: the path only needs to match the pull card. please see the remote website for example.)
Docker Hub
$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION
(Note docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build)

Exec - executes an EXTERNAL COMMAND
$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND

Shell - shells into an existing container
singularity shell IMAGE_NAME.sif
*Note: Your home directory is mounted by default

Run - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
$ singularity run IMAGE_NAME.sif

Build (BIG TO DO: Very important)
$ singularity build IMAGE_NAME.sif <source>
Sources include
Another Image either docker or singularity
Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
$ singularity shell docker://ubuntu

Creating Definition Files:
Workflow:
Set up complex workflows with Recipe File:
Alternatively-
Sandbox Directory Prototype Final Container: sudo singularity build --sandbox ubuntu_s docker://ubuntu

WRITE RECIPE:
File Name: Singularity.def

Bootstrap: docker
From: ubuntu:18.04
%post
apt-get -y update
apt-get -y install python3
%files
helloWorld.py /

%runscript
python3 /hello_world.py

$ sudo singularity build CONTAINER_NAME.sif Singularity.def
$ sudo singularity run CONTAINER_NAME.sif
Result:
Hello World

Resources:
Singularity Guide:
Singularity Help:

Pre-Made Containers:

TACC Resources:

MVAPICH Dependances
MPICH Dependencies:

User:Llocsin

2020-08-14T00:33:40Z

Llocsin: /* SVN to Git Plan */

==SVN to Git Plan==
Convert SVN repository to a Git Repository.
====Steps:====
#Download svn2git tool: https://github.com/nirvdrum/svn2git
<pre>git clone <remote></pre>
#Pull desired SVN repository
##<pre>svn checkout <remote></pre>
#Create Text Document of Authors to link to Git Accounts
##create authors.txt
##add in all authors with the following format:
###jcoglan = James Coglan <jcoglan@never-you-mind.com>
###stnick = Santa Claus <nicholas@lapland.com> (Recommended to link everyone’s account but if you can’t, it’s not required to link everyone’s account, but you must provide an email, if only a made up one.)
#Convert the repositories with the git commands:
##$ svn2git http://source.usc.edu/svn/mesh_partitioner/ --trunk trunk --tags tags --nobranches --authors ~/authors.txt (Note: --trunk trunk (the trunk argument is the name of the trunk repository-tags tags (tags is the name of your repository of tags))
#Push to the desired git remote. (In this example we use github)
##Add Remote
###git remote add <remote-name> <remote-url>##git remote origin git@github.com:SCECcode/CyberShake.git
##Commit
###git add .
###git commit -m “Initial Commit of Converted SVN to Git Repository Code”
##Use tags command
###git push --tags

===Resources:===
#svn2git tool (from the guide): https://github.com/nirvdrum/svn2git
#Official Git Documentation:
#Guide: https://viastudio.com/migrate-svn-git/: https://github.github.com/training-kit/downloads/subversion-migration/
#Github Importer Tool (alternate): https://docs.github.com/en/github/importing-your-projects-to-github/about-github-importer

Install Singularity
To Do:

Basic Singularity Commands
Pull - pulls a container image from a remote source
$ sudo singularity pull <remote source>
<remote source>
Singularity Container Services
$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION
Singularity Hub
$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION (Note: the path only needs to match the pull card. please see the remote website for example.)
Docker Hub
$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION
(Note docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build)

Exec - executes an EXTERNAL COMMAND
$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND

Shell - shells into an existing container
singularity shell IMAGE_NAME.sif
*Note: Your home directory is mounted by default

Run - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
$ singularity run IMAGE_NAME.sif

Build (BIG TO DO: Very important)
$ singularity build IMAGE_NAME.sif <source>
Sources include
Another Image either docker or singularity
Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
$ singularity shell docker://ubuntu

Creating Definition Files:
Workflow:
Set up complex workflows with Recipe File:
Alternatively-
Sandbox Directory Prototype Final Container: sudo singularity build --sandbox ubuntu_s docker://ubuntu

WRITE RECIPE:
File Name: Singularity.def

Bootstrap: docker
From: ubuntu:18.04
%post
apt-get -y update
apt-get -y install python3
%files
helloWorld.py /

%runscript
python3 /hello_world.py

$ sudo singularity build CONTAINER_NAME.sif Singularity.def
$ sudo singularity run CONTAINER_NAME.sif
Result:
Hello World

Resources:
Singularity Guide:
Singularity Help:

Pre-Made Containers:

TACC Resources:

MVAPICH Dependances
MPICH Dependencies:

Containers for CyberShake

2020-08-08T02:09:17Z

Llocsin: /* Install Singularity */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charliecloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charliecloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
Check the Newest Release Version [https://github.com/hpcng/singularity/tags]
<pre>export VERSION=3.6.1 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
#Another Image either docker or singularity
#Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====
Basic Ubuntu
Basic Ubuntu Container with Python

===== Frontera =====

Basic Ubuntu with Mvapich

===== Summit =====
Basic Ubuntu with ???

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers. Containers do not seem to support singularity exec)

ORNL
#Container Builder Tool - [https://github.com/olcf/container-builder]

Containers for CyberShake

2020-08-08T02:08:58Z

Llocsin: /* Install Singularity */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charliecloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charliecloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
Check the Release Version [https://github.com/hpcng/singularity/tags]
<pre>export VERSION=3.6.1 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
#Another Image either docker or singularity
#Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====
Basic Ubuntu
Basic Ubuntu Container with Python

===== Frontera =====

Basic Ubuntu with Mvapich

===== Summit =====
Basic Ubuntu with ???

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers. Containers do not seem to support singularity exec)

ORNL
#Container Builder Tool - [https://github.com/olcf/container-builder]

Containers for CyberShake

2020-08-08T02:06:58Z

Llocsin: /* Resources */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charliecloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charliecloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
#Another Image either docker or singularity
#Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====
Basic Ubuntu
Basic Ubuntu Container with Python

===== Frontera =====

Basic Ubuntu with Mvapich

===== Summit =====
Basic Ubuntu with ???

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers. Containers do not seem to support singularity exec)

ORNL
#Container Builder Tool - [https://github.com/olcf/container-builder]

Containers for CyberShake

2020-08-07T22:32:55Z

Llocsin: /* Prebuilt Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charliecloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charliecloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
#Another Image either docker or singularity
#Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====
Basic Ubuntu
Basic Ubuntu Container with Python

===== Frontera =====

Basic Ubuntu with Mvapich

===== Summit =====
Basic Ubuntu with ???

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers. Containers do not seem to support singularity exec)

Containers for CyberShake

2020-08-07T08:03:03Z

Llocsin: /* Basic Singularity Commands */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charliecloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charliecloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
#Another Image either docker or singularity
#Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers. Containers do not seem to support singularity exec)

Containers for CyberShake

2020-08-07T08:02:05Z

Llocsin: /* Resources */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charliecloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charliecloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers. Containers do not seem to support singularity exec)

Containers for CyberShake

2020-08-07T08:00:02Z

Llocsin: /* Selection of Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charliecloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charliecloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:59:35Z

Llocsin: /* Selection of Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime. Charlie Cloud does not have a module in Frontera.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:55:53Z

Llocsin: /* Build or Pull a Singularity Image */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
*Note: This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:55:20Z

Llocsin: /* MPI */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
2. Hybrid/Host Approach [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]

For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:53:51Z

Llocsin: /* MPI */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
# Bind Approach (cannot be used)
You cannot use the bind on a Frontera.
#Hybrid/Host Approach
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:52:59Z

Llocsin: /* Execute Command in from Outside Container */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

======Execute Command in from Outside Container======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:52:51Z

Llocsin: /* Get Image */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
======Get Image======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

=======Execute Command in from Outside Container=======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:52:44Z

Llocsin: /* Setting up a serial container (on your computer) */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
=======Get Image=======
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

=======Execute Command in from Outside Container=======
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:52:27Z

Llocsin: /* Setting up a serial container (on your computer) */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
====Get Image====
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

====Execute Command in from Outside Container====
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:51:25Z

Llocsin: /* Compile Program */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image======
The singularity image file NEED the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>
This work on Frontera. MPI Library: mvapich

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:50:14Z

Llocsin: /* Make MPI Program */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

Make Example File: sum_sqrt.c
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image====== with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:49:43Z

Llocsin: /* MPI Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

======Make MPI Program======

- (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

======Compile Program======
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

======Build or Pull a Singularity Image====== with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

======Execute your command======
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:46:47Z

Llocsin: /* Installing Singularity */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

=====Install Dependencies=====
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

=====Download Go=====
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

=====Set Up Go=====
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

=====Install Singularity=====
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

=====Check if Singularity Works=====
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:46:02Z

Llocsin: /* Building Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH (see basic commands > pull)

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:45:32Z

Llocsin: /* Building Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
# Install Singularity
# Pull a Basic Image and use the --sandbox flag
# Install desired dependencies in the sandbox
## Build Dependencies
## Correct MPI Library and set environment variables
## Add any files you want to run
# Create a Definition File - Transferring the setup commands that you tested in the build file into a definition file.
# Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:44:32Z

Llocsin: /* Building Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because you do not have sudo and --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote. Just do that.

To build from scratch:
1. Install Singularity
2. Pull a Basic Image and use the --sandbox flag
3. Install desired dependencies in the sandbox
a. Build Dependencies
b. Correct MPI Library and set environment variables
c. Add any files you want to run
4. Create a Definition File
Transferring the setup commands that you tested in the build file into a definition file.
5. Upload Dependency file to Singularity Container Library or Singularity Hub

To retrieve your container, just pull from library://USER/PATH or sub://USER/PATH

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:42:24Z

Llocsin: /* Building Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====
You cannot build containers on Frontera because --fakeroot does not work.
--fakeroot did not work for me in Frontera because it could not find me as a user.
<pre>
$ singularity build --fakeroot hello.sif ubunutu18-mvapich.def
FATAL: could not use fakeroot: no user entry found for llocsin
</pre>

--remote flag is just a step away from building it yourself and uploading it to a remote.

To build from scratch:
1. Install Singularity
2. Pull a Basic Image and use the --sandbox flag
3. Install desired dependencies in the sandbox
a. Build Dependencies
b. Correct MPI Library and set environment variables
c. Add any files you want to run
4. Create a Definition File
Transferring the setup commands that you tested in the build file into a definition file.

====== MPI ======
Singularity uses two methods.
1. Bind (cannot be used)
You cannot use the bind on a Frontera. Also, The
2.
For MPI containers, you have to use the hybrid approach. Frontera does not support bind.

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:28:40Z

Llocsin: /* Using Prebuilt Containers or Building Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

==== Building Containers ====

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:26:02Z

Llocsin: /* Using Prebuilt Containers or Building Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

==== Prebuilt Containers ====

===== Frontera =====

===== Summit =====

===== Generic =====

==== Building Containers ====

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:25:35Z

Llocsin: /* Building or Using Prebuilt Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Using Prebuilt Containers or Building Containers==

===== Frontera =====

===== Summit =====

===== Generic =====

==== Building Containers ====

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:21:49Z

Llocsin: /* Basic Singularity Commands */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (see build section for more details)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

== Building or Using Prebuilt Containers ==
=== Frontera ===

=== Summit ===

=== Generic ===

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:20:43Z

Llocsin: /* Building or Using Pre Made Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (BIG TO DO: Very important... a lot of details and opinions)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

Creating Definition Files: (To Do)
Set up complex workflows with Recipe File:
Alternatively-
Sandbox Directory Prototype Final Container: sudo singularity build --sandbox ubuntu_s docker://ubuntu

== Building or Using Prebuilt Containers ==
=== Frontera ===

=== Summit ===

=== Generic ===

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:19:29Z

Llocsin: /* MPI Containers */

This page is to document the steps involved in enabling the CyberShake codebase to run in a container environment.

== Selection of Containers ==

The available HPC Containers at the time of selection were Singularity, Charlie Cloud, and Shifter. Between the 3 of these container technologies, Singularity was widely adapted and had more open source tools. Because of this wide adaptation the module already existed in the Frontera system. Singularity has built-in support for different MPI libraries from OpenMPI, MPICH, and IntelMPI to name a few. Shifter, although light weight, is highly reliant on MPICH ABI. This would require site-specific MPI libraries to be copied to the container at runtime.

== Installing Singularity ==
Recommended for people who want to run Singularity locally or create there own custom containers. Use of premade containers does not require installation.

Install Dependencies
<pre>sudo apt-get update && sudo apt-get install -y \
build-essential \
uuid-dev \
libgpgme-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup-bin</pre>

Download Go
<pre>export VERSION=1.13.5 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz</pre>

Set Up Go
<pre>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc</pre>

Install Singularity
<pre>export VERSION=3.5.2 && # adjust this as necessary \
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity</pre>

Check if Singularity Works
<pre>git clone https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v3.5.2</pre>

== Setting up a serial container (on your computer) ==
Get Image
singularity pull <source>*
<pre>$ singularity build myPythonContainer.sif library://default/ubuntu:latest</pre>
*<sources> include Singularity Container Library (library), Singularity Hub (shub) and Docker Hub (docker).

Execute Command in from Outside Container
singularity exec imageName command
<pre>$ singularity exec myPythonContainer.sif cat /etc/lsb-release</pre>

singularity exec image_name command
<pre>$ singularity exec myPythonContainer.sif python3 helloWorld.py</pre>

Find Size of Container:
<pre>$ singularity cache list</pre>

*Note: Singularity cannot run on the Login Node

== Basic Singularity Commands ==
'''Pull''' - pulls a container image from a remote source.
<pre>$ sudo singularity pull <remote source></pre>
<remote source>:

1. Singularity Container Services [https://cloud.sylabs.io/library]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif library://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
2. Singularity Hub [https://singularity-hub.org/]
<pre>$ sudo singularity pull --name CONTAINER_NAME.sif shub://USER/PULL_PATH:VERSION</pre>
*Note: the path only needs to match the pull card. please see the remote website for example.
3. Docker Hub [https://hub.docker.com/]
<pre>$ sudo singularity build CONTAINER_NAME.sif docker://USER/PULL_PATH:VERSION</pre>
*Note 1: docker images have layers and it needs to be merged into 1 singularity image. For that to happen you MUST use: build
*Note 2: the path only needs to match the pull card. please see the remote website for example.

'''Exec''' - executes an EXTERNAL COMMAND
<pre>$ singularity exec IMAGE_NAME.sif EXTERNAL_COMMAND</pre>

'''Shell''' - shells into an existing container
<pre>$ singularity shell IMAGE_NAME.sif</pre>
*Note: Your home directory is mounted by default

'''Run''' - runs an image. Run is based on the Run Script parameters that were placed into the container when the image was built based the recipe
<pre>$ singularity run IMAGE_NAME.sif</pre>

'''Build''' (BIG TO DO: Very important... a lot of details and opinions)
<pre>$ singularity build IMAGE_NAME.sif <source></pre>
<source> include
-Another Image either docker or singularity
-Singularity definition file (use to be known as a recipe file), usually denoted with name.def

Note:
You can shell into a docker UI - explore different containers without pulling or building
<pre>$ singularity shell docker://ubuntu</pre>

Creating Definition Files: (To Do)
Set up complex workflows with Recipe File:
Alternatively-
Sandbox Directory Prototype Final Container: sudo singularity build --sandbox ubuntu_s docker://ubuntu

== Building or Using Pre Made Containers ==
=== Frontera ===

=== Summit ===

=== Generic ===

== Containers on Frontera ==

=== Serial Containers ===
1. Prepare
#Make helloWorld.py <code>$ echo "print(\"Hello World\")" > helloWorld.py</code>
#Install Module (only if using Supercomputer): <code>$ module load tacc-singularity</code> *Note: module save (if you plan to use singularity a lot)

2. Get a Singularity Image on Frontera
(*Note: If you want to write a particular program, you must have the dependencies installed in the container)
Options:
#By copying a image from your local to Frontera with scp
#Pull from the Computation Node
<pre>idev -N 1; singularity pull singularity pull library://libii/scec/ubuntu18.10-python3:sha256.522b070ad79309ef7526f87c34f0f8518e7d7acc6399aa6372fb0cf28fea25a1 </pre>
*Note: This command works in a sbatch file.

3-1. Interface with Computation Node

a. idev session
<pre>idev
ibrun singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>
b. sbatch (recommended)
<pre>
#!/bin/bash

#SBATCH -p development
#SBATCH -t 00:05:00
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -J test-singularity-python
#SBATCH -o test-singularity-python.o%j
#SBATCH -e test-singularity-python.e%j

# Run the actual program
singularity exec ubuntu18.10-python3_latest.sif python3 helloPython.py
</pre>

3-2. Execute from Local Computer (if Singularity is installed)
<pre>$ singularity exec ubuntu18.10-python3_latest.sif python3 helloWorld.py</pre>

=== MPI Containers ===

1. Make MPI Program - (Ex: named sum_sqrt.c)
<pre>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char** argv) {
//Grab Argument
char* temp = argv[1];
int numN = atoi(temp); //N
printf("Argument N: %d \n", numN);

// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
printf("Processes: %d \n", world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

//Local Variables
int nglobal = numN;
int block = nglobal/world_size;
int my_lo = (world_rank*block)+1, my_hi = (world_rank+1)*block;
/** Blocks
* int nlocal = nglobal/psize; flipped -> 1000/32 = 31.25 -> 31
* 31
int my_lo = (myrank*nlocal)+1, my_hi = (myrank+1)*nlocal);

rank low high inclusive
0 1 31 <=TO Do: Add loop to process 0 nlocal-1
1 32 62
2 63 93
3 93 124
4 124 135
* */

if(world_rank==0){ //master process
int mySum=0;
int pSum=0;
int totalSum=0;

printf("Main Process Start\n");

//send to P processors
for(int myprocessor=1; myprocessor <world_size; myprocessor++){
MPI_Send(&block, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD);
}

//process my block
for(int i=1 ; i <= block; i++){
mySum+=(i*i);
}

//process rounded truncated block
for(int left_over=block*world_size+1; left_over <= numN; left_over++){
mySum+=(left_over*left_over);
}
totalSum+=mySum;

//receive P processors
for(int myprocessor=1; myprocessor < world_size; myprocessor++){
MPI_Recv(&pSum, 1, MPI_INT, myprocessor, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
totalSum+=pSum;
printf("MpSum: %d\n", pSum);
}

//print final total
printf("Sum of Squares for %d is %d\n", numN, totalSum);

printf("Main Process End");
}else if(world_rank != 0){ //worker process
printf("Start Process: %d\n", world_rank);
int mySum=0;
//receive
MPI_Recv(&block, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

//calculate my sum of square
for(int i=my_lo; i < my_hi; i++){
mySum+=(i*i);
}

//send my sum
MPI_Send(&mySum, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD);
printf("End Process: %d\n", world_rank);
}

// Finalize the MPI environment.
MPI_Finalize();

return 0;
}
</pre>

2. Compile Program
<pre>$ mpicc -o sum_sqrt sum_sqrt.c</pre>

3. Build or Pull a Singularity Image with the same MPI library installed inside the container [https://sylabs.io/guides/3.6/user-guide/mpi.html#hybrid-model]
mvapich preinstalled in this container
<pre>$ idev -N 1
$ singularity pull shub://mkandes/ubuntu-mvapich</pre>

4. Execute your command
<pre>$ ibrun singularity exec ubuntu-mvapich_latest.sif ./sum_sqrt 100000</pre>

== Resources ==
#Singularity Guide [https://sylabs.io/docs/]
#Singularity Repository [https://github.com/hpcng/singularity]
#Singularity Container Library [https://cloud.sylabs.io/library]
#Singularity Hub [https://singularity-hub.org/]
#Docker Hub [https://hub.docker.com/]

TACC - Frontera
#TACC Containers [https://github.com/TACC/tacc-containers/tree/master/containers] (More geared for people who are familiar with Docker Containers)

Containers for CyberShake

2020-08-07T07:11:32Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T07:11:05Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T07:10:47Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T07:09:47Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T07:09:20Z

Llocsin: /* Containers on Frontera */

Containers for CyberShake

2020-08-07T05:08:42Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T05:08:19Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T05:08:04Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T05:07:49Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T05:07:06Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T05:05:49Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T05:04:43Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T05:04:00Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T04:46:09Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T04:27:36Z

Llocsin: /* Installing Singularity */

Containers for CyberShake

2020-08-07T04:26:59Z

Llocsin: /* Installing Singularity */

Containers for CyberShake

2020-08-07T04:24:53Z

Llocsin: /* Resources */

Containers for CyberShake

2020-08-07T04:15:49Z

Llocsin: /* Serial Containers */

Containers for CyberShake

2020-08-07T04:15:30Z

Llocsin: /* Serial Containers */