CARC BBP Setup

SCEC has migrated our BBP server from University Park server room to USC's new research Computing group CARC.

Installing Broadband on CARC

The users account must be configured using the CARC module system. The module systems sets up the users computing environment, including the compilers and libraries. The module system provides a lot of flexibility in building the BBP software. We have tested the broadband platform on broadband_carc using a specific module configuration using recent gnu compilers, that we describe in these installation notes.

Server Access

One your CARC login account on discovery and endeavour works, you will have access to broadband. Login through endeavour.

• ssh username@endeavour.usc.edu
• ssh broadband

when I log into endeavour, I just log into 'broadband'
here's what I use, first I log into endeavour
ssh -o "ServerAliveInterval 60" endeavour.usc.edu
then I log into broadband
ssh -o "ServerAliveInterval 60" broadband

Disk Quota

• Users have 10GB quota on /home1/username
• Users have 10TB+ quota on /project/scec_608/<username>

We will install broadband software on /project, and configure it to write results to this storage system.

CARC File systems

CARC users have a home directory on the /home/username filesystem. This directory has a quota of about 10GB. To do a Broadband installation, we will need to make use of SCEC storage on the CARC /project/scec_608 filesystem.

To create a broadband installation in on the broadband_carc server, we recommend using the following filesystems:

• /home1/username

This is the users home login account. Edit the .bashrc in this account to setup the required modules and environment variables.

• /project/scec_608/username/scecbbp

This should be the installation directory for bbp. All bbp related software and data (except the .bashrc file) can be found here. Typically, users will log in and move into this directory to run the broadband platform. This is the directory that where the user clones the bbp repo and runs the bbp installation script.

• /project/scec_608/username/scecbbp/bbp_gf
• /project/scec_608/username/scecbbp/bbp_val
• /project/scec_608/username/scecbbp/bbp_data

These are bbp static data (gf, val), and output data (bbp_data). When the bbp platform installation process runs, these where the outputs are writting including logs, temp files, and results.

Anaconda Installation

To support users running broadband platform on computers where we don't have root, we recommend use of Anaconda python, so the software is managed (adding packages) without root access. Retreived command line installer using curl on broadbnad

• curl -O https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh

Then, it will ask you where to install anaconda. We recommend you install it on /project/scec_608/username directory. Don't accept the default installation directory, rather, change it to poing to the project file system. eg: /project/scec_608/maechlin/anaconda3

When anaconada asks to update your .bashrc, say yes.

WARNING:
    You currently have a PYTHONPATH environment variable set. This may cause
    unexpected behavior when running the Python interpreter in Anaconda3.
    For best results, please verify that your PYTHONPATH only points to
    directories of packages that are compatible with the Python interpreter
    in Anaconda3: /project/scec_608/maechlin/anaconda3
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]

It will add a section like this:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/project/maechlin_162/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/project/maechlin_162/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Then conda install pyproj

• $condal install pyproj

.bashrc

Add this entries, modified with your username, to your .bashrc

(base) [maechlin@broadband ~]$ more .bashrc
# CARC Module Load system
module purge
module load gcc/8.3.0

# Setup BBP Environment
export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp
export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf
export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data
alias bbp="cd /project/scec_608/maechlin/scecbbp"

export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:$PYTHONPATH
export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH
ulimit -s unlimited
export PATH

Login and retrieve from github

Log into the broadband server, cd to the bbp installation directory, in my case.

• cd /project/scec_608/maechlin/scecbbp

Clone the scecbbp github repository. Be sure to retrieve the dev branch which has been ported to the newest version of the CARC systems.

• $git clone --single-branch --branch dev https://github.com/sceccode/bbp.git

bbp easy install

cd to bbp installation home, where the clone was done.

• $cd /project/scec_608/maechlin/scecbbp/bbp/setup
• $./easy_install_bbp_19.4.0.sh

Install Questions About Regions

The BBP installer will ask you which regions you want to use. Respond using a number 1, or 2. Only one region is required to pass the acceptance tests. Including more regions increases the installation size and time.

 Please select what velocity models (regions) you would like to install,
 using '1' for Yes, or '2' for No:

==> Would you like to install the Northern California region (27GB)?
1) Yes
2) No
#? y
#? 1
==> Would you like to install the LA Basin region (25GB)
    needed for unit and acceptance tests to run?
1) Yes
2) No
#? 1
==> Would you like to install the Central California region (14GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Mojave region (27GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Central Japan region (28GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Western Japan region (28GB)?
1) Yes
2) No
#? 1
=> Installing Broadband Platform Velocity Model Packages
==> Northern California

Run Unit Tests

Once installation completed, log out, log bach in, and run the unittests

• $cd /project/scec_608/maechlin/scecbbp/bbp/bbp/tests
• ./UnitTests.py

This runs 40 unit tests which should all pass.

Then run Acceptance Tests.

• ./AcceptanceTests.py

Once these UnitTests and Acceptance tests pass, the broadband is correctly installed and ready for research use.

Related Entries

• Broadband Platform
• SCEC Software