Difference between revisions of "CARC BBP Setup"
| Line 95: | Line 95: | ||
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data | export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data | ||
| − | # These help repair an issue on CARC fftw. These are not needed on standard BBP Centos installations | + | # These help repair an issue on CARC fftw. These point to an Installation done by fsilva. |
| − | export FFTW_INCDIR=/ | + | # These are not needed on standard BBP Centos installations |
| − | export FFTW_LIBDIR=/ | + | export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include |
| + | export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib | ||
| − | export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:$PYTHONPATH | + | export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/comps/PySeismoSoil:$PYTHONPATH |
export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH | export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH | ||
| Line 204: | Line 205: | ||
# Add these to .bashrc to help repair an issue on CARC fftw. | # Add these to .bashrc to help repair an issue on CARC fftw. | ||
# These are not needed on standard BBP Centos installations | # These are not needed on standard BBP Centos installations | ||
| − | export FFTW_INCDIR=/ | + | export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include |
| − | export FFTW_LIBDIR=/ | + | export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib |
</pre> | </pre> | ||
| Line 237: | Line 238: | ||
# Add the FFT variables and save file | # Add the FFT variables and save file | ||
# These help repair an issue on CARC fftw. These are not needed on standard BBP Centos installations | # These help repair an issue on CARC fftw. These are not needed on standard BBP Centos installations | ||
| − | export FFTW_INCDIR=/ | + | export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include |
| − | export FFTW_LIBDIR=/ | + | export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib |
source .bashrc | source .bashrc | ||
</pre> | </pre> | ||
| − | Then return to the root of the bbp source directory structure and type make. An example from my account shown here: | + | Then return to the root of the bbp source directory structure and type make. An example from my account showning the correct subdirectory where to run the make command is shown here: |
<pre> | <pre> | ||
(base) [maechlin@broadband bbp]$ ls | (base) [maechlin@broadband bbp]$ ls | ||
| Line 289: | Line 290: | ||
OK | OK | ||
| + | |||
| + | Once these UnitTests and Acceptance tests pass, the broadband is correctly installed and ready for research use. | ||
</pre> | </pre> | ||
| − | + | ||
| + | == Command Line == | ||
| + | |||
| + | |||
| + | == Output Files == | ||
| + | |||
| + | |||
| + | The Broadband Platform writes all it output files, including calculated ground motions, as well as a temp directory, and log files, the BBP_DATA directory. You can find where that is defined above, or by typing: env | grep BBP | ||
| + | |||
| + | Each "job" is assigned a simulation ID. This ID is assigned to all the data products associated with the simulation done. Sim ID and then data type is a common output file name. Images are identified as *.png files. | ||
| + | |||
| + | |||
| + | |||
== Related Entries == | == Related Entries == | ||
*[[Broadband Platform]] | *[[Broadband Platform]] | ||
*[[SCEC Software]] | *[[SCEC Software]] | ||
Revision as of 23:41, 16 August 2021
SCEC has migrated our BBP server from University Park server room to USC's new research Computing group CARC.
Contents
Installing Broadband on CARC
The users account must be configured using the CARC module system. The module systems sets up the users computing environment, including the compilers and libraries. The module system provides a lot of flexibility in building the BBP software. We have tested the broadband platform on broadband_carc using a specific module configuration using recent gnu compilers, that we describe in these installation notes.
Server Access
One your CARC login account on discovery and endeavour works, you will have access to broadband. Login through endeavour.
- ssh username@endeavour.usc.edu
- ssh broadband
Users can log into endeavour, then just log into 'broadband'.
Adding these server alive options helps extend the time terminal connections will stay open, and reduce the number of times you are logged out of CARC due to inactivity.
ssh -o "ServerAliveInterval 60" endeavour.usc.edu ssh -o "ServerAliveInterval 60" broadband
Disk Quota
- Users have 10GB quota on /home1/username
- Users have 10TB+ quota on /project/scec_608/<username>
We will install broadband software on /project, and configure it to write results to this storage system.
CARC File systems
CARC users have a home directory on the /home/username filesystem. This directory has a quota of about 10GB. To do a Broadband installation, we will need to make use of SCEC storage on the CARC /project/scec_608 filesystem.
To create a broadband installation in on the broadband_carc server, we recommend using the following filesystems:
- /home1/username
This is the users home login account. Edit the .bashrc in this account to setup the required modules and environment variables.
- /project/scec_608/username/scecbbp
This should be the installation directory for bbp. All bbp related software and data (except the .bashrc file) can be found here. Typically, users will log in and move into this directory to run the broadband platform. This is the directory that where the user clones the bbp repo and runs the bbp installation script.
- /project/scec_608/username/scecbbp/bbp_gf
- /project/scec_608/username/scecbbp/bbp_val
- /project/scec_608/username/scecbbp/bbp_data
These are bbp static data (gf, val), and output data (bbp_data). When the bbp platform installation process runs, these where the outputs are writting including logs, temp files, and results.
Anaconda Installation
To support users running broadband platform on computers where we don't have root, we recommend use of Anaconda python, so the software is managed (adding packages) without root access. Retreived command line installer using curl on broadbnad
Then, it will ask you where to install anaconda. We recommend you install it on /project/scec_608/username directory. Don't accept the default installation directory, rather, change it to poing to the project file system. eg: /project/scec_608/maechlin/anaconda3
When anaconada asks to update your .bashrc, say yes.
WARNING:
You currently have a PYTHONPATH environment variable set. This may cause
unexpected behavior when running the Python interpreter in Anaconda3.
For best results, please verify that your PYTHONPATH only points to
directories of packages that are compatible with the Python interpreter
in Anaconda3: /project/scec_608/maechlin/anaconda3
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
It will add a section like this:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/project/maechlin_162/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh" ]; then
. "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/project/maechlin_162/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
Then conda install pyproj
- $condal install pyproj
.bashrc
Add this entries, modified with your username, to your .bashrc
(base) [maechlin@broadband ~]$ more .bashrc # CARC Module Load system module purge module load gcc/8.3.0 openmpi/4.0.1 module load fftw/3.3.8 # Setup BBP Environment export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp/bbp export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data # These help repair an issue on CARC fftw. These point to an Installation done by fsilva. # These are not needed on standard BBP Centos installations export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/comps/PySeismoSoil:$PYTHONPATH export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH export PATH ulimit -s unlimited alias bbp="cd /project/scec_608/maechlin/scecbbp"
Source this .bashrc $ source .bashrc
Then, you can move to the bbp directory using the alias: $ bbp
Using this "bbp" alias can save a lot of typing pathname like this: $ cd /project/scec_608/bbp
Login and retrieve from github
Log into the broadband server, cd to the bbp installation directory, in my case.
- cd /project/scec_608/maechlin/scecbbp
Clone the scecbbp github repository. Be sure to retrieve the dev branch which has been ported to the newest version of the CARC systems.
- $git clone --single-branch --branch dev https://github.com/sceccode/bbp.git
bbp easy install
cd to bbp installation home, where the clone was done.
- $cd /project/scec_608/maechlin/scecbbp/bbp/setup
- $./easy_install_bbp_19.4.0.sh
Install Questions About Regions
The BBP installer will ask you which regions you want to use. Respond using a number 1, or 2. Only one region is required to pass the acceptance tests. Including more regions increases the installation size and time.
Please select what velocity models (regions) you would like to install,
using '1' for Yes, or '2' for No:
==> Would you like to install the Northern California region (27GB)?
1) Yes
2) No
#? y
#? 1
==> Would you like to install the LA Basin region (25GB)
needed for unit and acceptance tests to run?
1) Yes
2) No
#? 1
==> Would you like to install the Central California region (14GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Mojave region (27GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Central Japan region (28GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Western Japan region (28GB)?
1) Yes
2) No
#? 1
=> Installing Broadband Platform Velocity Model Packages
==> Northern California
==> LA Basin
==> Central California
==> Mojave
==> Central Japan
==> Western Japan
==> Completed!
=> Installing Broadband Platform Validation Packages
==> LOMAP
==> Alum Rock
==> NR
==> Whittier Narrows
==> Chino Hills
==> Parkfield
==> San Simeon
==> Landers
==> North Palm Springs
==> Chuetsu
==> Iwate
==> Niigata
==> Tottori
==> GMPEs
==> Completed!
=> All Done!
Please add the following lines to your bash_profile:
export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp/bbp
export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf
export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val
export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data
export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH
ulimit -s unlimited
The BBP installation script directs you to edit your .bashrc. However, if you are following these instructions, you have made the required edits already. Also the .bashrc on CARC should be updated to add these variables to the correct fftw library.
# Add these to .bashrc to help repair an issue on CARC fftw. # These are not needed on standard BBP Centos installations export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib
Compile FFTW
To fix issues with fftw libraries on CARC, we are directing broadband users to replace the makefile for the GP rupture generator, and set appropriate FFT environment variables in order to get link with the correct FFT libraries.
Users should move into the source directory for the GP method in the BBP source files. Using the installation, the correct path in my account is:
$ cd /project/scec_608/maechlin/bbp/scecbbp/bbp/bbp/src/gp/GenRandV5.0 (base) [maechlin@broadband GenRandV5.0]$ pwd /project/scec_608/maechlin/scecbbp/bbp/bbp/src/gp/GenRandV5.0 (base) [maechlin@broadband GenRandV5.0]$ ls defs.h genslip-v5.2.2.c get_rvfrac-gaus.c include.h README srf_gethypo.c test_loop.c fourg.f genslip-v5.4.1.c get_rvfrac-unif.c iofunc.c ruptime.c srf_subs.c test_rhypo.c function.h geoproj_subs.c gslip_sliprate_subs.c makefile slip.c structure.h test_seed.c genslip-v5.0.1.c get_rvfrac.c gslip_srf_subs.c misc.c sliprate_subs.c test_gausrand.c wafront2d-rwg.f
While in this directory, copy this file
$cp /project/scec_608/fsilva/git/BBP/bbp/src/gp/GenRandV5.0/makefile .
Then return to your home directory, and edit your .bashrc to set these two environment variables. Then source the .bashrc
cd ~ vi .bashrc # Add the FFT variables and save file # These help repair an issue on CARC fftw. These are not needed on standard BBP Centos installations export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib source .bashrc
Then return to the root of the bbp source directory structure and type make. An example from my account showning the correct subdirectory where to run the make command is shown here:
(base) [maechlin@broadband bbp]$ ls bbp doc LICENSE README.md setup (base) [maechlin@broadband bbp]$ cd bbp (base) [maechlin@broadband bbp]$ ls comps makefile mod_data plot src tests utils (base) [maechlin@broadband bbp]$ pwd /project/scec_608/maechlin/scecbbp/bbp/bbp (base) [maechlin@broadband bbp]$ make
Run Unit and Acceptance Tests
Once installation completed, log out, log back in, and run the unittests
- $cd /project/scec_608/maechlin/scecbbp/bbp/bbp/tests
- ./UnitTests.py
This runs 40 unit tests which should all pass.
Then run Acceptance Tests. There are 14 but they take longer because they run multiple simulation methods.
- ./AcceptanceTests.py
This includes updated acceptance tests and everything passes now with most of the running time (just over 2 hours) due to the UCSB tests, which are the only ones that didn't benefit from the updated GP low frequency code improvement. $ ./AcceptTests.py ==> Number of tests to run: 14 test_user-EXSIM (__main__.BBPAcceptanceTests) ... Running: mkdir -p /home/broadband-03/fsilva/bbp/bbp_data_fas/logs/acceptance_test_logs ok test_user-GP (__main__.BBPAcceptanceTests) ... ok test_user-IRIKURA_RECIPE_M1 (__main__.BBPAcceptanceTests) ... ok test_user-IRIKURA_RECIPE_M2 (__main__.BBPAcceptanceTests) ... ok test_user-SDSU (__main__.BBPAcceptanceTests) ... ok test_user-SONG (__main__.BBPAcceptanceTests) ... ok test_user-UCSB (__main__.BBPAcceptanceTests) ... ok test_valid-northridge-EXSIM (__main__.BBPAcceptanceTests) ... ok test_valid-northridge-GP (__main__.BBPAcceptanceTests) ... ok test_valid-northridge-IRIKURA_RECIPE_M1 (__main__.BBPAcceptanceTests) ... ok test_valid-northridge-IRIKURA_RECIPE_M2 (__main__.BBPAcceptanceTests) ... ok test_valid-northridge-SDSU (__main__.BBPAcceptanceTests) ... ok test_valid-northridge-SONG (__main__.BBPAcceptanceTests) ... ok test_valid-northridge-UCSB (__main__.BBPAcceptanceTests) ... ok ---------------------------------------------------------------------- Ran 14 tests in 7620.925s OK Once these UnitTests and Acceptance tests pass, the broadband is correctly installed and ready for research use.
Command Line
Output Files
The Broadband Platform writes all it output files, including calculated ground motions, as well as a temp directory, and log files, the BBP_DATA directory. You can find where that is defined above, or by typing: env | grep BBP
Each "job" is assigned a simulation ID. This ID is assigned to all the data products associated with the simulation done. Sim ID and then data type is a common output file name. Images are identified as *.png files.
