Difference between revisions of "CARC BBP Setup"

From SCECpedia
Jump to navigationJump to search
 
(43 intermediate revisions by the same user not shown)
Line 2: Line 2:
  
 
== Installing Broadband on CARC ==
 
== Installing Broadband on CARC ==
 +
There are three alternative BBP installations at CARC.
 +
(1) Installation in Discovery (standard CARC queues)
 +
(2) Installation on Endeavor (SCEC queueu)
 +
(3) broadband_carc (dedicated server)
 +
 +
== Default Installation - Discovery ==
 
The users account must be configured using the CARC module system. The module systems sets up the users computing environment, including the compilers and libraries. The module system provides a lot of flexibility in building the BBP software. We have tested the broadband platform on broadband_carc using a specific module configuration using recent gnu compilers, that we describe in these installation notes.
 
The users account must be configured using the CARC module system. The module systems sets up the users computing environment, including the compilers and libraries. The module system provides a lot of flexibility in building the BBP software. We have tested the broadband platform on broadband_carc using a specific module configuration using recent gnu compilers, that we describe in these installation notes.
  
−
== Server Access ==
+
These server alive options will help extend the time terminal connections will stay open, and reduce the number of times you are logged out of CARC due to inactivity.
 +
<pre>
 +
ssh -o "ServerAliveInterval 60" discovery.usc.edu
 +
</pre>
 +
 
 +
 
 +
== Endeavour and bbp_carc Access ==
 
One your CARC login account on discovery and endeavour works, you will have access to broadband. Login through endeavour.
 
One your CARC login account on discovery and endeavour works, you will have access to broadband. Login through endeavour.
 
* ssh username@endeavour.usc.edu
 
* ssh username@endeavour.usc.edu
 
* ssh broadband
 
* ssh broadband
 +
 +
Users can log into endeavour, then just log into 'broadband'.
 +
 +
Adding these server alive options helps extend the time terminal connections will stay open, and reduce the number of times you are logged out of CARC due to inactivity.
 +
<pre>
 +
ssh -o "ServerAliveInterval 60" endeavour.usc.edu
 +
ssh -o "ServerAliveInterval 60" broadband
 +
</pre>
  
 
== Disk Quota ==
 
== Disk Quota ==
 
* Users have 10GB quota on /home1/username
 
* Users have 10GB quota on /home1/username
−
* Users have 1TB+ quota on /project/scec_608/<username>
+
* Users have 10TB+ quota on /project/scec_608/<username>
  
 
We will install broadband software on /project, and configure it to write results to this storage system.
 
We will install broadband software on /project, and configure it to write results to this storage system.
  
 
== CARC File systems ==
 
== CARC File systems ==
−
CARC users have a home directory on the /home/username filesystem. This directory has a quota of about 10GB. To do a Broadband installation, we will need to make use of SCEC storage on the CARC /project/scec_608 filesystem.
+
CARC users have a home directory on the /home1/username filesystem. This directory has a quota of about 10GB. To do a Broadband installation, we will need to make use of SCEC storage on the CARC /project/scec_608 filesystem.
  
 
To create a broadband installation in on the broadband_carc server, we recommend using the following filesystems:
 
To create a broadband installation in on the broadband_carc server, we recommend using the following filesystems:
−
*/home1/username
+
;/home1/username
−
This is the users home login account. Edit the .bashrc in this account to setup the required modules and environment variables.
+
:This is the users home login account. Edit the .bashrc in this account to setup the required modules and environment variables.
  
−
*/project/scec_608/username/scecbbp
+
;/project/scec_608/username/scecbbp
−
This should be the installation directory for bbp. All bbp related software and data (except the .bashrc file) can be found here. Typically, users will log in and move into this directory to run the broadband platform. This is the directory that where the user clones the bbp repo and runs the bbp installation script.
+
:This should be the installation directory for bbp. All bbp related software and data (except the .bashrc file) can be found here. Typically, users will log in and move into this directory to run the broadband platform. This is the directory that where the user clones the bbp repo and runs the bbp installation script.
  
−
*/project/scec_608/username/scecbbp/bbp_gf
+
;/project/scec_608/username/scecbbp/bbp_gf
−
*/project/scec_608/username/scecbbp/bbp_val
+
;/project/scec_608/username/scecbbp/bbp_val
−
*/project/scec_608/username/scecbbp/bbp_data
+
;/project/scec_608/username/scecbbp/bbp_data
−
These are bbp static data (gf, val), and output data (bbp_data). When the bbp platform installation process runs, these where the outputs are writting including logs, temp files, and results.
+
:These are bbp static data (gf, val), and output data (bbp_data). When the bbp platform installation process runs, these where the outputs are writting including logs, temp files, and results.
  
 
== Anaconda Installation ==
 
== Anaconda Installation ==
 
To support users running broadband platform on computers where we don't have root, we recommend use of Anaconda python, so the software is managed (adding packages) without root access.
 
To support users running broadband platform on computers where we don't have root, we recommend use of Anaconda python, so the software is managed (adding packages) without root access.
 
Retreived command line installer using curl on broadbnad
 
Retreived command line installer using curl on broadbnad
−
*curl -O https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh
+
*curl -O https://repo.anaconda.com/archive/Anaconda3-2024.02-1-Linux-x86_64.sh
  
 
Then, it will ask you where to install anaconda. We recommend you install it on /project/scec_608/username directory. Don't accept the default installation directory, rather, change it to poing to the project file system.
 
Then, it will ask you where to install anaconda. We recommend you install it on /project/scec_608/username directory. Don't accept the default installation directory, rather, change it to poing to the project file system.
 
eg: /project/scec_608/maechlin/anaconda3
 
eg: /project/scec_608/maechlin/anaconda3
  
−
== .bashrc ==
+
When anaconada asks to update your .bashrc, say yes.  
−
Add this entries, modified with your username, to your .bashrc
 
 
<pre>
 
<pre>
−
(base) [maechlin@broadband ~]$ more .bashrc
+
WARNING:
−
# .bashrc
+
    You currently have a PYTHONPATH environment variable set. This may cause
 +
    unexpected behavior when running the Python interpreter in Anaconda3.
 +
    For best results, please verify that your PYTHONPATH only points to
 +
    directories of packages that are compatible with the Python interpreter
 +
    in Anaconda3: /project/scec_608/maechlin/anaconda3
 +
Do you wish the installer to initialize Anaconda3
 +
by running conda init? [yes|no]
 +
</pre>
 +
 
 +
It will add a section like this:
  
−
# Source global definitions
+
<pre>
−
if [ -f /etc/bashrc ]; then
+
# >>> conda initialize >>>
−
. /etc/bashrc
+
# !! Contents within this block are managed by 'conda init' !!
 +
__conda_setup="$('/project/maechlin_162/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
 +
if [ $? -eq 0 ]; then
 +
    eval "$__conda_setup"
 +
else
 +
    if [ -f "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh" ]; then
 +
        . "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh"
 +
    else
 +
        export PATH="/project/maechlin_162/anaconda3/bin:$PATH"
 +
    fi
 
fi
 
fi
 +
unset __conda_setup
 +
# <<< conda initialize <<<
 +
</pre>
  
−
# Uncomment the following line if you don't like systemctl's auto-paging feature:
+
Then, you need to add pyproj to anaconda3. Do this by running conda install pyproj like this:
−
# export SYSTEMD_PAGER=
+
* $conda install pyproj
  
 +
== .bashrc ==
 +
These directions assume your default shell on CARC is a bash shell. To configure your CARC environment to run the BBP software, add these entries, modified with your username, to your .bashrc
 +
<pre>
 +
(base) [maechlin@broadband ~]$ more .bashrc
 
# CARC Module Load system
 
# CARC Module Load system
 
module purge
 
module purge
−
module load gcc/8.3.0
+
module load gcc/8.3.0 openmpi/4.0.1
 +
module load fftw/3.3.8
  
 
# Setup BBP Environment
 
# Setup BBP Environment
−
export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp
+
export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp/bbp
 
export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf
 
export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf
 
export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val
 
export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val
 
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data
 
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data
 +
 +
# These help repair an issue on CARC fftw. These point to an Installation done by fsilva.
 +
# These are not needed on standard BBP Centos installations
 +
export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include
 +
export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib
 +
 +
export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/comps/PySeismoSoil:$PYTHONPATH
 +
 +
export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH
 +
 +
# Increase the max stack size to unlimited due to recursion in bbtoolbox raytracer
 +
ulimit -s unlimited
 +
 +
# Define an alias to the top level bbp directory.
 +
# By convention, we call this directory: scecbbp
 +
# All bbp files including source code, GF, output data
 +
# are located in subdirectories of this top level bbp directory
 
alias bbp="cd /project/scec_608/maechlin/scecbbp"
 
alias bbp="cd /project/scec_608/maechlin/scecbbp"
 +
</pre>
  
−
export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:$PYTHONPATH
+
Source this .bashrc
 +
$ source .bashrc
 +
 
 +
Then, you can move to the bbp directory using the alias:
 +
$ bbp
 +
 
 +
Using this "bbp" alias can save a lot of typing pathname like this:
 +
$ cd /project/scec_608/scecbbp
 +
 
 +
== Login and retrieve from github ==
 +
Log into the broadband server, cd to the bbp installation directory, in my case.
 +
*cd /project/scec_608/maechlin/scecbbp
 +
 
 +
Clone the scecbbp github repository. Be sure to retrieve the dev branch which has been ported to the newest version of the CARC systems.
 +
* $git clone --single-branch --branch dev https://github.com/sceccode/bbp.git
 +
 
 +
== bbp easy install ==
 +
cd to bbp installation home, where the clone was done.
 +
* $cd /project/scec_608/maechlin/scecbbp/bbp/setup
 +
* $./easy_install_bbp_19.4.0.sh
 +
 
 +
== Install Questions About Regions ==
 +
The BBP installer will ask you which regions you want to use. Respond using a number 1, or 2. Only one region is required to pass the acceptance tests. Including more regions increases the installation size and time.
 +
<pre>
 +
Please select what velocity models (regions) you would like to install,
 +
using '1' for Yes, or '2' for No:
 +
 
 +
==> Would you like to install the Northern California region (27GB)?
 +
1) Yes
 +
2) No
 +
#? y
 +
#? 1
 +
==> Would you like to install the LA Basin region (25GB)
 +
    needed for unit and acceptance tests to run?
 +
1) Yes
 +
2) No
 +
#? 1
 +
==> Would you like to install the Central California region (14GB)?
 +
1) Yes
 +
2) No
 +
#? 1
 +
==> Would you like to install the Mojave region (27GB)?
 +
1) Yes
 +
2) No
 +
#? 1
 +
==> Would you like to install the Central Japan region (28GB)?
 +
1) Yes
 +
2) No
 +
#? 1
 +
==> Would you like to install the Western Japan region (28GB)?
 +
1) Yes
 +
2) No
 +
#? 1
 +
=> Installing Broadband Platform Velocity Model Packages
 +
==> Northern California
 +
==> LA Basin
 +
==> Central California
 +
==> Mojave
 +
==> Central Japan
 +
==> Western Japan
 +
==> Completed!
 +
=> Installing Broadband Platform Validation Packages
 +
==> LOMAP
 +
==> Alum Rock
 +
==> NR
 +
==> Whittier Narrows
 +
==> Chino Hills
 +
==> Parkfield
 +
==> San Simeon
 +
==> Landers
 +
==> North Palm Springs
 +
==> Chuetsu
 +
==> Iwate
 +
==> Niigata
 +
==> Tottori
 +
==> GMPEs
 +
==> Completed!
 +
 
 +
=> All Done!
 +
 
 +
Please add the following lines to your bash_profile:
 +
 
 +
export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp/bbp
 +
export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf
 +
export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val
 +
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data
 +
export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/comps/PySeismoSoil:$PYTHONPATH
 
export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH
 
export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH
 
ulimit -s unlimited
 
ulimit -s unlimited
−
export PATH
+
</pre>
 +
 
 +
These instructions from the BBP installation script direct you to edit your .bashrc. However, the bbp installation script was does not include updated required to work properly on USC CARC.
 +
These instruction give in this wiki entry are more up-to-date for CARC than the directions in the bbp installation script. If you are following these instructions on this wiki, you have made the required .bashrc edits already. Also the .bashrc on CARC should be updated to add these variables to the correct fftw library.
 +
<pre>
 +
# Add these to .bashrc to help repair an issue on CARC fftw.
 +
# These are not needed on standard BBP Centos installations
 +
export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include
 +
export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib
 +
</pre>
 +
 
 +
== Compile FFTW ==
 +
To fix issues with fftw libraries on CARC, we are directing broadband users to replace the makefile for the GP rupture generator, and set appropriate FFT environment variables in order to get link with the correct FFT libraries.
 +
 
 +
Users should move into the source directory for the GP method in the BBP source files.
 +
Using the installation, the correct path in my account is:
 +
<pre>
 +
$ cd /project/scec_608/maechlin/bbp/scecbbp/bbp/bbp/src/gp/GenRandV5.0
  
 +
(base) [maechlin@broadband GenRandV5.0]$ pwd
 +
/project/scec_608/maechlin/scecbbp/bbp/bbp/src/gp/GenRandV5.0
 +
(base) [maechlin@broadband GenRandV5.0]$ ls
 +
defs.h            genslip-v5.2.2.c  get_rvfrac-gaus.c      include.h  README          srf_gethypo.c    test_loop.c
 +
fourg.f          genslip-v5.4.1.c  get_rvfrac-unif.c      iofunc.c  ruptime.c        srf_subs.c      test_rhypo.c
 +
function.h        geoproj_subs.c    gslip_sliprate_subs.c  makefile  slip.c          structure.h      test_seed.c
 +
genslip-v5.0.1.c  get_rvfrac.c      gslip_srf_subs.c      misc.c    sliprate_subs.c  test_gausrand.c  wafront2d-rwg.f
 
</pre>
 
</pre>
  
−
== Login and retrieve from github ==
+
While in this directory, copy this file
−
Be sure to retrieve the dev branch which has been ported to the newest version of the CARC systems.
+
<pre>
 +
$cp /project/scec_608/fsilva/git/BBP/bbp/src/gp/GenRandV5.0/makefile .
 +
</pre>
 +
 
 +
Then return to your home directory, and edit your .bashrc to set these two environment variables. Then source the .bashrc
 +
 
 +
<pre>
 +
cd ~
 +
vi .bashrc
 +
# Add the FFT variables and save file
 +
# These help repair an issue on CARC fftw. These are not needed on standard BBP Centos installations
 +
export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include
 +
export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib
 +
source .bashrc
 +
</pre>
 +
 
 +
Then return to the root of the bbp source directory structure and type make. An example from my account showning the correct subdirectory where to run the make command is shown here:
 +
<pre>
 +
(base) [maechlin@broadband bbp]$ ls
 +
bbp  doc  LICENSE  README.md  setup
 +
(base) [maechlin@broadband bbp]$ cd bbp
 +
(base) [maechlin@broadband bbp]$ ls
 +
comps  makefile  mod_data  plot  src  tests  utils
 +
(base) [maechlin@broadband bbp]$ pwd
 +
/project/scec_608/maechlin/scecbbp/bbp/bbp
 +
(base) [maechlin@broadband bbp]$ make
 +
</pre>
 +
 
 +
== Run Unit and Acceptance Tests ==
 +
Once installation completed, log out, log back in, and run the unittests
 +
* $cd /project/scec_608/maechlin/scecbbp/bbp/bbp/tests
 +
* ./UnitTests.py
 +
 
 +
This runs 40 unit tests which should all pass.
 +
<pre>
 +
 
 +
(base) [maechlin@broadband tests]$ ./UnitTests.py
 +
test_runprog (test_bband_utils.TestBBandUtils)
 +
Test runprog function ... ok
 +
test_runprog2 (test_bband_utils.TestBBandUtils)
 +
Another test for the runprog function ... ok
 +
test_runprog3 (test_bband_utils.TestBBandUtils)
 +
Yet another test for the runprog function ... ok
 +
test_execute_platform_bbp (test_python_code.TestPythonCode)
 +
Run Broadband Plotform to make sure we can start it ... ok
 +
test_python_code_comps (test_python_code.TestPythonCode)
 +
Run Python with -tt flag to detect mix of tabs and spaces in the code ... ok
 +
test_python_code_tests (test_python_code.TestPythonCode)
 +
Run Python with -tt flag to detect mix of tabs and spaces in the code ... ok
 +
test_arias_duration (test_arias.TestArias)
 +
Run the arias intensity unit test ... ok
 +
test_bbp2peer (test_bbp_format.TestBBPFormat)
 +
Test for the bbp2peer converter ... ok
 +
test_exsim2bbp (test_bbp_format.TestBBPFormat)
 +
Test for the exsim2bbp converter ... ok
 +
test_peer2bbp (test_bbp_format.TestBBPFormat)
 +
Test for the peer2bbp converter ... ok
 +
test_pynga_ngae (test_pynga.TestPyNGA)
 +
Test PyNGA NGAE ... ok
 +
test_pynga_ngaw1 (test_pynga.TestPyNGA)
 +
Test PyNGA NGAW1 ... /project/scec_608/maechlin/scecbbp/bbp/bbp/comps/pynga/__init__.py:252: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
 +
  values = np.array(values)
 +
ok
 +
test_pynga_ngaw2 (test_pynga.TestPyNGA)
 +
Test PyNGA NGAW2 ... ok
 +
test_vm2ucsb (test_vm2vm.TestVm2vm)
 +
input a GP format file and get out a UCSB format file ... ok
 +
test_vm2ucsb_nga (test_vm2vm.TestVm2vm)
 +
input a GP format file and get out a UCSB format file ... ok
 +
test_vm2vm (test_vm2vm.TestVm2vm)
 +
input a GP format file and get out a SDSU format file ... ok
 +
test_vm2vm_nga (test_vm2vm.TestVm2vm)
 +
input a GP format file and get out a SDSU format file ... ok
 +
test_xy2ll (test_cc.TestCC)
 +
ll2xy mlon=-118 mlat=34 xazim=0 < stats.ll > stats_out.xy ... ok
 +
test_genslip (test_genslip.TestGenslip)
 +
Test GP rupture generator ... ok
 +
test_jbsim (test_jbsim.TestJbsim)
 +
Run Jbsim module ... ok
 +
test_hfsims (test_hfsims.TestHfsims)
 +
Test GP HFSims code ... ok
 +
test_wcc_siteamp (test_wcc_siteamp.TestWccSiteamp)
 +
Test GP site response module ... ok
 +
test_match (test_match.TestMatch)
 +
Test GP match code ... ok
 +
test_gensrf (test_gensrf.TestGenSRF)
 +
Test Irikura rupture generator ... ok
 +
test_irikura_hf (test_irikura_hf.TestIrikuraHF)
 +
Test Irikura HF code ... ok
 +
test_uc_fault_utils (test_uc_fault_utils.TestUCFaultUtils)
 +
Test UCSB fault utilities ... ok
 +
test_ucgen (test_ucrmg.TestUCrmg)
 +
Unit test for the UCSB rupture generator ... ok
 +
test_syn1d (test_syn1d.TestSyn1D) ... ok
 +
test_uc_site (test_uc_site.TestUCSite) ... ok
 +
test_bbtoolbox (test_bbtoolbox.TestBBToolbox)
 +
Test SDSU BBToolbox code ... ok
 +
test_exsim (test_exsim.TestExsim)
 +
Run ExSIM module ... ok
 +
test_rmg (test_rmg.TestRMG) ... ok
 +
test_rotd50 (test_rotd50.TestRotD50)
 +
Test the rotd50 module ... ok
 +
test_rotd100 (test_rotd100.TestRotD100)
 +
Test the rotd100 module ... ok
 +
test_gof (test_gp_gof.TestGPGof)
 +
Test GP GOF Code ... ok
 +
test_mogof_stat (test_sdsu_mogof.TestSDSUMOGof) ... ok
 +
test_anderson_gof (test_anderson_gof.TestAndersonGof)
 +
Run the Anderson GOF test ... ok
 +
test_rzz2015 (test_rzz2015.TestRZZ2015)
 +
Run the RZZ2015 GOF test ... ok
 +
test_as16 (test_as16.TestAS16)
 +
Run the AS16 GMPE test ... ok
 +
test_as16_testcases (test_as16.TestAS16)
 +
Run the AS16 test suite ... ok
 +
 
 +
----------------------------------------------------------------------
 +
Ran 40 tests in 588.206s
 +
 
 +
OK
 +
</pre>
 +
 
 +
 
 +
Then run Acceptance Tests. There are 14 but they take longer because they run multiple simulation methods.
 +
*./AcceptanceTests.py
 +
 
 +
<pre>
 +
This includes updated acceptance tests and everything passes now with most of the running time (just over 2 hours) due to the UCSB tests, which are the only ones that didn't benefit from the updated GP low frequency code improvement.
 +
 
 +
$ ./AcceptTests.py
 +
==> Number of tests to run: 14
 +
test_user-EXSIM (__main__.BBPAcceptanceTests) ... Running: mkdir -p /home/broadband-03/fsilva/bbp/bbp_data_fas/logs/acceptance_test_logs
 +
ok
 +
test_user-GP (__main__.BBPAcceptanceTests) ... ok
 +
test_user-IRIKURA_RECIPE_M1 (__main__.BBPAcceptanceTests) ... ok
 +
test_user-IRIKURA_RECIPE_M2 (__main__.BBPAcceptanceTests) ... ok
 +
test_user-SDSU (__main__.BBPAcceptanceTests) ... ok
 +
test_user-SONG (__main__.BBPAcceptanceTests) ... ok
 +
test_user-UCSB (__main__.BBPAcceptanceTests) ... ok
 +
test_valid-northridge-EXSIM (__main__.BBPAcceptanceTests) ... ok
 +
test_valid-northridge-GP (__main__.BBPAcceptanceTests) ... ok
 +
test_valid-northridge-IRIKURA_RECIPE_M1 (__main__.BBPAcceptanceTests) ... ok
 +
test_valid-northridge-IRIKURA_RECIPE_M2 (__main__.BBPAcceptanceTests) ... ok
 +
test_valid-northridge-SDSU (__main__.BBPAcceptanceTests) ... ok
 +
test_valid-northridge-SONG (__main__.BBPAcceptanceTests) ... ok
 +
test_valid-northridge-UCSB (__main__.BBPAcceptanceTests) ... ok
 +
 
 +
----------------------------------------------------------------------
 +
Ran 14 tests in 7620.925s
 +
 
 +
OK
 +
 
 +
Once these UnitTests and Acceptance tests pass, the broadband is correctly installed and ready for research use.
 +
</pre>
 +
 
 +
 
 +
== Command Line  ==
 +
The main executable for the Broadband Platform is a command line script called: run_bbp.py
 +
 
 +
When this top level script is run, it asks the users a series of questions to help them configure a BBP simulation. More advanced users will make use of bbp's script capiabilities which are described in the bbp users manual.
 +
 
 +
<pre>
 +
 
 +
(base) [maechlin@broadband scecbbp]$ run_bbp.py
 +
Welcome to the SCEC Broadband Platform version 19.4.0.
 +
================================================================================
 +
 
 +
Please select the Broadband Platform mode of operation:
 +
  * Validation - Simulates a historical event
 +
  * Scenario  - Runs a user-defined hypothetical event
 +
 
 +
Do you want to perform a validation simulation (y/n)? y
 +
================================================================================
 +
 
 +
Please select a validation event from the list below:
 +
 
 +
(1) Chino Hills
 +
(2) GMPE 5.5rv NoCal 20km --> velocity model NOCAL500 is missing!
 +
(3) GMPE 5.5rv NoCal 50km --> velocity model NOCAL500 is missing!
 +
(4) GMPE 5.5rv SoCal 20km
 +
(5) GMPE 5.5rv SoCal 50km
 +
(6) GMPE 6.2ss NoCal 20km --> velocity model NOCAL500 is missing!
 +
(7) GMPE 6.2ss NoCal 50km --> velocity model NOCAL500 is missing!
 +
(8) GMPE 6.2ss SoCal 20km
 +
(9) GMPE 6.2ss SoCal 50km
 +
(10) GMPE 6.6rv NoCal 20km --> velocity model NOCAL500 is missing!
 +
(11) GMPE 6.6rv NoCal 50km --> velocity model NOCAL500 is missing!
 +
(12) GMPE 6.6rv SoCal 20km
 +
(13) GMPE 6.6rv SoCal 50km
 +
(14) GMPE 6.6ss NoCal 20km --> velocity model NOCAL500 is missing!
 +
(15) GMPE 6.6ss NoCal 50km --> velocity model NOCAL500 is missing!
 +
(16) GMPE 6.6ss SoCal 20km
 +
(17) GMPE 6.6ss SoCal 50km
 +
(18) NR
 +
(19) Whittier
 +
? 1
 +
================================================================================
 +
 
 +
The Broadband Platform includes several scientific methods that can be used to calculate synthetic seismograms.
 +
 
 +
Choose a Method to use in this Broadband validation simulation:
 +
(1) GP (Graves & Pitarka)
 +
(2) UCSB
 +
(3) SDSU
 +
(4) EXSIM
 +
(5) Song
 +
(6) Irikura Recipe Method 1 (Irikura1)
 +
(7) Irikura Recipe Method 2 (Irikura2)
 +
? 1
 +
</pre>
 +
 
 +
The messages above about the missing NOCAL500 velocity model are due to incomplete installation in my account on CARC. These are warnings only, and will not affect the other validation results.
 +
 
 +
== Output Files ==
 +
The Broadband Platform writes all it output files, including calculated ground motions, as well as a temp directory, and log files, to the BBP_DATA directory. You can find where that directory is located a couple of ways. It is defined above, or you can type:
 +
<pre>
 +
(base) [maechlin@broadband tests]$ env | grep BBP_DATA
 +
BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data
 +
 
 +
$cd /project/scec_608/maechlin/scecbbp/bbp_data
 +
 
 +
(base) [maechlin@broadband bbp_data]$ pwd
 +
/project/scec_608/maechlin/scecbbp/bbp_data
 +
(base) [maechlin@broadband bbp_data]$ ls
 +
indata  logs  outdata  run  tmpdata  xm
 +
</pre>
 +
 
 +
BBP Simulation results are found in the outdata directory. All output files produced by the run_bbp.py script will be written to a directory with a unique SimID.
 +
 
 +
<pre>
 +
 
 +
(base) [maechlin@broadband bbp_data]$ ls
 +
indata  logs  outdata  run  tmpdata  xml
 +
(base) [maechlin@broadband bbp_data]$ cd outdata
 +
(base) [maechlin@broadband outdata]$ ls
 +
9155471  9155528  9155638  9155808  9156040  9156639  9156652  9156904  9156938  9157297  9157343  9157477  9157615  9157865
 +
9155472  9155532  9155670  9155811  9156063  9156641  9156679  9156905  9157141  9157332  9157346  9157572  9157619  9157866
 +
9155506  9155535  9155765  9155814  9156064  9156645  9156754  9156920  9157149  9157333  9157347  9157598  9157630
 +
9155508  9155537  9155792  9155827  9156604  9156648  9156783  9156922  9157172  9157336  9157373  9157599  9157834
 +
9155524  9155563  9155794  9156031  9156637  9156651  9156878  9156926  9157173  9157340  9157448  9157613  9157842
 +
</pre>
 +
 
 +
Each Broadband Platform "simulation" is assigned a simulation ID. This ID is assigned to all the data products associated with the simulation. Sim ID and then data type is a common output file name. Seismograms are output as *.bbp files. Images are identified as *.png files.
 +
 
 +
== Setting up SSH Keys ==
 +
 
 +
To set up a pair of ssh keys in order, to use the broadband server, is how to use the server. To set this up, first log into endeavour.usc.edu and use the ssh-keygen command to create your keys. You should end up with a pair of public/private keys on your .ssh folder on endeavour.  
  
−
$git clone --single-branch --branch dev https://github.com/sceccode/bbp.git
+
Copy the public key to a file called "authorized_keys" in the same .ssh folder on endeavour.  
  
 +
Then log out of endeavour server, and log back into endeavour, and you should be able to ssh into the broadband server again.
  
 +
More detailed technical description of SSH is posted here:
 +
*[https://serverpilot.io/docs/how-to-use-ssh-public-key-authentication/ SSH Info]
  
 
== Related Entries ==
 
== Related Entries ==
 
*[[Broadband Platform]]
 
*[[Broadband Platform]]
 
*[[SCEC Software]]
 
*[[SCEC Software]]

Latest revision as of 19:53, 12 March 2024

SCEC has migrated our BBP server from University Park server room to USC's new research Computing group CARC.

Installing Broadband on CARC

There are three alternative BBP installations at CARC. (1) Installation in Discovery (standard CARC queues) (2) Installation on Endeavor (SCEC queueu) (3) broadband_carc (dedicated server)

Default Installation - Discovery

The users account must be configured using the CARC module system. The module systems sets up the users computing environment, including the compilers and libraries. The module system provides a lot of flexibility in building the BBP software. We have tested the broadband platform on broadband_carc using a specific module configuration using recent gnu compilers, that we describe in these installation notes.

These server alive options will help extend the time terminal connections will stay open, and reduce the number of times you are logged out of CARC due to inactivity. 

ssh -o "ServerAliveInterval 60" discovery.usc.edu


Endeavour and bbp_carc Access

One your CARC login account on discovery and endeavour works, you will have access to broadband. Login through endeavour.

  • ssh username@endeavour.usc.edu
  • ssh broadband

Users can log into endeavour, then just log into 'broadband'.

Adding these server alive options helps extend the time terminal connections will stay open, and reduce the number of times you are logged out of CARC due to inactivity. 

ssh -o "ServerAliveInterval 60" endeavour.usc.edu
ssh -o "ServerAliveInterval 60" broadband

Disk Quota

  • Users have 10GB quota on /home1/username
  • Users have 10TB+ quota on /project/scec_608/<username>

We will install broadband software on /project, and configure it to write results to this storage system.

CARC File systems

CARC users have a home directory on the /home1/username filesystem. This directory has a quota of about 10GB. To do a Broadband installation, we will need to make use of SCEC storage on the CARC /project/scec_608 filesystem.

To create a broadband installation in on the broadband_carc server, we recommend using the following filesystems:

/home1/username
This is the users home login account. Edit the .bashrc in this account to setup the required modules and environment variables.
/project/scec_608/username/scecbbp
This should be the installation directory for bbp. All bbp related software and data (except the .bashrc file) can be found here. Typically, users will log in and move into this directory to run the broadband platform. This is the directory that where the user clones the bbp repo and runs the bbp installation script.
/project/scec_608/username/scecbbp/bbp_gf
/project/scec_608/username/scecbbp/bbp_val
/project/scec_608/username/scecbbp/bbp_data
These are bbp static data (gf, val), and output data (bbp_data). When the bbp platform installation process runs, these where the outputs are writting including logs, temp files, and results.

Anaconda Installation

To support users running broadband platform on computers where we don't have root, we recommend use of Anaconda python, so the software is managed (adding packages) without root access. Retreived command line installer using curl on broadbnad

Then, it will ask you where to install anaconda. We recommend you install it on /project/scec_608/username directory. Don't accept the default installation directory, rather, change it to poing to the project file system. eg: /project/scec_608/maechlin/anaconda3

When anaconada asks to update your .bashrc, say yes. 

WARNING:
    You currently have a PYTHONPATH environment variable set. This may cause
    unexpected behavior when running the Python interpreter in Anaconda3.
    For best results, please verify that your PYTHONPATH only points to
    directories of packages that are compatible with the Python interpreter
    in Anaconda3: /project/scec_608/maechlin/anaconda3
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]

It will add a section like this: 

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/project/maechlin_162/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/project/maechlin_162/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/project/maechlin_162/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Then, you need to add pyproj to anaconda3. Do this by running conda install pyproj like this:

  • $conda install pyproj

.bashrc

These directions assume your default shell on CARC is a bash shell. To configure your CARC environment to run the BBP software, add these entries, modified with your username, to your .bashrc 

(base) [maechlin@broadband ~]$ more .bashrc
# CARC Module Load system
module purge
module load gcc/8.3.0 openmpi/4.0.1
module load fftw/3.3.8

# Setup BBP Environment
export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp/bbp
export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf
export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data

# These help repair an issue on CARC fftw. These point to an Installation done by fsilva.
# These are not needed on standard BBP Centos installations
export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include
export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib

export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/comps/PySeismoSoil:$PYTHONPATH

export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH

# Increase the max stack size to unlimited due to recursion in bbtoolbox raytracer
ulimit -s unlimited

# Define an alias to the top level bbp directory.
# By convention, we call this directory: scecbbp 
# All bbp files including source code, GF, output data 
# are located in subdirectories of this top level bbp directory
alias bbp="cd /project/scec_608/maechlin/scecbbp"

Source this .bashrc $ source .bashrc

Then, you can move to the bbp directory using the alias: $ bbp

Using this "bbp" alias can save a lot of typing pathname like this: $ cd /project/scec_608/scecbbp

Login and retrieve from github

Log into the broadband server, cd to the bbp installation directory, in my case.

  • cd /project/scec_608/maechlin/scecbbp

Clone the scecbbp github repository. Be sure to retrieve the dev branch which has been ported to the newest version of the CARC systems.

bbp easy install

cd to bbp installation home, where the clone was done.

  • $cd /project/scec_608/maechlin/scecbbp/bbp/setup
  • $./easy_install_bbp_19.4.0.sh

Install Questions About Regions

The BBP installer will ask you which regions you want to use. Respond using a number 1, or 2. Only one region is required to pass the acceptance tests. Including more regions increases the installation size and time. 

 Please select what velocity models (regions) you would like to install,
 using '1' for Yes, or '2' for No:

==> Would you like to install the Northern California region (27GB)?
1) Yes
2) No
#? y
#? 1
==> Would you like to install the LA Basin region (25GB)
    needed for unit and acceptance tests to run?
1) Yes
2) No
#? 1
==> Would you like to install the Central California region (14GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Mojave region (27GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Central Japan region (28GB)?
1) Yes
2) No
#? 1
==> Would you like to install the Western Japan region (28GB)?
1) Yes
2) No
#? 1
=> Installing Broadband Platform Velocity Model Packages
==> Northern California
==> LA Basin
==> Central California
==> Mojave
==> Central Japan
==> Western Japan
==> Completed!
=> Installing Broadband Platform Validation Packages
==> LOMAP
==> Alum Rock
==> NR
==> Whittier Narrows
==> Chino Hills
==> Parkfield
==> San Simeon
==> Landers
==> North Palm Springs
==> Chuetsu
==> Iwate
==> Niigata
==> Tottori
==> GMPEs
==> Completed!

=> All Done!

Please add the following lines to your bash_profile:

export BBP_DIR=/project/scec_608/maechlin/scecbbp/bbp/bbp
export BBP_GF_DIR=/project/scec_608/maechlin/scecbbp/bbp_gf
export BBP_VAL_DIR=/project/scec_608/maechlin/scecbbp/bbp_val
export BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data
export PYTHONPATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/comps/PySeismoSoil:$PYTHONPATH
export PATH=/project/scec_608/maechlin/scecbbp/bbp/bbp/comps:/project/scec_608/maechlin/scecbbp/bbp/bbp/utils/batch:$PATH
ulimit -s unlimited

These instructions from the BBP installation script direct you to edit your .bashrc. However, the bbp installation script was does not include updated required to work properly on USC CARC. These instruction give in this wiki entry are more up-to-date for CARC than the directions in the bbp installation script. If you are following these instructions on this wiki, you have made the required .bashrc edits already. Also the .bashrc on CARC should be updated to add these variables to the correct fftw library. 

# Add these to .bashrc to help repair an issue on CARC fftw. 
# These are not needed on standard BBP Centos installations
export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include
export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib

Compile FFTW

To fix issues with fftw libraries on CARC, we are directing broadband users to replace the makefile for the GP rupture generator, and set appropriate FFT environment variables in order to get link with the correct FFT libraries.

Users should move into the source directory for the GP method in the BBP source files. Using the installation, the correct path in my account is: 

$ cd /project/scec_608/maechlin/bbp/scecbbp/bbp/bbp/src/gp/GenRandV5.0

(base) [maechlin@broadband GenRandV5.0]$ pwd
/project/scec_608/maechlin/scecbbp/bbp/bbp/src/gp/GenRandV5.0
(base) [maechlin@broadband GenRandV5.0]$ ls
defs.h            genslip-v5.2.2.c  get_rvfrac-gaus.c      include.h  README           srf_gethypo.c    test_loop.c
fourg.f           genslip-v5.4.1.c  get_rvfrac-unif.c      iofunc.c   ruptime.c        srf_subs.c       test_rhypo.c
function.h        geoproj_subs.c    gslip_sliprate_subs.c  makefile   slip.c           structure.h      test_seed.c
genslip-v5.0.1.c  get_rvfrac.c      gslip_srf_subs.c       misc.c     sliprate_subs.c  test_gausrand.c  wafront2d-rwg.f

While in this directory, copy this file 

$cp /project/scec_608/fsilva/git/BBP/bbp/src/gp/GenRandV5.0/makefile .

Then return to your home directory, and edit your .bashrc to set these two environment variables. Then source the .bashrc 

cd ~
vi .bashrc
# Add the FFT variables and save file
# These help repair an issue on CARC fftw. These are not needed on standard BBP Centos installations
export FFTW_INCDIR=/project/scec_608/fsilva/fftw-3.3.8/include
export FFTW_LIBDIR=/project/scec_608/fsilva/fftw-3.3.8/lib
source .bashrc

Then return to the root of the bbp source directory structure and type make. An example from my account showning the correct subdirectory where to run the make command is shown here: 

(base) [maechlin@broadband bbp]$ ls
bbp  doc  LICENSE  README.md  setup
(base) [maechlin@broadband bbp]$ cd bbp
(base) [maechlin@broadband bbp]$ ls
comps  makefile  mod_data  plot  src  tests  utils
(base) [maechlin@broadband bbp]$ pwd
/project/scec_608/maechlin/scecbbp/bbp/bbp
(base) [maechlin@broadband bbp]$ make

Run Unit and Acceptance Tests

Once installation completed, log out, log back in, and run the unittests

  • $cd /project/scec_608/maechlin/scecbbp/bbp/bbp/tests
  • ./UnitTests.py

This runs 40 unit tests which should all pass. 


(base) [maechlin@broadband tests]$ ./UnitTests.py 
test_runprog (test_bband_utils.TestBBandUtils)
Test runprog function ... ok
test_runprog2 (test_bband_utils.TestBBandUtils)
Another test for the runprog function ... ok
test_runprog3 (test_bband_utils.TestBBandUtils)
Yet another test for the runprog function ... ok
test_execute_platform_bbp (test_python_code.TestPythonCode)
Run Broadband Plotform to make sure we can start it ... ok
test_python_code_comps (test_python_code.TestPythonCode)
Run Python with -tt flag to detect mix of tabs and spaces in the code ... ok
test_python_code_tests (test_python_code.TestPythonCode)
Run Python with -tt flag to detect mix of tabs and spaces in the code ... ok
test_arias_duration (test_arias.TestArias)
Run the arias intensity unit test ... ok
test_bbp2peer (test_bbp_format.TestBBPFormat)
Test for the bbp2peer converter ... ok
test_exsim2bbp (test_bbp_format.TestBBPFormat)
Test for the exsim2bbp converter ... ok
test_peer2bbp (test_bbp_format.TestBBPFormat)
Test for the peer2bbp converter ... ok
test_pynga_ngae (test_pynga.TestPyNGA)
Test PyNGA NGAE ... ok
test_pynga_ngaw1 (test_pynga.TestPyNGA)
Test PyNGA NGAW1 ... /project/scec_608/maechlin/scecbbp/bbp/bbp/comps/pynga/__init__.py:252: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  values = np.array(values)
ok
test_pynga_ngaw2 (test_pynga.TestPyNGA)
Test PyNGA NGAW2 ... ok
test_vm2ucsb (test_vm2vm.TestVm2vm)
input a GP format file and get out a UCSB format file ... ok
test_vm2ucsb_nga (test_vm2vm.TestVm2vm)
input a GP format file and get out a UCSB format file ... ok
test_vm2vm (test_vm2vm.TestVm2vm)
input a GP format file and get out a SDSU format file ... ok
test_vm2vm_nga (test_vm2vm.TestVm2vm)
input a GP format file and get out a SDSU format file ... ok
test_xy2ll (test_cc.TestCC)
ll2xy mlon=-118 mlat=34 xazim=0 < stats.ll > stats_out.xy ... ok
test_genslip (test_genslip.TestGenslip)
Test GP rupture generator ... ok
test_jbsim (test_jbsim.TestJbsim)
Run Jbsim module ... ok
test_hfsims (test_hfsims.TestHfsims)
Test GP HFSims code ... ok
test_wcc_siteamp (test_wcc_siteamp.TestWccSiteamp)
Test GP site response module ... ok
test_match (test_match.TestMatch)
Test GP match code ... ok
test_gensrf (test_gensrf.TestGenSRF)
Test Irikura rupture generator ... ok
test_irikura_hf (test_irikura_hf.TestIrikuraHF)
Test Irikura HF code ... ok
test_uc_fault_utils (test_uc_fault_utils.TestUCFaultUtils)
Test UCSB fault utilities ... ok
test_ucgen (test_ucrmg.TestUCrmg)
Unit test for the UCSB rupture generator ... ok
test_syn1d (test_syn1d.TestSyn1D) ... ok
test_uc_site (test_uc_site.TestUCSite) ... ok
test_bbtoolbox (test_bbtoolbox.TestBBToolbox)
Test SDSU BBToolbox code ... ok
test_exsim (test_exsim.TestExsim)
Run ExSIM module ... ok
test_rmg (test_rmg.TestRMG) ... ok
test_rotd50 (test_rotd50.TestRotD50)
Test the rotd50 module ... ok
test_rotd100 (test_rotd100.TestRotD100)
Test the rotd100 module ... ok
test_gof (test_gp_gof.TestGPGof)
Test GP GOF Code ... ok
test_mogof_stat (test_sdsu_mogof.TestSDSUMOGof) ... ok
test_anderson_gof (test_anderson_gof.TestAndersonGof)
Run the Anderson GOF test ... ok
test_rzz2015 (test_rzz2015.TestRZZ2015)
Run the RZZ2015 GOF test ... ok
test_as16 (test_as16.TestAS16)
Run the AS16 GMPE test ... ok
test_as16_testcases (test_as16.TestAS16)
Run the AS16 test suite ... ok

----------------------------------------------------------------------
Ran 40 tests in 588.206s

OK


Then run Acceptance Tests. There are 14 but they take longer because they run multiple simulation methods.

  • ./AcceptanceTests.py
This includes updated acceptance tests and everything passes now with most of the running time (just over 2 hours) due to the UCSB tests, which are the only ones that didn't benefit from the updated GP low frequency code improvement.

$ ./AcceptTests.py
==> Number of tests to run: 14
test_user-EXSIM (__main__.BBPAcceptanceTests) ... Running: mkdir -p /home/broadband-03/fsilva/bbp/bbp_data_fas/logs/acceptance_test_logs
ok
test_user-GP (__main__.BBPAcceptanceTests) ... ok
test_user-IRIKURA_RECIPE_M1 (__main__.BBPAcceptanceTests) ... ok
test_user-IRIKURA_RECIPE_M2 (__main__.BBPAcceptanceTests) ... ok
test_user-SDSU (__main__.BBPAcceptanceTests) ... ok
test_user-SONG (__main__.BBPAcceptanceTests) ... ok
test_user-UCSB (__main__.BBPAcceptanceTests) ... ok
test_valid-northridge-EXSIM (__main__.BBPAcceptanceTests) ... ok
test_valid-northridge-GP (__main__.BBPAcceptanceTests) ... ok
test_valid-northridge-IRIKURA_RECIPE_M1 (__main__.BBPAcceptanceTests) ... ok
test_valid-northridge-IRIKURA_RECIPE_M2 (__main__.BBPAcceptanceTests) ... ok
test_valid-northridge-SDSU (__main__.BBPAcceptanceTests) ... ok
test_valid-northridge-SONG (__main__.BBPAcceptanceTests) ... ok
test_valid-northridge-UCSB (__main__.BBPAcceptanceTests) ... ok

----------------------------------------------------------------------
Ran 14 tests in 7620.925s

OK

Once these UnitTests and Acceptance tests pass, the broadband is correctly installed and ready for research use.


Command Line

The main executable for the Broadband Platform is a command line script called: run_bbp.py

When this top level script is run, it asks the users a series of questions to help them configure a BBP simulation. More advanced users will make use of bbp's script capiabilities which are described in the bbp users manual. 


(base) [maechlin@broadband scecbbp]$ run_bbp.py
Welcome to the SCEC Broadband Platform version 19.4.0.
================================================================================

Please select the Broadband Platform mode of operation:
   * Validation - Simulates a historical event
   * Scenario   - Runs a user-defined hypothetical event

Do you want to perform a validation simulation (y/n)? y
================================================================================

Please select a validation event from the list below:

(1) Chino Hills
(2) GMPE 5.5rv NoCal 20km --> velocity model NOCAL500 is missing!
(3) GMPE 5.5rv NoCal 50km --> velocity model NOCAL500 is missing!
(4) GMPE 5.5rv SoCal 20km
(5) GMPE 5.5rv SoCal 50km
(6) GMPE 6.2ss NoCal 20km --> velocity model NOCAL500 is missing!
(7) GMPE 6.2ss NoCal 50km --> velocity model NOCAL500 is missing!
(8) GMPE 6.2ss SoCal 20km
(9) GMPE 6.2ss SoCal 50km
(10) GMPE 6.6rv NoCal 20km --> velocity model NOCAL500 is missing!
(11) GMPE 6.6rv NoCal 50km --> velocity model NOCAL500 is missing!
(12) GMPE 6.6rv SoCal 20km
(13) GMPE 6.6rv SoCal 50km
(14) GMPE 6.6ss NoCal 20km --> velocity model NOCAL500 is missing!
(15) GMPE 6.6ss NoCal 50km --> velocity model NOCAL500 is missing!
(16) GMPE 6.6ss SoCal 20km
(17) GMPE 6.6ss SoCal 50km
(18) NR
(19) Whittier
? 1
================================================================================

The Broadband Platform includes several scientific methods that can be used to calculate synthetic seismograms.

Choose a Method to use in this Broadband validation simulation:
(1) GP (Graves & Pitarka)
(2) UCSB
(3) SDSU
(4) EXSIM
(5) Song
(6) Irikura Recipe Method 1 (Irikura1)
(7) Irikura Recipe Method 2 (Irikura2)
? 1

The messages above about the missing NOCAL500 velocity model are due to incomplete installation in my account on CARC. These are warnings only, and will not affect the other validation results.

Output Files

The Broadband Platform writes all it output files, including calculated ground motions, as well as a temp directory, and log files, to the BBP_DATA directory. You can find where that directory is located a couple of ways. It is defined above, or you can type: 

(base) [maechlin@broadband tests]$ env | grep BBP_DATA
BBP_DATA_DIR=/project/scec_608/maechlin/scecbbp/bbp_data

$cd /project/scec_608/maechlin/scecbbp/bbp_data

(base) [maechlin@broadband bbp_data]$ pwd
/project/scec_608/maechlin/scecbbp/bbp_data
(base) [maechlin@broadband bbp_data]$ ls
indata  logs  outdata  run  tmpdata  xm

BBP Simulation results are found in the outdata directory. All output files produced by the run_bbp.py script will be written to a directory with a unique SimID. 


(base) [maechlin@broadband bbp_data]$ ls
indata  logs  outdata  run  tmpdata  xml
(base) [maechlin@broadband bbp_data]$ cd outdata
(base) [maechlin@broadband outdata]$ ls
9155471  9155528  9155638  9155808  9156040  9156639  9156652  9156904  9156938  9157297  9157343  9157477  9157615  9157865
9155472  9155532  9155670  9155811  9156063  9156641  9156679  9156905  9157141  9157332  9157346  9157572  9157619  9157866
9155506  9155535  9155765  9155814  9156064  9156645  9156754  9156920  9157149  9157333  9157347  9157598  9157630
9155508  9155537  9155792  9155827  9156604  9156648  9156783  9156922  9157172  9157336  9157373  9157599  9157834
9155524  9155563  9155794  9156031  9156637  9156651  9156878  9156926  9157173  9157340  9157448  9157613  9157842

Each Broadband Platform "simulation" is assigned a simulation ID. This ID is assigned to all the data products associated with the simulation. Sim ID and then data type is a common output file name. Seismograms are output as *.bbp files. Images are identified as *.png files.

Setting up SSH Keys

To set up a pair of ssh keys in order, to use the broadband server, is how to use the server. To set this up, first log into endeavour.usc.edu and use the ssh-keygen command to create your keys. You should end up with a pair of public/private keys on your .ssh folder on endeavour.

Copy the public key to a file called "authorized_keys" in the same .ssh folder on endeavour.

Then log out of endeavour server, and log back into endeavour, and you should be able to ssh into the broadband server again.

More detailed technical description of SSH is posted here:

Related Entries

Retrieved from "https://strike.scec.org/scecwiki/index.php?title=CARC_BBP_Setup&oldid=28374"