BBP Development

This page lists current Broadband Platform development resources and activities:

Development Infrastructure

Broadband development uses several computation and storage resources at SCEC. Here is a summary of where each component is located:

• Broadband svn: source.usc.edu:/export/data/svn/broadband
• Broadband wiki: scec.usc.edu:/var/www/html/scecwiki
• Broadband trac: northridge.usc.edu:/export/data/trac/broadband
• Broadband release distribution site: hypocenter.usc.edu/research/broadband
• Broadband Fling Study distribution site: /home/broadband-00/fling
• Broadband mailing list: broadband-all@intensity.usc.edu
• Broadband user accounts: scec-00 and scec-01

Everything is backed up nightly on scecvault. The exceptions are:

• Release distribution site (and everything else on hypocenter)
• Mailing lists
• User data on /home/broadband-01 and /home/broadband-02

Development Work

With Broadband version v11.2.2., all unit tests and acceptance tests pass on broadband.usc.edu. However, we get test failures on intensity.usc.edu, and we are working to determine the reasons. Here are the current Unit test results on intensity.

Unit Tests on Intensity

bash-3.2$ ./UnitTests.py > out.txt &                                            
[1] 16243                                                                       
bash-3.2$ test_gensrf (test_genslip.Test_Genslip) ... ok                        
test_bbp_wid (test_jbsim.Test_Jbsim) ... ok                                     
test_bbp_wid (test_hfsims.Test_hfsims) ... ok                                   
test_bbp_wid (test_wcc_siteamp.Test_Wcc_siteamp) ... ok                         
test_bbp_wid (test_match.Test_Match) ... ok                                     
test_respect (test_respect.Test_Respect) ... ok                                 
test_gof (test_urs_gof.Test_URS_Gof) ... ok                                     
test_bbp_wid (test_uc_makefaultg.Test_UC_Makefaultg) ... ok                     
test_ucgen (test_ucrmg.Test_UCrmg) ... ok                                       
test_syn1d (test_syn1d.Test_syn1D) ... FAIL                                     
test_site_amp (test_uc_site.Test_UC_Site) ... FAIL                              
test_respect (test_respect.Test_Respect) ... ok                                 
test_gof (test_urs_gof.Test_URS_Gof) ... ok                                     
test_bbtoolbox (test_bbtoolbox.Test_BBToolbox) ... ok                           
test_amp_fac (test_amp_fac.Test_Amp_Fac) ... ok                                 
test_respect (test_respect.Test_Respect) ... ok                                 
test_gof (test_urs_gof.Test_URS_Gof) ... ok

======================================================================
FAIL: test_syn1d (test_syn1d.Test_syn1D)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/scec-00/maechlin/bbp/i_host/bbp_2g/tests/test_syn1d.py", line 71,in test_syn1d
    self.failIf(cmp_bbp.cmp_bbp(a_ref_file,a_newfile)!=0,errmsg)
AssertionError: Output file does not match reference file: /home/scec-00/maechlin/bbp/i_host/bbp_2g/tmpdata/5222974/s01.3comp

======================================================================
FAIL: test_site_amp (test_uc_site.Test_UC_Site)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/scec-00/maechlin/bbp/i_host/bbp_2g/tests/test_uc_site.py", line 92, in test_site_amp
    self.failIf(cmp_bbp.cmp_bbp(a_ref_file_no_header, test_file_no_header, tolerance=0.035)!=0,errmsg)
AssertionError: Output file /home/scec-00/maechlin/bbp/i_host/bbp_2g/tmpdata/5223105/s04.3comp does not match reference file /home/scec-00/maechlin/bbp/i_host/bbp_2g/ref_data/ucsb/s04.site.3comp

----------------------------------------------------------------------
Ran 17 tests in 2125.133s

FAILED (failures=2)

Top Development Priorities

1. Create wiki entry describing setting random SEED for three High Freq Modules
2. Create "optional" patch distributions separate from official bbp 11.2.2 release
   1. Define acceptance test patch
   2. Define option parsing patch
3. Define wiki entry on how to run one acceptance test to run one at a time
4. Define Graves study (including list of output files) and create wiki entry
5. Define Kottke study (including list of output files) and create wiki entry
6. Setup Graves study using batch tools
7. Setup Kottke study using batch tools
8. Create a wiki for batch tools for broadband platform

Development Activities

1. Modify broadband distribution to use gnu build tools
2. Create minimized software installation distribution for clusters
   1. Remove dependencies on gmt,imagemagic,...
3. Develop prototype distributed broadband queue manager
4. Distribution tracking
   1. Check for updates capability
   2. Download counter for software
   3. Optional registration announcement
5. Simplify distribution to use one scattering.dat file in SDSU High freq
6. Define data query interface for bbp results

Future Developments

1. 3D wave propagation capabilities
2. Concept for integrating 3D wave prop with BBP
3. Change name of main executable to remove 2g label
4. Define query interface for data products

Port bbp to mac

Steps installing bbp on mac

• install xcode to get gcc - must be registered mac developer
• install gfortran from mac hpc web site
• install gmt from gmt - put it in /opt/local
• include install of netcdf - put in /opt/local
• install macport
• install py27-matplotlib
• install py27-gtk
• reset python configure to use this new install
• edits to current bbp were required including
  • remove all of -D_LARGEFILE (and other similar) params
  • change g77 to gfortran
  • add declaration of DGAMM in URS code to stop return error problem
  • comment out headers like <features.h> and <sys/procs.h> or something like that

Problems:

• I had problems getting matplotlib to be recognized. I could import from the command line python interpreter, but scripts couldn't find it. Eventually it worked
• ifort compilers not install. Currently only testing with urs code

= Logs

zhs158-1:~ maechlin$ sudo port install py27-matplotlib Warning: xcodebuild exists but failed to execute Warning: Xcode appears to be installed but xcodebuild is unusable; some ports will likely fail to build. Warning: You may need to run `sudo xcode-select -switch /Applications/Xcode.app` ---> Computing dependencies for py27-matplotlib ---> Dependencies to be installed: freetype bzip2 zlib libpng py27-configobj python27 db46 gettext expat libiconv gperf ncurses libedit openssl python_select sqlite3 xz py27-dateutil py27-distribute py27-distribute py27-tz py27-tz py27-numpy fftw-3 py27-nose nosetests_select py27-pyobjc-cocoa py27-pyobjc libxml2 libxml2 py27-py2app py27-bdist_mpkg py27-macholib py27-modulegraph py27-altgraph py27-tkinter tk Xft2 fontconfig pkgconfig glib2 libffi perl5 perl5.12 gdbm perl5 perl5 xrender xorg-libX11 xorg-bigreqsproto xorg-inputproto xorg-kbproto xorg-libXau xorg-xproto xorg-libXdmcp xorg-libxcb xorg-libpthread-stubs xorg-xcb-proto xorg-xcmiscproto xorg-xextproto xorg-xf86bigfontproto xorg-xtrans xorg-renderproto tcl xorg-libXScrnSaver xorg-libXext autoconf help2man p5.12-locale-gettext m4 autoconf automake libtool xorg-util-macros xorg-scrnsaverproto

To make python 2.7 the default (i.e. the version you get when you run 'python'), please run:

sudo port select --set python python27

---> Installing py27-matplotlib @1.1.0_3+tkinter ---> Activating py27-matplotlib @1.1.0_3+tkinter

The default backend is the interactive Mac OS X backend. Different backends can be specified using the ~/.matplotlib/matplotlibrc file. More details regarding backends can be found in the matplotlib FAQ:

http://matplotlib.sourceforge.net/faq/installing_faq.html#what-is-a-backend

---> Cleaning py27-matplotlib zhs158-1:~ maechlin$

gmt install

All users must include /opt/local/gmt/man in their MANPATH All users should add /opt/local/gmt/share/doc/gmt/html/gmt_services.html to their browser bookmarks ./anim_01.sh: First frame plotted to anim_01.ps ./anim_02.sh: First frame plotted to anim_02.ps ./anim_03.sh: First frame plotted to anim_03.ps ./anim_04.sh: First frame plotted to anim_04.ps GMT installation complete. Remember to set these:

---

For csh or tcsh users: setenv NETCDFHOME /opt/local/netcdf set path=(/opt/local/gmt/bin $path)

For sh or bash users: export NETCDFHOME=/opt/local/netcdf export PATH=/opt/local/gmt/bin:$PATH

Note: if you installed netCDF as a shared library you may have to add the path to this library to LD_LIBRARY_PATH or place the library in a standard system path [see information on shared library for your OS].

For all users: Add /opt/local/gmt/man to MANPATH Add /opt/local/gmt/share/doc/gmt/html/gmt_services.html as browser bookmark

---

progress:Downloads maechlin$

Running URS Validations

/home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband -01/patrices/11.2.2/bbp_2g/xml/4081900.xml -s 4081900 -l /home/broadband-01/patr ices/11.2.2/bbp_2g/logs/nr_urs_validation.log /home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband -01/patrices/11.2.2/bbp_2g/xml/4081989.xml -s 4081989 -l /home/broadband-01/patr ices/11.2.2/bbp_2g/logs/lp_urs_validation.log /home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband -01/patrices/11.2.2/bbp_2g/xml/4082008.xml -s 4082008 -l /home/broadband-01/patr ices/11.2.2/bbp_2g/logs/la_urs_validation.log

Run Validation Events

Move three files into start directory. Point to options file. then invoke start run_bbp_2b.py -o ../../validations/la_options_urs.txt

Broadband development Python styling

Here are some pages with useful Python coding styling recommendations:

• Style Guide for Python Code
• Code Like a Pythonista
• Google Python Style Guide

Please try to follow the guidelines above unless there is a good reason to do otherwise.