Difference between revisions of "BBP Development"

From SCECpedia
Jump to navigationJump to search
 
(44 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page lists current Broadband Platform development activities:
+
This page lists current Broadband Platform development resources and activities:
  
== Development Priorities ==
+
== Development Infrastructure ==
#Organize patche on wiki as "optional" collection
+
 
#Define wiki entry on to to run one acceptance test to run one at a time
+
Broadband development uses several computation and storage resources at SCEC. Here is a summary of where each component is located:
#Define Graves and Kottke study "manifests"
+
 
#Document on wiki how to change random seed
+
* Broadband svn: source.usc.edu:/export/data/svn/broadband
#Patch to include Broadband batch tools tested, distributed, documented
+
* Broadband wiki: scec.usc.edu:/var/www/html/scecwiki
#Create two distributions of Broadband hanging wall inputs and results (Graves and Kottke) with documentation
+
* Broadband trac: northridge.usc.edu:/export/data/trac/broadband
#Modification to use gnu build tools
+
* Broadband release distribution site: hypocenter.usc.edu/research/broadband
#Minimal software installation distribution for clusters
+
* Broadband Fling Study distribution site: /home/broadband-00/fling
#Patch to remove dependencies on gmt,imagemagic,...
+
* Broadband mailing list: broadband-all@intensity.usc.edu
 +
* Broadband user accounts: scec-00 and scec-01
 +
 
 +
Everything is backed up nightly on scecvault. The exceptions are:
 +
 
 +
* Release distribution site (and everything else on hypocenter)
 +
* Mailing lists
 +
* User data on /home/broadband-01 and /home/broadband-02
 +
 
 +
== Development Work ==
 +
With Broadband version v11.2.2., all unit tests and acceptance tests pass on broadband.usc.edu. However, we get test failures on intensity.usc.edu, and we are working to determine the reasons. Here are the current Unit test results on intensity.
 +
 
 +
== Unit Tests on Intensity ==
 +
<pre>
 +
bash-3.2$ ./UnitTests.py > out.txt &                                           
 +
[1] 16243                                                                     
 +
bash-3.2$ test_gensrf (test_genslip.Test_Genslip) ... ok                       
 +
test_bbp_wid (test_jbsim.Test_Jbsim) ... ok                                   
 +
test_bbp_wid (test_hfsims.Test_hfsims) ... ok                                 
 +
test_bbp_wid (test_wcc_siteamp.Test_Wcc_siteamp) ... ok                       
 +
test_bbp_wid (test_match.Test_Match) ... ok                                   
 +
test_respect (test_respect.Test_Respect) ... ok                               
 +
test_gof (test_urs_gof.Test_URS_Gof) ... ok                                   
 +
test_bbp_wid (test_uc_makefaultg.Test_UC_Makefaultg) ... ok                   
 +
test_ucgen (test_ucrmg.Test_UCrmg) ... ok                                     
 +
test_syn1d (test_syn1d.Test_syn1D) ... FAIL                                   
 +
test_site_amp (test_uc_site.Test_UC_Site) ... FAIL                             
 +
test_respect (test_respect.Test_Respect) ... ok                               
 +
test_gof (test_urs_gof.Test_URS_Gof) ... ok                                   
 +
test_bbtoolbox (test_bbtoolbox.Test_BBToolbox) ... ok                         
 +
test_amp_fac (test_amp_fac.Test_Amp_Fac) ... ok                               
 +
test_respect (test_respect.Test_Respect) ... ok                               
 +
test_gof (test_urs_gof.Test_URS_Gof) ... ok
 +
 
 +
======================================================================
 +
FAIL: test_syn1d (test_syn1d.Test_syn1D)
 +
----------------------------------------------------------------------
 +
Traceback (most recent call last):
 +
  File "/home/scec-00/maechlin/bbp/i_host/bbp_2g/tests/test_syn1d.py", line 71,in test_syn1d
 +
    self.failIf(cmp_bbp.cmp_bbp(a_ref_file,a_newfile)!=0,errmsg)
 +
AssertionError: Output file does not match reference file: /home/scec-00/maechlin/bbp/i_host/bbp_2g/tmpdata/5222974/s01.3comp
 +
 
 +
======================================================================
 +
FAIL: test_site_amp (test_uc_site.Test_UC_Site)
 +
----------------------------------------------------------------------
 +
Traceback (most recent call last):
 +
  File "/home/scec-00/maechlin/bbp/i_host/bbp_2g/tests/test_uc_site.py", line 92, in test_site_amp
 +
    self.failIf(cmp_bbp.cmp_bbp(a_ref_file_no_header, test_file_no_header, tolerance=0.035)!=0,errmsg)
 +
AssertionError: Output file /home/scec-00/maechlin/bbp/i_host/bbp_2g/tmpdata/5223105/s04.3comp does not match reference file /home/scec-00/maechlin/bbp/i_host/bbp_2g/ref_data/ucsb/s04.site.3comp
 +
 
 +
----------------------------------------------------------------------
 +
Ran 17 tests in 2125.133s
 +
 
 +
FAILED (failures=2)
 +
</pre>
 +
 
 +
== List of Unit Test Output Directories ==
 +
 
 +
== Top Development Priorities ==
 +
#Create wiki entry describing setting random SEED for three High Freq Modules
 +
#Create "optional" patch distributions separate from official bbp 11.2.2 release
 +
##Define acceptance test patch
 +
##Define option parsing patch
 +
#Define wiki entry on how to run one acceptance test to run one at a time
 +
#Define Graves study (including list of output files) and create wiki entry
 +
#Define Kottke study (including list of output files) and create wiki entry
 +
#Setup Graves study using batch tools
 +
#Setup Kottke study using batch tools
 +
#Create a wiki for batch tools for broadband platform
 +
 
 +
== Development Activities ==
 +
#Modify broadband distribution to use gnu build tools
 +
#Create minimized software installation distribution for clusters
 +
##Remove dependencies on gmt,imagemagic,...
 +
#Develop prototype distributed broadband queue manager
 
#Distribution tracking
 
#Distribution tracking
 
##Check for updates capability
 
##Check for updates capability
Line 16: Line 90:
 
##Optional registration announcement
 
##Optional registration announcement
 
#Simplify distribution to use one scattering.dat file in SDSU High freq
 
#Simplify distribution to use one scattering.dat file in SDSU High freq
#User configurable random seed in three high freq codes
+
#Define data query interface for bbp results
  
 
== Future Developments ==
 
== Future Developments ==
Line 23: Line 97:
 
#Change name of main executable to remove 2g label
 
#Change name of main executable to remove 2g label
 
#Define query interface for data products
 
#Define query interface for data products
 +
 +
== Port bbp to mac ==
 +
Steps installing bbp on mac
 +
 +
*install xcode to get gcc - must be registered mac developer
 +
* install gfortran from mac hpc web site
 +
* install gmt from gmt - put it in /opt/local
 +
*include install of netcdf - put in /opt/local
 +
* install macport
 +
* install py27-matplotlib
 +
* install py27-gtk
 +
* reset python configure to use this new install
 +
* edits to current bbp were required including
 +
** remove all of -D_LARGEFILE (and other similar) params
 +
** change g77 to gfortran
 +
** add declaration of DGAMM in URS code to stop return error problem
 +
** comment out headers like <features.h> and <sys/procs.h> or something like that
 +
**
 +
 +
Problems:
 +
*I had problems getting matplotlib to be recognized. I could import from the command line python interpreter, but scripts couldn't find it. Eventually it worked
 +
*ifort compilers not install. Currently only testing with urs code
 +
*
 +
=== Logs ==
 +
zhs158-1:~ maechlin$ sudo port install py27-matplotlib
 +
Warning: xcodebuild exists but failed to execute
 +
Warning: Xcode appears to be installed but xcodebuild is unusable; some ports will likely fail to build.
 +
Warning: You may need to run `sudo xcode-select -switch /Applications/Xcode.app`
 +
--->  Computing dependencies for py27-matplotlib
 +
--->  Dependencies to be installed: freetype bzip2 zlib libpng py27-configobj python27 db46 gettext expat libiconv gperf ncurses libedit openssl python_select sqlite3 xz py27-dateutil py27-distribute py27-distribute py27-tz py27-tz py27-numpy fftw-3 py27-nose nosetests_select py27-pyobjc-cocoa py27-pyobjc libxml2 libxml2 py27-py2app py27-bdist_mpkg py27-macholib py27-modulegraph py27-altgraph py27-tkinter tk Xft2 fontconfig pkgconfig glib2 libffi perl5 perl5.12 gdbm perl5 perl5 xrender xorg-libX11 xorg-bigreqsproto xorg-inputproto xorg-kbproto xorg-libXau xorg-xproto xorg-libXdmcp xorg-libxcb xorg-libpthread-stubs xorg-xcb-proto xorg-xcmiscproto xorg-xextproto xorg-xf86bigfontproto xorg-xtrans xorg-renderproto tcl xorg-libXScrnSaver xorg-libXext autoconf help2man p5.12-locale-gettext m4 autoconf automake libtool xorg-util-macros xorg-scrnsaverproto
 +
 +
 +
To make python 2.7 the default (i.e. the version you get when you run 'python'),
 +
please run:
 +
 +
sudo port select --set python python27
 +
 +
--->  Installing py27-matplotlib @1.1.0_3+tkinter
 +
--->  Activating py27-matplotlib @1.1.0_3+tkinter
 +
 +
The default backend is the interactive Mac OS X backend. Different backends can
 +
be specified using the ~/.matplotlib/matplotlibrc file. More details regarding
 +
backends can be found in the matplotlib FAQ:
 +
 +
http://matplotlib.sourceforge.net/faq/installing_faq.html#what-is-a-backend
 +
 +
--->  Cleaning py27-matplotlib
 +
zhs158-1:~ maechlin$
 +
 +
 +
== gmt install ==
 +
All users must include /opt/local/gmt/man in their MANPATH
 +
All users should add /opt/local/gmt/share/doc/gmt/html/gmt_services.html to their browser bookmarks
 +
./anim_01.sh: First frame plotted to anim_01.ps
 +
./anim_02.sh: First frame plotted to anim_02.ps
 +
./anim_03.sh: First frame plotted to anim_03.ps
 +
./anim_04.sh: First frame plotted to anim_04.ps
 +
GMT installation complete. Remember to set these:
 +
 +
-----------------------------------------------------------------------
 +
For csh or tcsh users:
 +
setenv NETCDFHOME /opt/local/netcdf
 +
set path=(/opt/local/gmt/bin $path)
 +
 +
For sh or bash users:
 +
export NETCDFHOME=/opt/local/netcdf
 +
export PATH=/opt/local/gmt/bin:$PATH
 +
 +
Note: if you installed netCDF as a shared library you may have to add
 +
the path to this library to LD_LIBRARY_PATH or place the library in a
 +
standard system path [see information on shared library for your OS].
 +
 +
For all users:
 +
Add /opt/local/gmt/man to MANPATH
 +
Add /opt/local/gmt/share/doc/gmt/html/gmt_services.html as browser bookmark
 +
-----------------------------------------------------------------------
 +
progress:Downloads maechlin$
 +
 +
== Running URS Validations ==
 +
 +
/home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband
 +
-01/patrices/11.2.2/bbp_2g/xml/4081900.xml -s 4081900 -l /home/broadband-01/patr
 +
ices/11.2.2/bbp_2g/logs/nr_urs_validation.log
 +
/home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband
 +
-01/patrices/11.2.2/bbp_2g/xml/4081989.xml -s 4081989 -l /home/broadband-01/patr
 +
ices/11.2.2/bbp_2g/logs/lp_urs_validation.log
 +
/home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband
 +
-01/patrices/11.2.2/bbp_2g/xml/4082008.xml -s 4082008 -l /home/broadband-01/patr
 +
ices/11.2.2/bbp_2g/logs/la_urs_validation.log
 +
 +
== Run Validation Events ==
 +
Move three files into start directory. Point to options file. then invoke start
 +
run_bbp_2b.py -o ../../validations/la_options_urs.txt
 +
 +
== Broadband development Python styling ==
 +
 +
Here are some pages with useful Python coding styling recommendations:
 +
 +
* [http://www.python.org/dev/peps/pep-0008 Style Guide for Python Code]
 +
* [http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html Code Like a Pythonista]
 +
* [http://google-styleguide.googlecode.com/svn/trunk/pyguide.html Google Python Style Guide]
 +
 +
Please try to follow the guidelines above unless there is a good reason to do otherwise.

Latest revision as of 16:17, 21 June 2012

This page lists current Broadband Platform development resources and activities:

Development Infrastructure

Broadband development uses several computation and storage resources at SCEC. Here is a summary of where each component is located:

  • Broadband svn: source.usc.edu:/export/data/svn/broadband
  • Broadband wiki: scec.usc.edu:/var/www/html/scecwiki
  • Broadband trac: northridge.usc.edu:/export/data/trac/broadband
  • Broadband release distribution site: hypocenter.usc.edu/research/broadband
  • Broadband Fling Study distribution site: /home/broadband-00/fling
  • Broadband mailing list: broadband-all@intensity.usc.edu
  • Broadband user accounts: scec-00 and scec-01

Everything is backed up nightly on scecvault. The exceptions are:

  • Release distribution site (and everything else on hypocenter)
  • Mailing lists
  • User data on /home/broadband-01 and /home/broadband-02

Development Work

With Broadband version v11.2.2., all unit tests and acceptance tests pass on broadband.usc.edu. However, we get test failures on intensity.usc.edu, and we are working to determine the reasons. Here are the current Unit test results on intensity.

Unit Tests on Intensity

bash-3.2$ ./UnitTests.py > out.txt &                                            
[1] 16243                                                                       
bash-3.2$ test_gensrf (test_genslip.Test_Genslip) ... ok                        
test_bbp_wid (test_jbsim.Test_Jbsim) ... ok                                     
test_bbp_wid (test_hfsims.Test_hfsims) ... ok                                   
test_bbp_wid (test_wcc_siteamp.Test_Wcc_siteamp) ... ok                         
test_bbp_wid (test_match.Test_Match) ... ok                                     
test_respect (test_respect.Test_Respect) ... ok                                 
test_gof (test_urs_gof.Test_URS_Gof) ... ok                                     
test_bbp_wid (test_uc_makefaultg.Test_UC_Makefaultg) ... ok                     
test_ucgen (test_ucrmg.Test_UCrmg) ... ok                                       
test_syn1d (test_syn1d.Test_syn1D) ... FAIL                                     
test_site_amp (test_uc_site.Test_UC_Site) ... FAIL                              
test_respect (test_respect.Test_Respect) ... ok                                 
test_gof (test_urs_gof.Test_URS_Gof) ... ok                                     
test_bbtoolbox (test_bbtoolbox.Test_BBToolbox) ... ok                           
test_amp_fac (test_amp_fac.Test_Amp_Fac) ... ok                                 
test_respect (test_respect.Test_Respect) ... ok                                 
test_gof (test_urs_gof.Test_URS_Gof) ... ok

======================================================================
FAIL: test_syn1d (test_syn1d.Test_syn1D)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/scec-00/maechlin/bbp/i_host/bbp_2g/tests/test_syn1d.py", line 71,in test_syn1d
    self.failIf(cmp_bbp.cmp_bbp(a_ref_file,a_newfile)!=0,errmsg)
AssertionError: Output file does not match reference file: /home/scec-00/maechlin/bbp/i_host/bbp_2g/tmpdata/5222974/s01.3comp

======================================================================
FAIL: test_site_amp (test_uc_site.Test_UC_Site)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/scec-00/maechlin/bbp/i_host/bbp_2g/tests/test_uc_site.py", line 92, in test_site_amp
    self.failIf(cmp_bbp.cmp_bbp(a_ref_file_no_header, test_file_no_header, tolerance=0.035)!=0,errmsg)
AssertionError: Output file /home/scec-00/maechlin/bbp/i_host/bbp_2g/tmpdata/5223105/s04.3comp does not match reference file /home/scec-00/maechlin/bbp/i_host/bbp_2g/ref_data/ucsb/s04.site.3comp

----------------------------------------------------------------------
Ran 17 tests in 2125.133s

FAILED (failures=2)

List of Unit Test Output Directories

Top Development Priorities

  1. Create wiki entry describing setting random SEED for three High Freq Modules
  2. Create "optional" patch distributions separate from official bbp 11.2.2 release
    1. Define acceptance test patch
    2. Define option parsing patch
  3. Define wiki entry on how to run one acceptance test to run one at a time
  4. Define Graves study (including list of output files) and create wiki entry
  5. Define Kottke study (including list of output files) and create wiki entry
  6. Setup Graves study using batch tools
  7. Setup Kottke study using batch tools
  8. Create a wiki for batch tools for broadband platform

Development Activities

  1. Modify broadband distribution to use gnu build tools
  2. Create minimized software installation distribution for clusters
    1. Remove dependencies on gmt,imagemagic,...
  3. Develop prototype distributed broadband queue manager
  4. Distribution tracking
    1. Check for updates capability
    2. Download counter for software
    3. Optional registration announcement
  5. Simplify distribution to use one scattering.dat file in SDSU High freq
  6. Define data query interface for bbp results

Future Developments

  1. 3D wave propagation capabilities
  2. Concept for integrating 3D wave prop with BBP
  3. Change name of main executable to remove 2g label
  4. Define query interface for data products

Port bbp to mac

Steps installing bbp on mac

  • install xcode to get gcc - must be registered mac developer
  • install gfortran from mac hpc web site
  • install gmt from gmt - put it in /opt/local
  • include install of netcdf - put in /opt/local
  • install macport
  • install py27-matplotlib
  • install py27-gtk
  • reset python configure to use this new install
  • edits to current bbp were required including
    • remove all of -D_LARGEFILE (and other similar) params
    • change g77 to gfortran
    • add declaration of DGAMM in URS code to stop return error problem
    • comment out headers like <features.h> and <sys/procs.h> or something like that

Problems:

  • I had problems getting matplotlib to be recognized. I could import from the command line python interpreter, but scripts couldn't find it. Eventually it worked
  • ifort compilers not install. Currently only testing with urs code

= Logs

zhs158-1:~ maechlin$ sudo port install py27-matplotlib Warning: xcodebuild exists but failed to execute Warning: Xcode appears to be installed but xcodebuild is unusable; some ports will likely fail to build. Warning: You may need to run `sudo xcode-select -switch /Applications/Xcode.app` ---> Computing dependencies for py27-matplotlib ---> Dependencies to be installed: freetype bzip2 zlib libpng py27-configobj python27 db46 gettext expat libiconv gperf ncurses libedit openssl python_select sqlite3 xz py27-dateutil py27-distribute py27-distribute py27-tz py27-tz py27-numpy fftw-3 py27-nose nosetests_select py27-pyobjc-cocoa py27-pyobjc libxml2 libxml2 py27-py2app py27-bdist_mpkg py27-macholib py27-modulegraph py27-altgraph py27-tkinter tk Xft2 fontconfig pkgconfig glib2 libffi perl5 perl5.12 gdbm perl5 perl5 xrender xorg-libX11 xorg-bigreqsproto xorg-inputproto xorg-kbproto xorg-libXau xorg-xproto xorg-libXdmcp xorg-libxcb xorg-libpthread-stubs xorg-xcb-proto xorg-xcmiscproto xorg-xextproto xorg-xf86bigfontproto xorg-xtrans xorg-renderproto tcl xorg-libXScrnSaver xorg-libXext autoconf help2man p5.12-locale-gettext m4 autoconf automake libtool xorg-util-macros xorg-scrnsaverproto


To make python 2.7 the default (i.e. the version you get when you run 'python'), please run:

sudo port select --set python python27

---> Installing py27-matplotlib @1.1.0_3+tkinter ---> Activating py27-matplotlib @1.1.0_3+tkinter

The default backend is the interactive Mac OS X backend. Different backends can be specified using the ~/.matplotlib/matplotlibrc file. More details regarding backends can be found in the matplotlib FAQ:

http://matplotlib.sourceforge.net/faq/installing_faq.html#what-is-a-backend

---> Cleaning py27-matplotlib zhs158-1:~ maechlin$


gmt install

All users must include /opt/local/gmt/man in their MANPATH All users should add /opt/local/gmt/share/doc/gmt/html/gmt_services.html to their browser bookmarks ./anim_01.sh: First frame plotted to anim_01.ps ./anim_02.sh: First frame plotted to anim_02.ps ./anim_03.sh: First frame plotted to anim_03.ps ./anim_04.sh: First frame plotted to anim_04.ps GMT installation complete. Remember to set these:


For csh or tcsh users: setenv NETCDFHOME /opt/local/netcdf set path=(/opt/local/gmt/bin $path)

For sh or bash users: export NETCDFHOME=/opt/local/netcdf export PATH=/opt/local/gmt/bin:$PATH

Note: if you installed netCDF as a shared library you may have to add the path to this library to LD_LIBRARY_PATH or place the library in a standard system path [see information on shared library for your OS].

For all users: Add /opt/local/gmt/man to MANPATH Add /opt/local/gmt/share/doc/gmt/html/gmt_services.html as browser bookmark


progress:Downloads maechlin$

Running URS Validations

/home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband -01/patrices/11.2.2/bbp_2g/xml/4081900.xml -s 4081900 -l /home/broadband-01/patr ices/11.2.2/bbp_2g/logs/nr_urs_validation.log /home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband -01/patrices/11.2.2/bbp_2g/xml/4081989.xml -s 4081989 -l /home/broadband-01/patr ices/11.2.2/bbp_2g/logs/lp_urs_validation.log /home/broadband-01/patrices/11.2.2/bbp_2g/comps/run_bbp_2G.py -x /home/broadband -01/patrices/11.2.2/bbp_2g/xml/4082008.xml -s 4082008 -l /home/broadband-01/patr ices/11.2.2/bbp_2g/logs/la_urs_validation.log

Run Validation Events

Move three files into start directory. Point to options file. then invoke start run_bbp_2b.py -o ../../validations/la_options_urs.txt

Broadband development Python styling

Here are some pages with useful Python coding styling recommendations:

Please try to follow the guidelines above unless there is a good reason to do otherwise.