UCVM User Guide - Revision history

Davidgil: Redirected page to UCVM 14.3.0 User Guide

2014-06-09T23:52:21Z

Redirected page to UCVM 14.3.0 User Guide

Davidgil: /* Requirements */

2013-06-21T20:26:29Z

Requirements

Davidgil at 17:50, 21 June 2013

2013-06-21T17:50:04Z

Davidgil: /* Performance Enhancements for ucvm2mesh on Lustre Filesystems */

2013-05-02T22:23:46Z

Performance Enhancements for ucvm2mesh on Lustre Filesystems

Davidgil: /* Parallel ucvm2mesh */

2013-05-02T22:23:14Z

Parallel ucvm2mesh

Davidgil: /* Proj. 4 Error: major axis or radius = 0 or not given */

2012-12-10T01:12:21Z

Proj. 4 Error: major axis or radius = 0 or not given

Davidgil: /* Trouble-shooting */

2012-12-10T01:11:10Z

Trouble-shooting

Davidgil: /* Advanced Install */

2012-12-10T01:02:37Z

Advanced Install

Davidgil: /* Advanced Install */

2012-12-07T23:37:38Z

Advanced Install

Davidgil: /* Advanced Install */

2012-12-07T23:34:56Z

Advanced Install

← Older revision		Revision as of 23:52, 9 June 2014
Line 1:		Line 1:
		+	#REDIRECT [[UCVM 14.3.0 User Guide]]
		+
	= Overview =		= Overview =

@@ Line 23: / Line 23: @@
 # GNU gcc/gfortran compilers (MPI wrappers such as mpicc are OK)
 # tar for opening the compressed files
-# [http://www.cs.cmu.edu/~euclid/ Euclid Etree library] (recommend [http://hypocenter.usc.edu/research/ucvm/12.2.0/euclid3-1.3.tar.gz euclid3-1.3.tar.gz])
+# [http://www.cs.cmu.edu/~euclid/ Euclid Etree library] (recommend [http://hypocenter.usc.edu/research/ucvm/euclid3-1.3.tar.gz euclid3-1.3.tar.gz])
 # [http://trac.osgeo.org/proj/ Proj.4 Projection Library] (recommend [http://download.osgeo.org/proj/proj-4.7.0.tar.gz proj-4.7.0.tar.gz])
@@ Line 31: / Line 31: @@
 # Standard community velocity models: SCEC CVM-H, SCEC CVM-S, SCEC CVM-SI, SCEC CVM-NCI, Magistrale WFCVM, USGS CenCalVM, Graves Cape Mendocino, Lin-Thurber Statewide, Tape SoCal
 # NetCDF (network Common Data Form) library [http://www.unidata.ucar.edu/downloads/netcdf/index.jsp NetCDF]
 = Installation =

@@ Line 24: / Line 24: @@
 # tar for opening the compressed files
 # [http://www.cs.cmu.edu/~euclid/ Euclid Etree library] (recommend [http://hypocenter.usc.edu/research/ucvm/12.2.0/euclid3-1.3.tar.gz euclid3-1.3.tar.gz])
-# [http://trac.osgeo.org/proj/ Proj.4 Projection Library]
+# [http://trac.osgeo.org/proj/ Proj.4 Projection Library] (recommend [http://download.osgeo.org/proj/proj-4.7.0.tar.gz proj-4.7.0.tar.gz])

@@ Line 1,136: / Line 1,136: @@
 ==== Performance Enhancements for ucvm2mesh on Lustre Filesystems ====
-When running ucvm2mesh on Lustre filesystems (and other systems with similar structures), you may experience some performance issues that result in less than optimal performance. By default, Lustre uses one OST or disk per file. As such, running ucvm2mesh-mpi with a significant number of cores results in all those processes attempting to seek and write to the same file on the same OST/disk. This can result in very poor performance, where it may take up to 10 minutes to write a few hundred MBs.
+When running ucvm2mesh on Lustre filesystems (and other systems with similar structures), you may experience some performance issues that result in less than optimal performance. By default, Lustre uses one OST or disk per file. As such, running ucvm2mesh-mpi with a significant number of cores results in all those processes attempting to seek and write to the same file on the same OST/disk. This can result in a disk bottleneck that gives very poor performance, where it may take up to 10 minutes to write a few hundred MBs.
 This can be fixed by increasing the number of OSTs/disks to which the output directory and files are written. This is called striping. Generally, you will want to set the stripes for the output directory to be either the recommended amount for large files or the maximum amount allowed. The command to set the number of stripes on Kraken is lfs setstripe [dir] --count [count]. Count can be a fixed number or it can be -1 for the maximum supported by the operating system. The Kraken documentation suggests that 80 is a reasonable amount for very large files. So, for example, suppose we were generating a large mesh with our stage out directory being /lustre/scratch/username/meshes/outstripe. We would first make that directory and then run:

← Older revision		Revision as of 22:23, 2 May 2013
Line 1,134:		Line 1,134:
	</pre>		</pre>

		+	==== Performance Enhancements for ucvm2mesh on Lustre Filesystems ====
		+
		+	When running ucvm2mesh on Lustre filesystems (and other systems with similar structures), you may experience some performance issues that result in less than optimal performance. By default, Lustre uses one OST or disk per file. As such, running ucvm2mesh-mpi with a significant number of cores results in all those processes attempting to seek and write to the same file on the same OST/disk. This can result in very poor performance, where it may take up to 10 minutes to write a few hundred MBs.
		+
		+	This can be fixed by increasing the number of OSTs/disks to which the output directory and files are written. This is called striping. Generally, you will want to set the stripes for the output directory to be either the recommended amount for large files or the maximum amount allowed. The command to set the number of stripes on Kraken is lfs setstripe [dir] --count [count]. Count can be a fixed number or it can be -1 for the maximum supported by the operating system. The Kraken documentation suggests that 80 is a reasonable amount for very large files. So, for example, suppose we were generating a large mesh with our stage out directory being /lustre/scratch/username/meshes/outstripe. We would first make that directory and then run:
		+
		+	lfs setstripe /lustre/scratch/username/meshes/outstripe --count 80
		+
		+	Then we would have our configuration file output to that directory:
		+
		+	meshfile=/lustre/scratch/username/meshes/outstripe/USC_cvms_awp.media
		+	gridfile=/lustre/scratch/username/meshes/outstripe/USC_cvms_awp.media.grid
		+
		+	When we qsub the job, it will automatically take advantage of striping and distribute the load across multiple targets.
		+
		+	It is also important to remember that more cores is not always better. Additional cores can result in increased write requests that may cause performance decreases. It is important to find the optimal balance between the number of processes writing to the file and the number of stripes. For exampe, for a large 600km x 400km x 40km meshes, a core count of 600 and stripe count of 80 on NICS Kraken will result in reasonably good performance.

	=== grd_query ===		=== grd_query ===

@@ Line 297: / Line 297: @@
 === Proj. 4 Error: major axis or radius = 0 or not given ===
-On systems with home filesystems that are not viewable to compute nodes (such as NICS Kraken), you may encounter errors with Proj.4 when trying to run any component of UCVM on compute nodes. This is due to the fact that Proj.4 actually relies on a file called proj_defs.dat which is located in the ${MAKE_INSTALL_LOCATION}/share/proj directory. So for example, suppose you had configured Proj.4 with ./configure --prefix=/not_viewable_to_compute_nodes/proj-4.7.0, Proj. 4 would then search for proj_defs.dat in /not_viewable_to_compute_nodes/proj-4.7.0/share/proj/proj_defs.dat. Naturally this will fail once you submit your job.
+On systems with home filesystems that are not viewable to compute nodes (such as NICS Kraken), you may encounter errors with Proj.4 when trying to run any component of UCVM on compute nodes. This is due to the fact that Proj.4 actually relies on a file called proj_defs.dat which is located in the ${MAKE_INSTALL_LOCATION}/share/proj directory. So for example, suppose you had configured Proj.4 with ./configure --prefix=/not_viewable_to_compute_nodes/proj-4.7.0, Proj. 4 would then search for proj_defs.dat in /not_viewable_to_compute_nodes/proj-4.7.0/share/proj/proj_defs.dat. This will cause UCVM to throw the error "Proj.4 Error: major axis or radius = 0 or not given" and your job will fail.
 The only way it seems to solve this issue is to actually make sure your --prefix directory is actually visible to the compute nodes and do a make install there. Documentation suggests that you can set the PROJ_LIB environment variable, however this seems to not work correctly without modifications to the Proj.4 source code.

← Older revision		Revision as of 01:11, 10 December 2012
Line 295:		Line 295:
	Alternatively, if you do not want to work with shared libraries, UCVM can be compiled statically with the "--enable-static" configuration option.		Alternatively, if you do not want to work with shared libraries, UCVM can be compiled statically with the "--enable-static" configuration option.

		+	=== Proj. 4 Error: major axis or radius = 0 or not given ===
		+
		+	On systems with home filesystems that are not viewable to compute nodes (such as NICS Kraken), you may encounter errors with Proj.4 when trying to run any component of UCVM on compute nodes. This is due to the fact that Proj.4 actually relies on a file called proj_defs.dat which is located in the ${MAKE_INSTALL_LOCATION}/share/proj directory. So for example, suppose you had configured Proj.4 with ./configure --prefix=/not_viewable_to_compute_nodes/proj-4.7.0, Proj. 4 would then search for proj_defs.dat in /not_viewable_to_compute_nodes/proj-4.7.0/share/proj/proj_defs.dat. Naturally this will fail once you submit your job.
		+
		+	The only way it seems to solve this issue is to actually make sure your --prefix directory is actually visible to the compute nodes and do a make install there. Documentation suggests that you can set the PROJ_LIB environment variable, however this seems to not work correctly without modifications to the Proj.4 source code.

	=== Building Against CVM-H 11.2.0 ===		=== Building Against CVM-H 11.2.0 ===

@@ Line 192: / Line 192: @@
 </pre>
-<p>All components of UCVM must be compiled using the same compiler. NICS Kraken, for example, uses the Cray compiler wrappers (cc not gcc). As such, on this system, you '''must''' use cc to build the Euclid and Proj.4 libraries. Also, if the system requires static linking, you must disable shared library generation in Proj.4 with the flag, --enable-shared=no after ./configure. Failing to do so will result in the compiler throwing an error and the compilation process not completing successfully.</p>
+<p>It is generally advisable to compile all components of UCVM using the same compilation system. NICS Kraken, for example, uses the Cray compiler wrappers (cc not gcc). As such, on this system, you must use CC to compile the Euclid library.</p>
 A few notes:

@@ Line 191: / Line 191: @@
 $ make; make check; make install
 </pre>
 A few notes:
@@ Line 266: / Line 268: @@
    --with-tape-model-path  location of the TAPE model files
 </pre>
 == Unit and Acceptance Tests ==