CyberShake Study 22.12 - Revision history

Scottcal: /* Lessons Learned */

2025-08-13T18:41:37Z

Lessons Learned

Scottcal: /* Performance Metrics */

2025-06-06T05:49:03Z

Performance Metrics

Scottcal: /* Output Data Products */

2025-04-10T17:50:51Z

Output Data Products

Scottcal: /* Post-study directionality investigation */

2025-03-18T21:20:37Z

Post-study directionality investigation

Scottcal at 21:19, 18 March 2025

2025-03-18T21:19:01Z

Scottcal: /* Sites */

2025-03-18T05:30:32Z

Sites

Scottcal: /* Scatter Plots */

2024-12-18T22:34:08Z

Scatter Plots

Scottcal: /* Application Level Metrics */

2024-08-20T22:05:14Z

Application Level Metrics

Scottcal: /* Application Level Metrics */

2024-08-20T22:04:08Z

Application Level Metrics

Scottcal: /* Performance Metrics */

2024-08-20T22:03:07Z

Performance Metrics

← Older revision		Revision as of 18:41, 13 August 2025
Line 802:		Line 802:

	* Come up with a solution to the out/err file issue with the BB runs. With file-per-worker, PMC gets a seg fault when the final output is combined. With file-per-task, we end up with 160k files and the filesystem becomes angry.		* Come up with a solution to the out/err file issue with the BB runs. With file-per-worker, PMC gets a seg fault when the final output is combined. With file-per-task, we end up with 160k files and the filesystem becomes angry.
		+
		+	=== Subfault Perturbation ===
		+
		+	After running Study 24.8, we discovered that the SRF generation for both Study 22.12 and 24.8 had a misunderstanding leading to small but unintended consequences.
		+
		+	Both studies were run with perturb_subfault_location=1 in the rupture generator. This setting results in the location (lat/lon/depth) of the subfaults being perturbed slightly to capture fault roughness. Therefore, the subfault, or fault surface point, locations in the produced SRF are slightly different from the locations as listed in the rupture geometry file.
		+
		+	To save I/O, CyberShake uses the locations in the rupture geometry file to determine where to save the SGTs (in the PreSGT step), and also to determine where to read the SGTs from. Therefore, even if the subfault perturbations would shift the SGTs which are used, these shifts are ignored and don't have any impact in which SGTs are selected. Thus, running with perturb_subfault_location=1 results in the same low-frequency seismogram as with perturb_subfault_location=0, since the same SGTs would be used in both cases.
		+
		+	In the high-frequency stochastic runs, results are slightly different with perturb_subfault_location=1, because the distance calculations are slightly different.
		+
		+	Another place where the differences show up - and where this issue was diagnosed - is in the hypocenter locations in the database. If hypocenters are derived using perturb_subfault_location=1, they will be mapped to the nearest perturbed location, so when the hypocenters are plotted up it will result in a "fuzzy" fault trace rather than a sharp one. Since the SGTs were along the unperturbed locations, it is more accurate to regenerate SRFs with perturb_subfault_location off and then calculate hypocenters from these SRFs.

	== Stress Test ==		== Stress Test ==

@@ Line 947: / Line 947: @@
 **DB subworkflow: 10 jobs (2 dir create)
 **Post subworkflow: 3 jobs (1 dir create)
 == Site Response Analysis ==

@@ Line 698: / Line 698: @@
 *Seismograms: 2-component seismograms, 8000 timesteps (400 sec) each.  Study 18.8 used 10k timesteps, but without the sSAF events 8k seems fine.
 *PSA: X and Y spectral acceleration at 44 periods (10, 9.5, 9, 8.5, 8, 7.5, 7, 6.5, 6, 5.5, 5, 4.8, 4.6, 4.4, 4.2, 4, 3.8, 3.6, 3.4, 3.2, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.66667, 1.42857, 1.25, 1.11111, 1, .66667, .5, .4, .33333, .285714, .25, .22222, .2, .16667, .142857, .125, .11111, .1 sec)
-*RotD: PGV, and RotD50, the RotD50 azimuth, and RotD100 at 25 periods (20, 15, 12, 10, 8.5, 7.5, 6.5, 6, 5.5, 5, 4.4, 4, 3.5, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.7, 1.5, 1.3, 1.2, 1.1, 1)
+*RotD: PGV, and RotD50, RotD100, and RotD100 azimuth at 25 periods (20, 15, 12, 10, 8.5, 7.5, 6.5, 6, 5.5, 5, 4.4, 4, 3.5, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.7, 1.5, 1.3, 1.2, 1.1, 1)
 *Durations: for X and Y components, energy integral, Arias intensity, cumulative absolute velocity (CAV), and for both velocity and acceleration, 5-75%, 5-95%, and 20-80%.
@@ Line 705: / Line 705: @@
 *Seismograms: 2-component seismograms, 40000 timesteps (400 sec) each.
 *PSA: X and Y spectral acceleration at 44 periods (10, 9.5, 9, 8.5, 8, 7.5, 7, 6.5, 6, 5.5, 5, 4.8, 4.6, 4.4, 4.2, 4, 3.8, 3.6, 3.4, 3.2, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.66667, 1.42857, 1.25, 1.11111, 1, .66667, .5, .4, .33333, .285714, .25, .22222, .2, .16667, .142857, .125, .11111, .1 sec)
-*RotD: PGA, PGV, and RotD50, the RotD50 azimuth, and RotD100 at 66 periods (20, 15, 12, 10, 8.5, 7.5, 6.5, 6, 5.5, 5, 4.4, 4, 3.5, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.7, 1.5, 1.3, 1.2, 1.1, 1, 0.85, 0.75, 0.65, 0.6, 0.55, 0.5, 0.45, 0.4, 0.35, 0.3, 0.28, 0.26, 0.24, 0.22, 0.2, 0.17, 0.15, 0.13, 0.12, 0.11, 0.1, 0.085, 0.075, 0.065, 0.06, 0.055, 0.05, 0.045, 0.04, 0.035, 0.032, 0.029, 0.025, 0.022, 0.02, 0.017, 0.015, 0.013, 0.012, 0.011, 0.01)
+*RotD: PGA, PGV, and RotD50, RotD100, and RotD100 azimuth at 66 periods (20, 15, 12, 10, 8.5, 7.5, 6.5, 6, 5.5, 5, 4.4, 4, 3.5, 3, 2.8, 2.6, 2.4, 2.2, 2, 1.7, 1.5, 1.3, 1.2, 1.1, 1, 0.85, 0.75, 0.65, 0.6, 0.55, 0.5, 0.45, 0.4, 0.35, 0.3, 0.28, 0.26, 0.24, 0.22, 0.2, 0.17, 0.15, 0.13, 0.12, 0.11, 0.1, 0.085, 0.075, 0.065, 0.06, 0.055, 0.05, 0.045, 0.04, 0.035, 0.032, 0.029, 0.025, 0.022, 0.02, 0.017, 0.015, 0.013, 0.012, 0.011, 0.01)
 *Durations: for X and Y components, energy integral, Arias intensity, cumulative absolute velocity (CAV), and for both velocity and acceleration, 5-75%, 5-95%, and 20-80%.

@@ Line 911: / Line 911: @@
 == Post-study directionality investigation ==
-Alan Poulos and Evan Hirakawa investigated the directionality of the ground motions from this study, comparing the azimuths of the largest and smallest ground motions and comparing their ratios.  As a result of this work, they found a handful of sites with essentially infinite ratios, because the ground motions were entirely polarized, with 0 ground motion along one azimuth.  Additionally, the maximum and minimum azimuths aligned with the box angles.
+Alan Poulos and Evan Hirakawa investigated the directionality of the ground motions from this study in early 2025, comparing the azimuths of the largest and smallest ground motions and comparing their ratios.  As a result of this work, they found a handful of sites with essentially infinite ratios, because the ground motions were entirely polarized, with 0 ground motion along one azimuth.  Additionally, the maximum and minimum azimuths aligned with the box angles.
 We investigated these sites more closely.  We confirmed that we see the same directionality in both the low-frequency and broadband results, and in the RotD100 azimuths we calculated.  Looking at the workflow logs for these sites, we found that all of these sites had problems in their CheckSGT jobs.  These are the jobs that run at the beginning of the post-processing workflow to confirm that the MD5sum for the SGT file matches the previously calculated MD5sum.  For these sites, the X component check never ran, and the Y component check ran multiple times and failed because the MD5sums didn't match.  This is probably due to a number of issues we had with the Summit filesystem during the study.  The SGTs and MD5sums were copied from a temporary working directory to an SGT storage directory, and then the post-processing workflows symlinked to the storage directory.  We had a number of problems with the copy from the temporary working directory to the storage directory.  Sometimes Globus would record that the copy had completed successfully, but the file in the storage directory was incomplete.  We thought we'd identified and fixed these issues for all the sites that experienced them but some slipped through.
 s187, P5, P8, P24, PLS, s632, and s541 all had this problem with a checksum mismatch.  The checks for s319 also failed due to I/O errors.
 == Performance Metrics ==

← Older revision		Revision as of 21:19, 18 March 2025
Line 908:		Line 908:

	On 3/12, we ran out of disk space on moment, with about 25% of the broadband runs remaining to be inserted. We started deleting Study 15.12 off moment. To give it a head start, we found some runs which haven't finished their BB processing, so we turned back on the cronjob which kills workflows at the LoadAmps stage and are resuming the BB processing for these jobs.		On 3/12, we ran out of disk space on moment, with about 25% of the broadband runs remaining to be inserted. We started deleting Study 15.12 off moment. To give it a head start, we found some runs which haven't finished their BB processing, so we turned back on the cronjob which kills workflows at the LoadAmps stage and are resuming the BB processing for these jobs.
		+
		+	== Post-study directionality investigation ==
		+
		+	Alan Poulos and Evan Hirakawa investigated the directionality of the ground motions from this study, comparing the azimuths of the largest and smallest ground motions and comparing their ratios. As a result of this work, they found a handful of sites with essentially infinite ratios, because the ground motions were entirely polarized, with 0 ground motion along one azimuth. Additionally, the maximum and minimum azimuths aligned with the box angles.
		+
		+	We investigated these sites more closely. We confirmed that we see the same directionality in both the low-frequency and broadband results, and in the RotD100 azimuths we calculated. Looking at the workflow logs for these sites, we found that all of these sites had problems in their CheckSGT jobs. These are the jobs that run at the beginning of the post-processing workflow to confirm that the MD5sum for the SGT file matches the previously calculated MD5sum. For these sites, the X component check never ran, and the Y component check ran multiple times and failed because the MD5sums didn't match. This is probably due to a number of issues we had with the Summit filesystem during the study. The SGTs and MD5sums were copied from a temporary working directory to an SGT storage directory, and then the post-processing workflows symlinked to the storage directory. We had a number of problems with the copy from the temporary working directory to the storage directory. Sometimes Globus would record that the copy had completed successfully, but the file in the storage directory was incomplete. We thought we'd identified and fixed these issues for all the sites that experienced them but some slipped through.
		+
		+	s187, P5, P8, P24, PLS, s632, and s541 all had this problem with a checksum mismatch. The checks for s319 also failed due to I/O errors.

	== Performance Metrics ==		== Performance Metrics ==

@@ Line 72: / Line 72: @@
 |}
-*[[Media:Study_22.10_sites.csv|CSV site list]]
+*[[:Media:Study_22.10_sites.csv|CSV site list]]
 *[[Media:Study_22.10_sites_names.kml|KML site list with names]]
 *[[Media:Study_22.10_sites_no_names.kml|KML site list without names]]

@@ Line 499: / Line 499: @@
 |}
-We also modified the risetime_coef to 2.3, from the previous default of 1.6.
+We intended to run with the risetime_coef modified from the previous default of 1.6 to 2.3, but due to a code bug we continued to use 1.6.
 == High-frequency codes ==

@@ Line 922: / Line 922: @@
 *We used 728832.4 node-hours for the main study + 43493.9 for the stress test = <b>772326.3</b> node-hours total.
-*We used 772326.3 node-hrs /
+*We used 772326.3 node-hrs / 2330.9 hrs = an average of <b>331.3</b> nodes throughout the study, or 7.1% of Summit.
 *We ran a total of 965 top-level workflows: 20 for the stress test sites + 3*315 for the other sites.
 *We ran 20*83 + 315*(26+27+31) = 28,120 jobs successfully (retries are tracked elsewhere).