CVM S4 Testing - Revision history

Maechlin: /* Test 1 */

2022-04-25T20:13:32Z

Test 1

Maechlin: /* Search plans */

2022-04-25T19:58:22Z

Search plans

Maechlin at 19:48, 25 April 2022

2022-04-25T19:48:42Z

Maechlin at 19:48, 25 April 2022

2022-04-25T19:48:03Z

Maechlin: /* Test Results Summary */

2022-04-25T19:46:34Z

Test Results Summary

Maechlin: /* Test Results Summary */

2022-04-25T19:46:07Z

Test Results Summary

Maechlin: /* Test Results Summary */

2022-04-25T19:45:03Z

Test Results Summary

Maechlin at 19:44, 25 April 2022

2022-04-25T19:44:31Z

Maechlin at 19:44, 25 April 2022

2022-04-25T19:44:05Z

Maechlin: /* Test Results Summary */

2022-04-25T16:59:20Z

Test Results Summary

@@ Line 13: / Line 13: @@
 == Test 1 ==
-Create a grid file using ucvm makegrid.sh with these settings. This will search the LA basin region at 10km resolution.
+Create a grid file using ucvm makegrid.sh with these settings. This will search the LA basin region at 10m resolution.
    HOR_SPACING=0.001

@@ Line 258: / Line 258: @@
 == Search plans ==
 We will use ucvm makemesh.sh script to create query points for each of these regions. We'll define a search layer from 80m to 120m at 10m resolution to determine whether the alternative values are at a single depth, or in a range of depths. We will add the original observed points to end of the file to ensure we reproduce the initial observations.
 * Triple query version
-The third ucvm query will contain three duplicate points for each query. We will determine whether the second and third calls will create duplicates also.
+The ucvm query will contain three duplicate points for each query. We will determine whether the second and third calls will create duplicates also.
 * Query at different depths for same lat/lon

@@ Line 11: / Line 11: @@
 # The issue occurs at depths between 0-500m. It occurs most frequently at 300m. Drops off signifantly below 300m. Not observed for any calls below 500m.
 # More tests to further describe the problem are possible (e.g. test Salton Sea GTL region). But no obvious solution except contacting Kohler/Magistrale for code review.
 == Test 1 ==
@@ Line 231: / Line 220: @@
 Total found: 13311
 </pre>
 == Initial Observed Differences ==

@@ Line 10: / Line 10: @@
 # The issue occurs for a fraction of the total lat lons in the GTL region (<10%).
 # The issue occurs at depths between 0-500m. It occurs most frequently at 300m. Drops off signifantly below 300m. Not observed for any calls below 500m.
+# More tests to further describe the problem are possible (e.g. test Salton Sea GTL region). But no obvious solution except contacting Kohler/Magistrale for code review.
 == Test Results Details ==
 # Test 1 results indicate that ucvm_query calls to that same lat lon return the same value if there are ucvm_query call to a different lat long between them. In order words, the problem occurs only if consecutive calls to the same lat lon are made.

@@ Line 6: / Line 6: @@
 == Test Results Summary ==
 # Confirmed that repeated ucvm calls to CVM-S4 for the same point at the same depth can return different Vs values. We'll call this a stateful behavior issue.
-# Issue occurs when same point is queried, and no intervening points are queried. In other words, calls to another lat lon reset ucvm, and after this reset, a second query for a lat lon returns returns the same value as the first call.
+# Issue occurs when same point is queried at the same depth, and no intervening points are queried. In other words, calls to another lat lon reset ucvm, and after this reset, a second query for a lat lon returns returns the same value as the first call.
 # The issue occurs only in CVM-S4 GTL region. no stateful behavior was seen in outside the CVM-S4 GTL region.
 # The issue occurs for a fraction of the total lat lons in the GTL region (<10%).

@@ Line 5: / Line 5: @@
 To investigate this, we will run some tests to determine if all CVM-S4 regions are stateful, or only the GTL regions. We will also look at how large the differences
 == Test Results Summary ==
-. Confirmed that repeated ucvm calls to CVM-S4 for the same point at the same depth can return different Vs values. We'll call this stateful behavior.
+# Confirmed that repeated ucvm calls to CVM-S4 for the same point at the same depth can return different Vs values. We'll call this stateful behavior.
-. Issue occurs when same point is queried, with no intervening points are queried. In other words, calls to another lat lon reset ucvm, and the second query returns for a point returns the same value as the first call.
+# Issue occurs when same point is queried, with no intervening points are queried. In other words, calls to another lat lon reset ucvm, and the second query returns for a point returns the same value as the first call.
-. The issue occurs only in CVM-S4 GTL region. no stateful behavior was seen in outside the CVM-S4 GTL region.
+# The issue occurs only in CVM-S4 GTL region. no stateful behavior was seen in outside the CVM-S4 GTL region.
-. The issue occurs for a fraction of the total lat lons in the GTL region (<10%).
+# The issue occurs for a fraction of the total lat lons in the GTL region (<10%).
-. The issue occurs at depths between 0-500m. It occurs most frequently at 300m. Drops off signifantly below 300m. Not observed for any calls below 500m.
+# The issue occurs at depths between 0-500m. It occurs most frequently at 300m. Drops off signifantly below 300m. Not observed for any calls below 500m.

@@ Line 7: / Line 7: @@
 == Test Results Summary ==
 # Test 1 results indicate that ucvm_query calls to that same lat lon return the same value if there are ucvm_query call to a different lat long between them. In order words, the problem occurs only if consecutive calls to the same lat lon are made.
 # Test 2 results indicate that only a fraction of lat lons show this problem, about 3% of lat lons in the LA Basin area. Plotting of the lat lons in the region show all occurances are in the CVMS-4 GTL region
 # Test 2 results indicate the the issue occurs fairly evenly for repeated queries at 90m, 100m, and 110 m depths.
 # KML file showing distribution of sites where query issues occur: [[file:UCVM Coverage Region v22 5.kml | UCVM KML File]]
 # Test 3 results indicate that the issue does not occur in the CVMS-4 background region, outside the the CVM-S4 GTL zone.
 # Test 4 results indicate that the issue occures down to 500m, but less frequently as queries go deeper. 500m seems the max depth which shows the issue.
 # KML file showing distribution of sites where query issues occur at 300m: [[file:UCVM_Coverage_Region_test4.kml | 300m stateful queries KML locations]]
 Further testing
 # Check if the problem is observed with call to the same lat lon but different depths.