Difference between revisions of "CVM S4 Testing"
(→Test 3) |
|||
| Line 1: | Line 1: | ||
| + | == Test Results Summary == | ||
| + | # Test 1 results indicate that ucvm_query calls to that same lat lon return the same value if there are ucvm_query call to a different lat long between them. In order words, the problem occurs only if consecutive calls to the same lat lon are made. | ||
| + | |||
| + | # Test 2 results indicate that only a fraction of lat lons show this problem, about 3% of lat lons in the LA Basin area. Plotting of the lat lons in the region show all occurances are in the CVMS-4 GTL region | ||
| + | |||
| + | # Test 3 results indicate that the issue does not occur in the background region. | ||
| + | |||
| + | Further testing | ||
| + | # Check if the duplicate calls that create issues are at a particular depth, say 90, or if they occur at all three depths 90, 100, 110. | ||
| + | # Check if the problem is observed with call to the same lat lon but different depths. | ||
== Test 1 == | == Test 1 == | ||
Create a grid file using ucvm makegrid.sh with these settings. This will search the LA basin region at 10km resolution. | Create a grid file using ucvm makegrid.sh with these settings. This will search the LA basin region at 10km resolution. | ||
| Line 18: | Line 28: | ||
Resulting populated file wc: | Resulting populated file wc: | ||
6012006 102204102 1124245122 cvms4test.res | 6012006 102204102 1124245122 cvms4test.res | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
Results of this test indicate that the second call returns the same result as the first call. The two calls are separated by 3006003 calls to other locations. | Results of this test indicate that the second call returns the same result as the first call. The two calls are separated by 3006003 calls to other locations. | ||
| Line 39: | Line 33: | ||
== Test 2 == | == Test 2 == | ||
Create grid file with three duplicate lines this way: | Create grid file with three duplicate lines this way: | ||
| − | + | #makegrid.sh using min,max lat,lon above and depths 90 110 at 10m intervals | |
| − | + | #duplicate input lines using script on discovery duplic_pts.py | |
| − | + | #input grid with duplicat lines to ucvm_query | |
| − | + | #transfer populated results to laptop | |
| − | + | #add 3 lines for the known result into the poptulated results to confirm issue | |
| − | + | #run python script that reads populate results into dataframe an calculates differnce betweens cells 1 and 2, and 2 and 3. | |
(base) [maechlin@discovery2 test_ucvm]$ wc test2.grd | (base) [maechlin@discovery2 test_ucvm]$ wc test2.grd | ||
Revision as of 03:09, 24 April 2022
Contents
Test Results Summary
- Test 1 results indicate that ucvm_query calls to that same lat lon return the same value if there are ucvm_query call to a different lat long between them. In order words, the problem occurs only if consecutive calls to the same lat lon are made.
- Test 2 results indicate that only a fraction of lat lons show this problem, about 3% of lat lons in the LA Basin area. Plotting of the lat lons in the region show all occurances are in the CVMS-4 GTL region
- Test 3 results indicate that the issue does not occur in the background region.
Further testing
- Check if the duplicate calls that create issues are at a particular depth, say 90, or if they occur at all three depths 90, 100, 110.
- Check if the problem is observed with call to the same lat lon but different depths.
Test 1
Create a grid file using ucvm makegrid.sh with these settings. This will search the LA basin region at 10km resolution.
HOR_SPACING=0.001 VERT_SPACING=10 MIN_DEPTH=90 MAX_DEPTH=110 MIN_LON=-118.5 MAX_LON=-117.5 MIN_LAT=33.5 MAX_LAT=34.5
Output file wc, showing 3M lines
3006003 9018009 59118059 out.grd
This file is contentated with itself. This will result in two calls to each point, but not consecutively. This tests whether the state-ful ness is reset when horizontal change is made.
Resulting populated file wc:
6012006 102204102 1124245122 cvms4test.res
Results of this test indicate that the second call returns the same result as the first call. The two calls are separated by 3006003 calls to other locations.
Test 2
Create grid file with three duplicate lines this way:
- makegrid.sh using min,max lat,lon above and depths 90 110 at 10m intervals
- duplicate input lines using script on discovery duplic_pts.py
- input grid with duplicat lines to ucvm_query
- transfer populated results to laptop
- add 3 lines for the known result into the poptulated results to confirm issue
- run python script that reads populate results into dataframe an calculates differnce betweens cells 1 and 2, and 2 and 3.
(base) [maechlin@discovery2 test_ucvm]$ wc test2.grd
9018006 27054018 177354120 test2.grd
Add 3 rows with calls to the in the initial point where this issue was identified.
Summary results from Python scripts:
Found 3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 ) Found 0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )
Script can be modified to determine additional information including lat,lons and percent differences found, if needed.
Checking the results for queries 2 and 3:
cvms4_data = "/Users/maechlin/cvms4test2.res"
Difference between call 2 and call 3: -154.118000 for row: 1820002 Difference between call 2 and call 3: -94.489000 for row: 4820002 Difference between call 2 and call 3: -286.251000 for row: 7820002 Found 3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 ) Found 0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )
Test 3
Redo Test 2 but do the query in the background region, where there should not be any borehole data
cvms4_data = "/Users/maechlin/cvms4test3.res" Found 0.0 percent of 1 and 2 queries had differences ( 0 out of 3006002 ) Found 0.0 percent of 2 and 3 queries had differences: (0 out of 3006002 )
Add the initial observed points into file to confirm script finds it
Found 0.0 percent of 1 and 2 queries had differences ( 1 out of 3006003 ) Found 0.0 percent of 2 and 3 queries had differences: (0 out of 3006003 )
Initial Observations Of CVM-S4 Stateful Behavior
We have identified points where CVM-S4 queries are stateful, specifically in the GTL region (~100m depth). By this, we mean that if we query the same point twice, CVM-S4 (and UCVM) returns different Vp, Vs, and rho properties for each query.
To investigate this, we will run some tests to determine if all CVM-S4 regions are stateful, or only the GTL regions. We will also look at how large the differences
Initial Observed Differences
ucvm_query -m cvmsi -f $UCVM_HOME/conf/ucvm.conf < in > out with the input being: -118.481000 34.312000 100.0000 -118.481000 34.312000 100.0000 -118.481000 34.312000 100.0000 -118.481000 34.312000 100.0000 returns the output: -118.4810 34.3120 100.000 386.051 397.973 cvmsi 2185.167 989.497 . . . -118.4810 34.3120 100.000 386.051 397.973 cvmsi 2435.524 1237.215 . . . -118.4810 34.3120 100.000 386.051 397.973 cvmsi 2435.524 1237.215 . . . -118.4810 34.3120 100.000 386.051 397.973 cvmsi 2435.524 1237.215 . . .
Search Regions
Define two search regions, one in LA Basin where CVM-S4 has a GTL defined. The second in the mojave where it is CVM-S4 background model. We will discretize at ~10.0m resolution. Basedon this link: Decimal Degrees, we'll descritze at 0.0001 resolution.
Search plans
We will use ucvm makemesh.sh script to create query points for each of these regions. We'll define a search layer from 80m to 120m at 10m resolution to determine whether the alternative values are at a single depth, or in a range of depths. We will add the original observed points to end of the file to ensure we reproduce the initial observations.
- Single query version
The first UCVM query will be a single query call for each point. It will output the expect ucvm results with useful results in the final columns.
- Double query version
The second ucvm query will contain duplicate points for each query. We will expect that in some cases, especially the observed data points above, the second query will return different values
- Triple query version
The third ucvm query will contain three duplicate points for each query. We will determine whether the second and third calls will create duplicates also.
- Query order tests
Query at 90m, query at 100m, query at 90m check if the two 90m queries are different.
- Query at different depths for same lat/lon
Indications are that two queries for the same lat/lon at say depth 20m, and at depth 100m, return different results than a single query at 100m.
Analysis
Post-processing scripts will determine:
- Total number of points in each file:
- Number of point in which first and second query are different
- Percent difference between first and second queries
- Location of points that produce differences
- Number of points in which second and third query are different
- Number of points in which first and third query are different
LABasin GTL Region
34.5 -118.5 33.5 -117.5
in KML:
<coordinates> -117.5,33.5,0 -117.5,34.5,0 -118.5,34.5,0 -118.5,33.5,0 -117.5,33.5,0 </coordinates>
Background Region
35.5 -117.5 34.5 -116.5
<coordinates> -116.5,34.5,0 -116.5,35.5,0 -117.5,35.5,0 -117.5,34.5,0 -116.5,34.5,0 </coordinates>
makemesh
We use the ucvm makemesh script to create the query points.
