Difference between revisions of "CVM S4 Testing"

From SCECpedia
Jump to navigationJump to search
(Created page with " We have identified points where CVM-S4 queries are stateful, specifically in the GTL region (~100m depth). By this, we mean that if we query the same point twice, CVM-S4 (and...")
 
 
(48 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
+
== Initial Observations Of CVM-S4 Stateful Behavior ==
 
We have identified points where CVM-S4 queries are stateful, specifically in the GTL region (~100m depth).
 
We have identified points where CVM-S4 queries are stateful, specifically in the GTL region (~100m depth).
 
By this, we mean that if we query the same point twice, CVM-S4 (and UCVM) returns different Vp, Vs, and rho properties for each query.
 
By this, we mean that if we query the same point twice, CVM-S4 (and UCVM) returns different Vp, Vs, and rho properties for each query.
  
 
To investigate this, we will run some tests to determine if all CVM-S4 regions are stateful, or only the GTL regions. We will also look at how large the differences
 
To investigate this, we will run some tests to determine if all CVM-S4 regions are stateful, or only the GTL regions. We will also look at how large the differences
 +
== Test Results Summary ==
 +
# Confirmed that repeated ucvm calls to CVM-S4 for the same point at the same depth can return different Vs values. We'll call this a stateful behavior issue.
 +
# Issue occurs when same point is queried at the same depth, and no intervening points are queried. In other words, calls to another lat lon reset ucvm, and after this reset, a second query for a lat lon returns returns the same value as the first call.
 +
# The issue occurs only in CVM-S4 GTL region. no stateful behavior was seen in outside the CVM-S4 GTL region.
 +
# The issue occurs for a fraction of the total lat lons in the GTL region (<10%).
 +
# The issue occurs at depths between 0-500m. It occurs most frequently at 300m. Drops off signifantly below 300m. Not observed for any calls below 500m.
 +
# More tests to further describe the problem are possible (e.g. test Salton Sea GTL region). But no obvious solution except contacting Kohler/Magistrale for code review.
 +
 +
== Test 1 ==
 +
Create a grid file using ucvm makegrid.sh with these settings. This will search the LA basin region at 10m resolution.
 +
 +
  HOR_SPACING=0.001
 +
  VERT_SPACING=10
 +
  MIN_DEPTH=90
 +
  MAX_DEPTH=110
 +
  MIN_LON=-118.5
 +
  MAX_LON=-117.5
 +
  MIN_LAT=33.5
 +
  MAX_LAT=34.5
 +
 +
Output file wc, showing 3M lines
 +
  3006003 9018009 59118059 out.grd
 +
 +
This file is contentated with itself. This will result in two calls to each point, but not consecutively. This tests whether the state-ful ness is reset when horizontal change is made.
 +
 +
Resulting populated file wc:
 +
  6012006 102204102 1124245122 cvms4test.res
 +
 +
Results of this test indicate that the second call returns the same result as the first call. The two calls are separated by 3006003 calls to other locations.
 +
 +
== Test 2 ==
 +
Create grid file with three duplicate lines this way:
 +
#makegrid.sh using min,max lat,lon above and depths 90 110 at 100m intervals
 +
#duplicate input lines using script on discovery duplic_pts.py
 +
#input grid with duplicat lines to ucvm_query
 +
#transfer populated results to laptop
 +
#add 3 lines for the known result into the poptulated results to confirm issue
 +
#run python script that reads populate results into dataframe an calculates differnce betweens cells 1 and 2, and 2 and 3.
 +
 +
(base) [maechlin@discovery2 test_ucvm]$ wc test2.grd
 +
  9018006  27054018 177354120 test2.grd
 +
 +
Add 3 rows with calls to the in the initial point where this issue was identified.
 +
 +
Summary results from Python scripts:
 +
 +
  Found  3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 )
 +
  Found  0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )
 +
 +
Script can be modified to determine additional information including lat,lons and percent differences found, if needed.
 +
 +
Checking the results for queries 2 and 3:
 +
 +
  cvms4_data = "/Users/maechlin/cvms4test2.res"
 +
 +
  Difference between call 2 and call 3: -154.118000 for row: 1820002
 +
  Difference between call 2 and call 3: -94.489000 for row: 4820002
 +
  Difference between call 2 and call 3: -286.251000 for row: 7820002
 +
  Found  3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 )
 +
  Found  0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )
 +
 +
After adding check for depths
 +
 +
  /Users/maechlin/anaconda3/bin/python /Users/maechlin/PycharmProjects/cvm4tests/main.py
 +
  Difference between call 2 and call 3: -154.118000 for row: 1820002
 +
  Difference between call 2 and call 3: -94.489000 for row: 4820002
 +
  Difference between call 2 and call 3: -286.251000 for row: 7820002
 +
  Found  3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 )
 +
  Found  0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )
 +
  Diffs at 90: 34476
 +
  Diffs at 100: 33708
 +
  Diffs at 110: 32811
 +
 +
== Test 3 ==
 +
Redo Test 2 but do the query in the background region, where there should not be any borehole data
 +
 +
  cvms4_data = "/Users/maechlin/cvms4test3.res"
 +
  Found  0.0 percent of 1 and 2 queries had differences ( 0 out of 3006002 )
 +
  Found  0.0 percent of 2 and 3 queries had differences: (0 out of 3006002 )
 +
 +
Add the initial observed points into file to confirm script finds it
 +
 +
  Found  0.0 percent of 1 and 2 queries had differences ( 1 out of 3006003 )
 +
  Found  0.0 percent of 2 and 3 queries had differences: (0 out of 3006003 )
 +
 +
== Test 4 ==
 +
Create a grid file at depths from 0-400m (at lower resolution). Duplicate the ucvm_calls at each depth 3 times. since new lat lons are called for each depth point, ucvm_query willl reset between duplicate calls. count at which depths we get alternative values between first and second ucvm-query calls for same depth.
 +
 +
  (base) [maechlin@discovery1 test_ucvm]$ wc test4.grd
 +
  1560750  4682250 27756873 test4.grd
 +
  (base) [maechlin@discovery1 test_ucvm]$ wc depth0to500.grd
 +
  520251 1560753 9252307 depth0to500.grd
 +
 +
HOR_SPACING=0.01
 +
VERT_SPACING=10
 +
MIN_DEPTH=0
 +
MAX_DEPTH=500
 +
MIN_LON=-118.5
 +
MAX_LON=-117.5
 +
MIN_LAT=33.5
 +
MAX_LAT=34.5
 +
 +
Mesh specifications:
 +
# each level has 10,201 pts
 +
# There are 51 levels for a 520251 pts in the original mesh file
 +
# each point is duplicated 3 time for 30603 pts per level and there are 51 levels which gives 1,560,753 (pts)
 +
 +
Results
 +
 +
<pre>
 +
/Users/maechlin/anaconda3/bin/python /Users/maechlin/PycharmProjects/cvm4tests/main.py
 +
Found  2.6 percent of 1 and 2 queries had differences ( 13311 out of 520251 )
 +
Found  0.0 percent of 2 and 3 queries had differences: (0 out of 520251 )
 +
Found 0 errors at depth 0
 +
Found 1 errors at depth 10
 +
Found 7 errors at depth 20
 +
Found 54 errors at depth 30
 +
Found 191 errors at depth 40
 +
Found 192 errors at depth 50
 +
Found 192 errors at depth 60
 +
Found 193 errors at depth 70
 +
Found 215 errors at depth 80
 +
Found 371 errors at depth 90
 +
Found 365 errors at depth 100
 +
Found 356 errors at depth 110
 +
Found 355 errors at depth 120
 +
Found 359 errors at depth 130
 +
Found 362 errors at depth 140
 +
Found 359 errors at depth 150
 +
Found 572 errors at depth 160
 +
Found 563 errors at depth 170
 +
Found 573 errors at depth 180
 +
Found 605 errors at depth 190
 +
Found 566 errors at depth 200
 +
Found 590 errors at depth 210
 +
Found 606 errors at depth 220
 +
Found 583 errors at depth 230
 +
Found 606 errors at depth 240
 +
Found 624 errors at depth 250
 +
Found 654 errors at depth 260
 +
Found 685 errors at depth 270
 +
Found 717 errors at depth 280
 +
Found 733 errors at depth 290
 +
Found 761 errors at depth 300
 +
Found 16 errors at depth 310
 +
Found 16 errors at depth 320
 +
Found 16 errors at depth 330
 +
Found 15 errors at depth 340
 +
Found 15 errors at depth 350
 +
Found 15 errors at depth 360
 +
Found 15 errors at depth 370
 +
Found 15 errors at depth 380
 +
Found 15 errors at depth 390
 +
Found 15 errors at depth 400
 +
Found 15 errors at depth 410
 +
Found 15 errors at depth 420
 +
Found 15 errors at depth 430
 +
Found 15 errors at depth 440
 +
Found 15 errors at depth 450
 +
Found 15 errors at depth 460
 +
Found 15 errors at depth 470
 +
Found 15 errors at depth 480
 +
Found 14 errors at depth 490
 +
Found 14 errors at depth 500
 +
Found 0 errors at depth 510
 +
Found 0 errors at depth 520
 +
Found 0 errors at depth 530
 +
Found 0 errors at depth 540
 +
Found 0 errors at depth 550
 +
Found 0 errors at depth 560
 +
Found 0 errors at depth 570
 +
Found 0 errors at depth 580
 +
Found 0 errors at depth 590
 +
Found 0 errors at depth 600
 +
Found 0 errors at depth 610
 +
Found 0 errors at depth 620
 +
Found 0 errors at depth 630
 +
Found 0 errors at depth 640
 +
Found 0 errors at depth 650
 +
Found 0 errors at depth 660
 +
Found 0 errors at depth 670
 +
Found 0 errors at depth 680
 +
Found 0 errors at depth 690
 +
Found 0 errors at depth 700
 +
Found 0 errors at depth 710
 +
Found 0 errors at depth 720
 +
Found 0 errors at depth 730
 +
Found 0 errors at depth 740
 +
Found 0 errors at depth 750
 +
Found 0 errors at depth 760
 +
Found 0 errors at depth 770
 +
Found 0 errors at depth 780
 +
Found 0 errors at depth 790
 +
Found 0 errors at depth 800
 +
Found 0 errors at depth 810
 +
Found 0 errors at depth 820
 +
Found 0 errors at depth 830
 +
Found 0 errors at depth 840
 +
Found 0 errors at depth 850
 +
Found 0 errors at depth 860
 +
Found 0 errors at depth 870
 +
Found 0 errors at depth 880
 +
Found 0 errors at depth 890
 +
Found 0 errors at depth 900
 +
Found 0 errors at depth 910
 +
Found 0 errors at depth 920
 +
Found 0 errors at depth 930
 +
Found 0 errors at depth 940
 +
Found 0 errors at depth 950
 +
Found 0 errors at depth 960
 +
Found 0 errors at depth 970
 +
Found 0 errors at depth 980
 +
Found 0 errors at depth 990
 +
Found 0 errors at depth 1000
 +
Total found: 13311
 +
</pre>
 +
 +
== Test Results Details ==
 +
# Test 1 results indicate that ucvm_query calls to that same lat lon return the same value if there are ucvm_query call to a different lat long between them. In order words, the problem occurs only if consecutive calls to the same lat lon are made.
 +
# Test 2 results indicate that only a fraction of lat lons show this problem, about 3% of lat lons in the LA Basin area. Plotting of the lat lons in the region show all occurances are in the CVMS-4 GTL region
 +
# Test 2 results indicate the the issue occurs fairly evenly for repeated queries at 90m, 100m, and 110 m depths.
 +
# KML file showing distribution of sites where query issues occur: [[file:UCVM Coverage Region v22 5.kml | UCVM KML File]]
 +
# Test 3 results indicate that the issue does not occur in the CVMS-4 background region, outside the the CVM-S4 GTL zone.
 +
# Test 4 results indicate that the issue occures down to 500m, but less frequently as queries go deeper. 500m seems the max depth which shows the issue.
 +
# KML file showing distribution of sites where query issues occur at 300m: [[file:UCVM_Coverage_Region_test4.kml | 300m stateful queries KML locations]]
 +
 +
Further testing
 +
# Check if the problem is observed with call to the same lat lon but different depths.
  
 
== Initial Observed Differences ==
 
== Initial Observed Differences ==
 +
  ucvm_query -m cvmsi -f $UCVM_HOME/conf/ucvm.conf < in > out
 +
 +
  with the input being:
 +
 +
  -118.481000 34.312000 100.0000
 +
  -118.481000 34.312000 100.0000
 +
  -118.481000 34.312000 100.0000
 +
  -118.481000 34.312000 100.0000
 +
 +
  returns the output:
 +
 +
  -118.4810    34.3120    100.000    386.051    397.973      cvmsi  2185.167    989.497  . . .
 +
  -118.4810    34.3120    100.000    386.051    397.973      cvmsi  2435.524  1237.215  . . .
 +
  -118.4810    34.3120    100.000    386.051    397.973      cvmsi  2435.524  1237.215  . . .
 +
  -118.4810    34.3120    100.000    386.051    397.973      cvmsi  2435.524  1237.215  . . .
 +
 +
== Search Regions ==
 +
Define two search regions, one in LA Basin where CVM-S4 has a GTL defined. The second in the mojave where it is CVM-S4 background model.
 +
We will discretize at ~10.0m resolution. Basedon this link: [http://wiki.gis.com/wiki/index.php/Decimal_degrees Decimal Degrees], we'll descritze at 0.0001 resolution.
 +
 +
[[Image:CVM_S4_Regions.png|right|250px]]
 +
 +
== Search plans ==
 +
We will use ucvm makemesh.sh script to create query points for each of these regions. We'll define a search layer from 80m to 120m at 10m resolution to determine whether the alternative values are at a single depth, or in a range of depths. We will add the original observed points to end of the file to ensure we reproduce the initial observations.
 +
 +
* Triple query version
 +
The ucvm query will contain three duplicate points for each query. We will determine whether the second and third calls will create duplicates also.
  
== Search Regions ===
+
* Query at different depths for same lat/lon
Define two search regions, one in LA Basin where CVM-S4 has a GTL define. The second in the mojave where it is CVM-S4 background model.
+
Indications are that two queries for the same lat/lon at say depth 20m, and at depth 100m, return different results than a single query at 100m.
We will discretize at ~10.0m resolution. Basedon this link: [http://wiki.gis.com/wiki/index.php/Decimal_degrees Decimal Degress], we'll descritze at 0.00001 resolution.
+
 
 +
== Analysis ==
 +
Post-processing scripts will determine:
 +
*Total number of points in each file:
 +
*Number of point in which first and second query are different
 +
*Percent difference between first and second queries
 +
*Location of points that produce differences
 +
*Number of points in which second and third query are different
 +
*Number of points in which first and third query are different
  
 
== LABasin GTL Region ==
 
== LABasin GTL Region ==
 
+
  34.5 -118.5
33.5 -118.5
+
  33.5 -117.5
34.5 -117.5
 
  
 
in KML:
 
in KML:
 
<pre>
 
<pre>
 
<coordinates>
 
<coordinates>
-117.4878354283172,33.50078334085499,0 -117.506525464026,34.49927596019899,0 -118.4974992809605,34.49415374897883,0 -118.4984281529209,33.50753106210945,0 -117.4878354283172,33.50078334085499,0  
+
-117.5,33.5,0 -117.5,34.5,0 -118.5,34.5,0 -118.5,33.5,0 -117.5,33.5,0  
 
</coordinates>
 
</coordinates>
 
</pre>
 
</pre>
Line 25: Line 287:
 
== Background Region ==
 
== Background Region ==
  
-117.5 -116.5
+
35.5 -117.5
34.5   35.5
+
34.5 -116.5
  
 
<pre>
 
<pre>
 
<coordinates>
 
<coordinates>
-116.5009162254955,34.50477958412391,0 -116.5031773420153,35.49513073811349,0 -117.498189162515,35.50401461257112,0 -117.4987935697126,34.50294855960493,0 -116.5009162254955,34.50477958412391,0  
+
-116.5,34.5,0 -116.5,35.5,0 -117.5,35.5,0 -117.5,34.5,0 -116.5,34.5,0  
 
</coordinates>
 
</coordinates>
 
</pre>
 
</pre>
 +
 +
== makemesh ==
 +
We use the ucvm makemesh script to create the query points.
 +
 +
== Related Entries ==
 +
*[[UCVM]]

Latest revision as of 20:13, 25 April 2022

Initial Observations Of CVM-S4 Stateful Behavior

We have identified points where CVM-S4 queries are stateful, specifically in the GTL region (~100m depth). By this, we mean that if we query the same point twice, CVM-S4 (and UCVM) returns different Vp, Vs, and rho properties for each query.

To investigate this, we will run some tests to determine if all CVM-S4 regions are stateful, or only the GTL regions. We will also look at how large the differences

Test Results Summary

  1. Confirmed that repeated ucvm calls to CVM-S4 for the same point at the same depth can return different Vs values. We'll call this a stateful behavior issue.
  2. Issue occurs when same point is queried at the same depth, and no intervening points are queried. In other words, calls to another lat lon reset ucvm, and after this reset, a second query for a lat lon returns returns the same value as the first call.
  3. The issue occurs only in CVM-S4 GTL region. no stateful behavior was seen in outside the CVM-S4 GTL region.
  4. The issue occurs for a fraction of the total lat lons in the GTL region (<10%).
  5. The issue occurs at depths between 0-500m. It occurs most frequently at 300m. Drops off signifantly below 300m. Not observed for any calls below 500m.
  6. More tests to further describe the problem are possible (e.g. test Salton Sea GTL region). But no obvious solution except contacting Kohler/Magistrale for code review.

Test 1

Create a grid file using ucvm makegrid.sh with these settings. This will search the LA basin region at 10m resolution.

 HOR_SPACING=0.001
 VERT_SPACING=10
 MIN_DEPTH=90
 MAX_DEPTH=110
 MIN_LON=-118.5
 MAX_LON=-117.5
 MIN_LAT=33.5
 MAX_LAT=34.5

Output file wc, showing 3M lines

 3006003 9018009 59118059 out.grd

This file is contentated with itself. This will result in two calls to each point, but not consecutively. This tests whether the state-ful ness is reset when horizontal change is made.

Resulting populated file wc:

 6012006 102204102 1124245122 cvms4test.res

Results of this test indicate that the second call returns the same result as the first call. The two calls are separated by 3006003 calls to other locations.

Test 2

Create grid file with three duplicate lines this way:

  1. makegrid.sh using min,max lat,lon above and depths 90 110 at 100m intervals
  2. duplicate input lines using script on discovery duplic_pts.py
  3. input grid with duplicat lines to ucvm_query
  4. transfer populated results to laptop
  5. add 3 lines for the known result into the poptulated results to confirm issue
  6. run python script that reads populate results into dataframe an calculates differnce betweens cells 1 and 2, and 2 and 3.

(base) [maechlin@discovery2 test_ucvm]$ wc test2.grd

 9018006  27054018 177354120 test2.grd

Add 3 rows with calls to the in the initial point where this issue was identified.

Summary results from Python scripts:

 Found  3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 )
 Found  0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )

Script can be modified to determine additional information including lat,lons and percent differences found, if needed.

Checking the results for queries 2 and 3:

 cvms4_data = "/Users/maechlin/cvms4test2.res"
 Difference between call 2 and call 3: -154.118000 for row: 1820002
 Difference between call 2 and call 3: -94.489000 for row: 4820002
 Difference between call 2 and call 3: -286.251000 for row: 7820002
 Found  3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 )
 Found  0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )

After adding check for depths

 /Users/maechlin/anaconda3/bin/python /Users/maechlin/PycharmProjects/cvm4tests/main.py
 Difference between call 2 and call 3: -154.118000 for row: 1820002
 Difference between call 2 and call 3: -94.489000 for row: 4820002
 Difference between call 2 and call 3: -286.251000 for row: 7820002
 Found  3.4 percent of 1 and 2 queries had differences ( 100995 out of 3006003 )
 Found  0.0 percent of 2 and 3 queries had differences: (3 out of 3006003 )
 Diffs at 90: 34476
 Diffs at 100: 33708
 Diffs at 110: 32811

Test 3

Redo Test 2 but do the query in the background region, where there should not be any borehole data

 cvms4_data = "/Users/maechlin/cvms4test3.res"
 Found  0.0 percent of 1 and 2 queries had differences ( 0 out of 3006002 )
 Found  0.0 percent of 2 and 3 queries had differences: (0 out of 3006002 )

Add the initial observed points into file to confirm script finds it

 Found  0.0 percent of 1 and 2 queries had differences ( 1 out of 3006003 )
 Found  0.0 percent of 2 and 3 queries had differences: (0 out of 3006003 )

Test 4

Create a grid file at depths from 0-400m (at lower resolution). Duplicate the ucvm_calls at each depth 3 times. since new lat lons are called for each depth point, ucvm_query willl reset between duplicate calls. count at which depths we get alternative values between first and second ucvm-query calls for same depth.

 (base) [maechlin@discovery1 test_ucvm]$ wc test4.grd
  1560750  4682250 27756873 test4.grd
 (base) [maechlin@discovery1 test_ucvm]$ wc depth0to500.grd 
 520251 1560753 9252307 depth0to500.grd

HOR_SPACING=0.01 VERT_SPACING=10 MIN_DEPTH=0 MAX_DEPTH=500 MIN_LON=-118.5 MAX_LON=-117.5 MIN_LAT=33.5 MAX_LAT=34.5

Mesh specifications:

  1. each level has 10,201 pts
  2. There are 51 levels for a 520251 pts in the original mesh file
  3. each point is duplicated 3 time for 30603 pts per level and there are 51 levels which gives 1,560,753 (pts)

Results

/Users/maechlin/anaconda3/bin/python /Users/maechlin/PycharmProjects/cvm4tests/main.py
Found  2.6 percent of 1 and 2 queries had differences ( 13311 out of 520251 )
Found  0.0 percent of 2 and 3 queries had differences: (0 out of 520251 )
Found 0 errors at depth 0
Found 1 errors at depth 10
Found 7 errors at depth 20
Found 54 errors at depth 30
Found 191 errors at depth 40
Found 192 errors at depth 50
Found 192 errors at depth 60
Found 193 errors at depth 70
Found 215 errors at depth 80
Found 371 errors at depth 90
Found 365 errors at depth 100
Found 356 errors at depth 110
Found 355 errors at depth 120
Found 359 errors at depth 130
Found 362 errors at depth 140
Found 359 errors at depth 150
Found 572 errors at depth 160
Found 563 errors at depth 170
Found 573 errors at depth 180
Found 605 errors at depth 190
Found 566 errors at depth 200
Found 590 errors at depth 210
Found 606 errors at depth 220
Found 583 errors at depth 230
Found 606 errors at depth 240
Found 624 errors at depth 250
Found 654 errors at depth 260
Found 685 errors at depth 270
Found 717 errors at depth 280
Found 733 errors at depth 290
Found 761 errors at depth 300
Found 16 errors at depth 310
Found 16 errors at depth 320
Found 16 errors at depth 330
Found 15 errors at depth 340
Found 15 errors at depth 350
Found 15 errors at depth 360
Found 15 errors at depth 370
Found 15 errors at depth 380
Found 15 errors at depth 390
Found 15 errors at depth 400
Found 15 errors at depth 410
Found 15 errors at depth 420
Found 15 errors at depth 430
Found 15 errors at depth 440
Found 15 errors at depth 450
Found 15 errors at depth 460
Found 15 errors at depth 470
Found 15 errors at depth 480
Found 14 errors at depth 490
Found 14 errors at depth 500
Found 0 errors at depth 510
Found 0 errors at depth 520
Found 0 errors at depth 530
Found 0 errors at depth 540
Found 0 errors at depth 550
Found 0 errors at depth 560
Found 0 errors at depth 570
Found 0 errors at depth 580
Found 0 errors at depth 590
Found 0 errors at depth 600
Found 0 errors at depth 610
Found 0 errors at depth 620
Found 0 errors at depth 630
Found 0 errors at depth 640
Found 0 errors at depth 650
Found 0 errors at depth 660
Found 0 errors at depth 670
Found 0 errors at depth 680
Found 0 errors at depth 690
Found 0 errors at depth 700
Found 0 errors at depth 710
Found 0 errors at depth 720
Found 0 errors at depth 730
Found 0 errors at depth 740
Found 0 errors at depth 750
Found 0 errors at depth 760
Found 0 errors at depth 770
Found 0 errors at depth 780
Found 0 errors at depth 790
Found 0 errors at depth 800
Found 0 errors at depth 810
Found 0 errors at depth 820
Found 0 errors at depth 830
Found 0 errors at depth 840
Found 0 errors at depth 850
Found 0 errors at depth 860
Found 0 errors at depth 870
Found 0 errors at depth 880
Found 0 errors at depth 890
Found 0 errors at depth 900
Found 0 errors at depth 910
Found 0 errors at depth 920
Found 0 errors at depth 930
Found 0 errors at depth 940
Found 0 errors at depth 950
Found 0 errors at depth 960
Found 0 errors at depth 970
Found 0 errors at depth 980
Found 0 errors at depth 990
Found 0 errors at depth 1000
Total found: 13311

Test Results Details

  1. Test 1 results indicate that ucvm_query calls to that same lat lon return the same value if there are ucvm_query call to a different lat long between them. In order words, the problem occurs only if consecutive calls to the same lat lon are made.
  2. Test 2 results indicate that only a fraction of lat lons show this problem, about 3% of lat lons in the LA Basin area. Plotting of the lat lons in the region show all occurances are in the CVMS-4 GTL region
  3. Test 2 results indicate the the issue occurs fairly evenly for repeated queries at 90m, 100m, and 110 m depths.
  4. KML file showing distribution of sites where query issues occur: File:UCVM Coverage Region v22 5.kml
  5. Test 3 results indicate that the issue does not occur in the CVMS-4 background region, outside the the CVM-S4 GTL zone.
  6. Test 4 results indicate that the issue occures down to 500m, but less frequently as queries go deeper. 500m seems the max depth which shows the issue.
  7. KML file showing distribution of sites where query issues occur at 300m: File:UCVM Coverage Region test4.kml

Further testing

  1. Check if the problem is observed with call to the same lat lon but different depths.

Initial Observed Differences

 ucvm_query -m cvmsi -f $UCVM_HOME/conf/ucvm.conf < in > out

 with the input being:

 -118.481000 34.312000 100.0000
 -118.481000 34.312000 100.0000
 -118.481000 34.312000 100.0000
 -118.481000 34.312000 100.0000

 returns the output:

  -118.4810    34.3120    100.000    386.051    397.973      cvmsi   2185.167    989.497   . . .
  -118.4810    34.3120    100.000    386.051    397.973      cvmsi   2435.524   1237.215   . . .
  -118.4810    34.3120    100.000    386.051    397.973      cvmsi   2435.524   1237.215   . . .
  -118.4810    34.3120    100.000    386.051    397.973      cvmsi   2435.524   1237.215   . . .

Search Regions

Define two search regions, one in LA Basin where CVM-S4 has a GTL defined. The second in the mojave where it is CVM-S4 background model. We will discretize at ~10.0m resolution. Basedon this link: Decimal Degrees, we'll descritze at 0.0001 resolution.

CVM S4 Regions.png

Search plans

We will use ucvm makemesh.sh script to create query points for each of these regions. We'll define a search layer from 80m to 120m at 10m resolution to determine whether the alternative values are at a single depth, or in a range of depths. We will add the original observed points to end of the file to ensure we reproduce the initial observations.

  • Triple query version

The ucvm query will contain three duplicate points for each query. We will determine whether the second and third calls will create duplicates also.

  • Query at different depths for same lat/lon

Indications are that two queries for the same lat/lon at say depth 20m, and at depth 100m, return different results than a single query at 100m.

Analysis

Post-processing scripts will determine:

  • Total number of points in each file:
  • Number of point in which first and second query are different
  • Percent difference between first and second queries
  • Location of points that produce differences
  • Number of points in which second and third query are different
  • Number of points in which first and third query are different

LABasin GTL Region 

 34.5 -118.5
 33.5 -117.5

in KML:

<coordinates>
-117.5,33.5,0 -117.5,34.5,0 -118.5,34.5,0 -118.5,33.5,0 -117.5,33.5,0 
</coordinates>

Background Region

35.5 -117.5
34.5 -116.5
<coordinates>
-116.5,34.5,0 -116.5,35.5,0 -117.5,35.5,0 -117.5,34.5,0 -116.5,34.5,0 
</coordinates>

makemesh

We use the ucvm makemesh script to create the query points.

Related Entries