CSEP1 Status

CSEP Working Group Home Page

Introduction

The CSEP1 system has been in production for over a decade with dozens of models running prospectively during this time period. Currently, we do not have an understanding of the status of the operational system with respect to missing forecasts and their evaluations. This page provides the status of the CSEP1 system and outlines our work toward understanding the status of the CSEP1 operational system with respect to one-day rate-based forecasting models.

Results

The link below shows the results for the CSEP1 SCEC testing center.

http://hypocenter.usc.edu/research/csep1/CSEP1.0_Status_v0.9.html

Data Model

CSEP1 simulation meta data represented as SQL schema. Extraction scripts can be found on GitHub at https://github.com/wsavran/csep_db/.

   CREATE TABLE IF NOT EXISTS ScheduledForecasts (
       scheduled_forecast_id INTEGER PRIMARY KEY,
       date_time TEXT NOT NULL
   );

   CREATE TABLE IF NOT EXISTS ScheduledEvaluations (
       scheduled_evaluation_id INTEGER PRIMARY KEY,
       date_time TEXT NOT NULL
   );

   CREATE TABLE IF NOT EXISTS Dispatchers (
       dispatcher_id INTEGER PRIMARY KEY,
       script_name TEXT NOT NULL,
       config_file_name TEXT NOT NULL
   );

   CREATE TABLE IF NOT EXISTS ForecastGroups (
       forecastgroup_id INTEGER PRIMARY KEY,
       group_name TEXT NOT NULL,
       config_filepath TEXT NOT NULL,
       dispatcher_id INTEGER NOT NULL,
       FOREIGN KEY(dispatcher_id) REFERENCES Dispatchers
   );

   CREATE TABLE IF NOT EXISTS Forecasts (
       forecast_id INTEGER PRIMARY KEY,
       schedule_id INTEGER NOT NULL,
       group_id INTEGER NOT NULL,
       name TEXT NOT NULL,
       filepath TEXT,
       meta_filepath TEXT,
       waiting_period TEXT,
       logfile TEXT,
       status TEXT,
       FOREIGN KEY(schedule_id) REFERENCES ScheduledForecasts,
       FOREIGN KEY(group_id) REFERENCES ForecastGroups
   );

   CREATE TABLE IF NOT EXISTS Evaluations (
       evaluation_id INTEGER PRIMARY KEY,
       scheduled_id INTEGER NOT NULL,
       forecast_id INTEGER NOT NULL,
       compute_datetime TEXT,
       filepath TEXT,
       name TEXT,
       status TEXT,
       FOREIGN KEY(scheduled_id) REFERENCES ScheduledEvaluations,
       FOREIGN KEY(forecast_id) REFERENCES Forecasts
   );

Verifying Database

In this section, we manually verify the integrity of the csep_db contents to ensure that the database is accurately representing the information present on the CSEP operational system. We show the verification results for a single dispatcher script.

Dispatcher Scripts

sqlite> select * from Dispatchers;
1|/usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.tcsh|31|/usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.init.xml
2|/usr/local/csep/cronjobs/dispatcher_ANSS1985_M2_95.tcsh|31|/usr/local/csep/cronjobs/dispatcher_ANSS1985_M2_95.init.xml
3|/usr/local/csep/cronjobs/dispatcher_ANSS1932_notFiltered_Md2_one_day.tcsh|31|/usr/local/csep/cronjobs/dispatcher_ANSS1932_notFiltered_Md2_one_day.init.xml

Contents of dispatcher_id=1

cat /usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.tcsh
#!/bin/tcsh

# dispatcher_ANSS1985.tcsh:
#
# Run daily Dispatcher for one-day and RELM forecasts models generation and evaluation.
# All groups use ANSS catalog with start date of 1985-01-01 and Magnitude>=3.0.
#

source ~/.tcshrc

set year=`date '+%Y'`
# Don't pad integer values with zero's
set month=`date '+%_m'`
set day=`date '+%_d'`

# Pad time values with zero's
set hour=`date '+%H'`
set min=`date '+%M'`
set sec=`date '+%S'`


# Capture all output produced by Dispatcher into the daily log file
set logdir=$CSEP/dispatcher/logs/"$year"_"$month"
mkdir -p "$logdir"

set logfile="$logdir"/dailyANSS1985_"$year-$month-$day-$hour$min$sec"


# Invoke Dispatcher for "today" (with 31 day delay for the testing date):
nohup python3 $CENTERCODE/src/generic/Dispatcher.py --year="$year" --month="$month" --day="$day" --configFile=/usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.init.xml --waitingPeriod=31 
--enableForecastXMLTemplate --enableForecastCompression --enableForecastMap 
--publishServer=csep-usc@cseppublishing.usc.edu --publishDirectory=/export/cseppublishing/csep/us/results/data/us/usc/california --logFile="$logfile" >& "$logfile" &

Forecast Groups

Here, we verify that the correct forecast groups were parsed from the Dispatcher configuration file, and the forecast group configuration files are properly represented in the database. Forecast groups that have no scheduled evaluation tests are not included in the database.

1. Group names and filepaths

sqlite> select group_name, group_path from ForecastGroups;
one-day-models|/home/csep/operations/SCEC-natural-laboratory/one-day-models
one-day-models-V9.1|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1
one-day-models-V10.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10
one-day-models-V12.7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7
one-day-models-V12.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10
one-day-models-V16.4|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.4
one-day-models-V16.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10
one-day-models-V16.7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.7
one-day-models-RELMTests-V16.7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-RELMTests-V16.7
one-day-models-Md2-V12.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10

Note: Forecast groups with no evaluations scheduled are not included in this report.

[csep@csep-op ~]$ cat /usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.init.xml
<?xml version="1.0" encoding="utf-8"?>
<Dispatcher xmlns="http://www.scec.org/xml-ns/csep/0.1">
   <directory>/home/csep/operations/dispatcher/runs</directory>

   <email smtphost="email.usc.edu"
          from="johnyu@usc.edu"
          subjectPrefix="California one-day models evaluation (ANSS: 1985-01-01)">
       fsilva@usc.edu maechlin@usc.edu wsavran@usc.edu
   </email>

   <!-- One day models -->
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models</forecastGroup>
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1</forecastGroup>
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10</forecastGroup>
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7</forecastGroup>   
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10</forecastGroup>
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.4</forecastGroup>
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10</forecastGroup>
   <forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-optimized</forecastGroup>

</Dispatcher>

Note: the one-day-models-optimized group does not contain any evaluations; hence it's not listed in the database.

2. Config Files

sqlite> select rowid, config_filepath from ForecastGroups;
1|/home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml
2|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1/forecast.init.xml
3|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml
4|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/forecast.init.xml
5|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10/forecast.init.xml
6|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.4/forecast.init.xml
7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10/forecast.init.xml
8|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.7/forecast.init.xml
9|/home/csep/operations/SCEC-natural-laboratory/one-day-models-RELMTests-V16.7/forecast.init.xml
10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10/forecast.init.xml

Showing first 3 forecast group configuration files exist on the system.

[csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml
/home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml

[csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1/forecast.init.xml
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1/forecast.init.xml

[csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml

[csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/forecast.init.xml
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/forecast.init.xml

3. Expected Forecasts in each group

select ForecastGroups.group_name, group_concat(distinct Forecasts.name) from ForecastGroups join Forecasts on ForecastGroups.forecastgroup_id=Forecasts.group_id group by ForecastGroups.group_name;
1|one-day-models|ETAS
2|one-day-models-V9.1|KJSSOneDayCalifornia
3|one-day-models-V10.10|STEPJAVA
4|one-day-models-V12.7|ETASV1.1
5|one-day-models-V12.10|ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3
6|one-day-models-V16.4|OneDayBayesianBMA,OneDayBayesianSeqBMA
7|one-day-models-V16.10|GSF_ISO,GSF_ANISO
8|one-day-models-V16.7|ETASSYN_DROneDayMd2.95
10|one-day-models-Md2-V12.10|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,ETAS_DROneDayMd2,ETAS_DROneDayPPEMd2,JANUSOneDay,JANUSOneDayPPE,JANUSOneDayEEPAS1F,JANUSOneDayTV,SE2OneDay

Note: In some cases forecasts are present in multiple forecasts groups, but are archived in the same location. In these cases, the models are associated with the earliest forecast group. Showing results for Groups (1, 3, 7).

cat /home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml
<ns0:ForecastGroup xmlns:ns0="http://www.scec.org/xml-ns/csep/0.1" name="One-day models">
  <ns0:forecastDir>forecasts</ns0:forecastDir>
  <ns0:resultDir>results</ns0:resultDir>
  <ns0:catalogDir>observations</ns0:catalogDir>
  <ns0:postProcessing>OneDayModel</ns0:postProcessing>
  <ns0:entryDate>2007-08-01 00:00:00</ns0:entryDate>
  <ns0:models filenameIncludesStartDate="True" files="ETAS_5_5_2018-fromXML.dat.targz">
      ETAS

[csep@csep-op ~]$ cat /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml
<ns0:ForecastGroup xmlns:ns0="http://www.scec.org/xml-ns/csep/0.1" name="One-day models introduced in CSEP V10.10.0 due to submission of STEP_Java model by Matt Gerstenberger ">
  <ns0:forecastDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/forecasts</ns0:forecastDir>
  <ns0:resultDir>results</ns0:resultDir>
  <ns0:catalogDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/observations</ns0:catalogDir>
  <ns0:postProcessing>OneDayModel</ns0:postProcessing>
  <ns0:entryDate>2010-10-01 00:00:00</ns0:entryDate>
  <ns0:models filenameIncludesStartDate="True" files="ETAS_5_5_2018-fromXML.dat.targz KJSSOneDayCalifornia_5_5_2018-fromXML.dat.targz STEPJAVA_5_5_2018-fromXML.dat.targz">
      ETAS KJSSOneDayCalifornia STEPJAVA

cat forecast.init.xml
<ns0:ForecastGroup xmlns:ns0="http://www.scec.org/xml-ns/csep/0.1" name="CSEP V12.10.0 new forecast group due to submission of ETAS_HW, K3 models by Agnes Helmstetter and Max Werner, and ETAS_DR model by David Rhoades">
  <ns0:forecastDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/forecasts</ns0:forecastDir>
  <ns0:resultDir>results</ns0:resultDir>
  <ns0:catalogDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/observations</ns0:catalogDir>
  <ns0:postProcessing>OneDayModel</ns0:postProcessing>
  <ns0:entryDate>2012-10-01 00:00:00</ns0:entryDate>
  <ns0:models filenameIncludesStartDate="True" files="ETASV1.1_2_5_2017-fromXML.dat.targz ETASV1.1_9_26_2016-fromXML.dat.targz ETAS_9_26_2016-fromXML.dat.targz ETAS_HWMd3_2_5_2017-fromXML.dat.targz ETAS_HWMd3_9_26_2016-fromXML.dat.targz K3Md3_9_26_2016-fromXML.dat.targz KJSSOneDayCalifornia_9_26_2016-fromXML.dat.targz STEPJAVA_9_26_2016-fromXML.dat.targz">
      ETAS KJSSOneDayCalifornia STEPJAVA ETASV1.1 ETAS_HW K3 ETAS_DROneDay
      <ns0:inputs>||||Md=3|Md=3|Md=3</ns0:inputs>
     <ns0:hybridModel name="ETAS_HW_K3_AVERAGE_Md3">
        <ns0:component>
           <ns0:group weights="[0.5, 0.5]">ETAS_HW K3</ns0:group>
         </ns0:component>
      </ns0:hybridModel>

4. Start date of each forecast and the associated Forecast Group

sqlite> select ForecastGroups.group_name, Forecasts.name, min(ScheduledForecasts.date_time) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id join ForecastGroups on Forecasts.group_id=ForecastGroups.forecastgroup_id group by Forecasts.name;
one-day-models|ETAS|2007-08-01 00:00:00
one-day-models-V9.1|KJSSOneDayCalifornia|2009-01-01 00:00:00
one-day-models-V10.10|STEPJAVA|2010-10-01 00:00:00
one-day-models-V12.7|ETASV1.1|2012-07-01 00:00:00
one-day-models-Md2-V12.10|ETAS_DROneDayMd2|2012-10-01 00:00:00
one-day-models-V12.10|ETAS_DROneDayMd3|2012-10-01 00:00:00
one-day-models-Md2-V12.10|ETAS_DROneDayPPEMd2|2012-10-01 00:00:00
one-day-models-V12.10|ETAS_DROneDayPPEMd3|2012-10-01 00:00:00
one-day-models-Md2-V12.10|ETAS_HWMd2|2012-10-01 00:00:00
one-day-models-V12.10|ETAS_HWMd3|2012-10-01 00:00:00
one-day-models-Md2-V12.10|ETAS_HW_K3_AVERAGE_Md2|2012-10-01 00:00:00
one-day-models-V12.10|ETAS_HW_K3_AVERAGE_Md3|2012-10-01 00:00:00
one-day-models-Md2-V12.10|JANUSOneDay|2012-10-01 00:00:00
one-day-models-Md2-V12.10|JANUSOneDayEEPAS1F|2012-10-01 00:00:00
one-day-models-Md2-V12.10|JANUSOneDayPPE|2012-10-01 00:00:00
one-day-models-Md2-V12.10|JANUSOneDayTV|2012-10-01 00:00:00
one-day-models-Md2-V12.10|K3Md2|2012-10-01 00:00:00
one-day-models-V12.10|K3Md3|2012-10-01 00:00:00
one-day-models-V16.4|OneDayBayesianBMA|2012-10-01 00:00:00
one-day-models-V16.4|OneDayBayesianSeqBMA|2012-10-01 00:00:00
one-day-models-Md2-V12.10|SE2OneDay|2012-10-01 00:00:00
one-day-models-V16.7|ETASSYN_DROneDayMd2.95|2016-07-01 00:00:00
one-day-models-V16.10|GSF_ANISO|2016-10-01 00:00:00
one-day-models-V16.10|GSF_ISO|2016-10-01 00:00:00

Forecasts

1. Missing Forecasts in Each Group

We will manually evaluate a subset of the forecasts and show inductively the data are correct.

2007-08-01 00:00:00|ETAS|0
2009-01-01 00:00:00|KJSSOneDayCalifornia|0
2010-10-01 00:00:00|STEPJAVA|0
2012-07-01 00:00:00|ETASV1.1|4
2012-10-01 00:00:00|ETAS_DROneDayMd2|17
2012-10-01 00:00:00|ETAS_DROneDayMd3|5
2012-10-01 00:00:00|ETAS_DROneDayPPEMd2|17
2012-10-01 00:00:00|ETAS_DROneDayPPEMd3|5
2012-10-01 00:00:00|ETAS_HWMd2|601
2012-10-01 00:00:00|ETAS_HWMd3|5
2012-10-01 00:00:00|ETAS_HW_K3_AVERAGE_Md2|601
2012-10-01 00:00:00|ETAS_HW_K3_AVERAGE_Md3|10
2012-10-01 00:00:00|JANUSOneDay|17
2012-10-01 00:00:00|JANUSOneDayEEPAS1F|17
2012-10-01 00:00:00|JANUSOneDayPPE|17
2012-10-01 00:00:00|JANUSOneDayTV|17
2012-10-01 00:00:00|K3Md2|601
2012-10-01 00:00:00|K3Md3|5
2012-10-01 00:00:00|OneDayBayesianBMA|595
2012-10-01 00:00:00|OneDayBayesianSeqBMA|595
2012-10-01 00:00:00|SE2OneDay|84
2016-07-01 00:00:00|ETASSYN_DROneDayMd2.95|11
2016-10-01 00:00:00|GSF_ANISO|1
2016-10-01 00:00:00|GSF_ISO|1

2. Missing Forecasts by Day

We can expand this list to show the missing forecasts grouped by day, which we manually verify for a few different forecasts. We start by selecting the first 10 days containing missing forecasts for all of the one-day-models found on the system.

sqlite> select ScheduledForecasts.date_time, group_concat(Forecasts.name) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id where status='Missing' and ScheduledForecasts.date_time < date('2018-05-10') group by ScheduledForecasts.date_time limit 10;
2016-09-20 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2
2016-09-21 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-22 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-23 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-24 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-25 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-26 00:00:00|ETAS_HW_K3_AVERAGE_Md3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-27 00:00:00|ETASV1.1,ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-28 00:00:00|ETASV1.1,ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-29 00:00:00|ETASV1.1,ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay

We take a subset of this group, namely one-day-models-Md2-V12.10 (id=10).

sqlite> select ScheduledForecasts.date_time, group_concat(Forecasts.name) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id where status='Missing' and ScheduledForecasts.date_time < date('2018-05-10') and Forecasts.group_id=10 group by ScheduledForecasts.date_time limit 10;
2016-09-20 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2
2016-09-21 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-22 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-23 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-24 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-25 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-26 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-27 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-28 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-29 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay

Notice, the SE2OneDay model does not show forecasts past 2016-09-20 in the archive dir /home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10/forecasts/archive/2016_9/.

[csep@csep-op 2016_9]$ pwd
/home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10/forecasts/archive/2016_9
[csep@csep-op 2016_9]$ ls *SE2OneDay*
SE2OneDay_9_10_2016-fromXML.dat.meta        SE2OneDay_9_12_2016-fromXML.dat.targz.meta  SE2OneDay_9_15_2016-fromXML.xml.meta        SE2OneDay_9_19_2016-fromXML.dat.targz       SE2OneDay_9_3_2016-fromXML.xml             SE2OneDay_9_7_2016-fromXML.dat.meta
SE2OneDay_9_10_2016-fromXML.dat.targz       SE2OneDay_9_12_2016-fromXML.xml             SE2OneDay_9_16_2016-fromXML.dat.meta        SE2OneDay_9_19_2016-fromXML.dat.targz.meta  SE2OneDay_9_3_2016-fromXML.xml.meta        SE2OneDay_9_7_2016-fromXML.dat.targz
SE2OneDay_9_10_2016-fromXML.dat.targz.meta  SE2OneDay_9_12_2016-fromXML.xml.meta        SE2OneDay_9_16_2016-fromXML.dat.targz       SE2OneDay_9_19_2016-fromXML.xml             SE2OneDay_9_4_2016-fromXML.dat.meta        SE2OneDay_9_7_2016-fromXML.dat.targz.meta
SE2OneDay_9_10_2016-fromXML.xml             SE2OneDay_9_13_2016-fromXML.dat.meta        SE2OneDay_9_16_2016-fromXML.dat.targz.meta  SE2OneDay_9_19_2016-fromXML.xml.meta        SE2OneDay_9_4_2016-fromXML.dat.targz       SE2OneDay_9_7_2016-fromXML.xml
SE2OneDay_9_10_2016-fromXML.xml.meta        SE2OneDay_9_13_2016-fromXML.dat.targz       SE2OneDay_9_16_2016-fromXML.xml             SE2OneDay_9_20_2016-fromXML.dat.meta        SE2OneDay_9_4_2016-fromXML.dat.targz.meta  SE2OneDay_9_7_2016-fromXML.xml.meta
SE2OneDay_9_11_2016-fromXML.dat.meta        SE2OneDay_9_13_2016-fromXML.dat.targz.meta  SE2OneDay_9_16_2016-fromXML.xml.meta        SE2OneDay_9_20_2016-fromXML.dat.targz       SE2OneDay_9_4_2016-fromXML.xml             SE2OneDay_9_8_2016-fromXML.dat.meta
SE2OneDay_9_11_2016-fromXML.dat.targz       SE2OneDay_9_13_2016-fromXML.xml             SE2OneDay_9_17_2016-fromXML.dat.meta        SE2OneDay_9_20_2016-fromXML.dat.targz.meta  SE2OneDay_9_4_2016-fromXML.xml.meta        SE2OneDay_9_8_2016-fromXML.dat.targz
SE2OneDay_9_11_2016-fromXML.dat.targz.meta  SE2OneDay_9_13_2016-fromXML.xml.meta        SE2OneDay_9_17_2016-fromXML.dat.targz       SE2OneDay_9_20_2016-fromXML.xml             SE2OneDay_9_5_2016-fromXML.dat.meta        SE2OneDay_9_8_2016-fromXML.dat.targz.meta
SE2OneDay_9_11_2016-fromXML.xml             SE2OneDay_9_14_2016-fromXML.dat.meta        SE2OneDay_9_17_2016-fromXML.dat.targz.meta  SE2OneDay_9_20_2016-fromXML.xml.meta        SE2OneDay_9_5_2016-fromXML.dat.targz       SE2OneDay_9_8_2016-fromXML.xml
SE2OneDay_9_11_2016-fromXML.xml.meta        SE2OneDay_9_14_2016-fromXML.dat.targz       SE2OneDay_9_17_2016-fromXML.xml             SE2OneDay_9_2_2016-fromXML.dat.meta         SE2OneDay_9_5_2016-fromXML.dat.targz.meta  SE2OneDay_9_8_2016-fromXML.xml.meta
SE2OneDay_9_1_2016-fromXML.dat.meta         SE2OneDay_9_14_2016-fromXML.dat.targz.meta  SE2OneDay_9_17_2016-fromXML.xml.meta        SE2OneDay_9_2_2016-fromXML.dat.targz        SE2OneDay_9_5_2016-fromXML.xml             SE2OneDay_9_9_2016-fromXML.dat.meta
SE2OneDay_9_1_2016-fromXML.dat.targz        SE2OneDay_9_14_2016-fromXML.xml             SE2OneDay_9_18_2016-fromXML.dat.meta        SE2OneDay_9_2_2016-fromXML.dat.targz.meta   SE2OneDay_9_5_2016-fromXML.xml.meta        SE2OneDay_9_9_2016-fromXML.dat.targz
SE2OneDay_9_1_2016-fromXML.dat.targz.meta   SE2OneDay_9_14_2016-fromXML.xml.meta        SE2OneDay_9_18_2016-fromXML.dat.targz       SE2OneDay_9_2_2016-fromXML.xml              SE2OneDay_9_6_2016-fromXML.dat.meta        SE2OneDay_9_9_2016-fromXML.dat.targz.meta
SE2OneDay_9_1_2016-fromXML.xml              SE2OneDay_9_15_2016-fromXML.dat.meta        SE2OneDay_9_18_2016-fromXML.dat.targz.meta  SE2OneDay_9_2_2016-fromXML.xml.meta         SE2OneDay_9_6_2016-fromXML.dat.targz       SE2OneDay_9_9_2016-fromXML.xml
SE2OneDay_9_1_2016-fromXML.xml.meta         SE2OneDay_9_15_2016-fromXML.dat.targz       SE2OneDay_9_18_2016-fromXML.xml             SE2OneDay_9_3_2016-fromXML.dat.meta         SE2OneDay_9_6_2016-fromXML.dat.targz.meta  SE2OneDay_9_9_2016-fromXML.xml.meta
SE2OneDay_9_12_2016-fromXML.dat.meta        SE2OneDay_9_15_2016-fromXML.dat.targz.meta  SE2OneDay_9_18_2016-fromXML.xml.meta        SE2OneDay_9_3_2016-fromXML.dat.targz        SE2OneDay_9_6_2016-fromXML.xml
SE2OneDay_9_12_2016-fromXML.dat.targz       SE2OneDay_9_15_2016-fromXML.xml             SE2OneDay_9_19_2016-fromXML.dat.meta        SE2OneDay_9_3_2016-fromXML.dat.targz.meta   SE2OneDay_9_6_2016-fromXML.xml.meta

Next, will check the completed forecasts for 2016-09-20 to verify that all forecasts for this day were identified by the system.

sqlite> select ScheduledForecasts.date_time, group_concat(Forecasts.name) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id where status='Complete' and ScheduledForecasts.date_time = datetime('2016-09-21 00:00:00') and group_id=10 limit 10;
2016-09-21 00:00:00|ETAS_DROneDayMd2,ETAS_DROneDayPPEMd2,JANUSOneDay,JANUSOneDayEEPAS1F,JANUSOneDayPPE,JANUSOneDayTV

We list the files in the archive directory to verify these files are present on the operational system.

[csep@csep-op 2016_9]$ ls *9_21_2016*
ETAS_DROneDayMd2_9_21_2016.dat                     ETAS_DROneDayPPEMd2_9_21_2016.dat.meta                JANUSOneDay_9_21_2016-fromXML.dat.targz.meta         JANUSOneDayEEPAS1F_9_21_2016.xml                 JANUSOneDayPPE_9_21_2016.xml.meta
ETAS_DROneDayMd2_9_21_2016.dat.meta                ETAS_DROneDayPPEMd2_9_21_2016-fromXML.dat.targz       JANUSOneDay_9_21_2016-fromXML.xml                    JANUSOneDayEEPAS1F_9_21_2016.xml.meta            JANUSOneDayTV_9_21_2016.dat
ETAS_DROneDayMd2_9_21_2016-fromXML.dat.targz       ETAS_DROneDayPPEMd2_9_21_2016-fromXML.dat.targz.meta  JANUSOneDay_9_21_2016-fromXML.xml.meta               JANUSOneDayPPE_9_21_2016.dat                     JANUSOneDayTV_9_21_2016.dat.meta
ETAS_DROneDayMd2_9_21_2016-fromXML.dat.targz.meta  ETAS_DROneDayPPEMd2_9_21_2016.xml                     JANUSOneDayEEPAS1F_9_21_2016.dat                     JANUSOneDayPPE_9_21_2016.dat.meta                JANUSOneDayTV_9_21_2016-fromXML.dat.targz
ETAS_DROneDayMd2_9_21_2016.xml                     ETAS_DROneDayPPEMd2_9_21_2016.xml.meta                JANUSOneDayEEPAS1F_9_21_2016.dat.meta                JANUSOneDayPPE_9_21_2016-fromXML.dat.targz       JANUSOneDayTV_9_21_2016-fromXML.dat.targz.meta
ETAS_DROneDayMd2_9_21_2016.xml.meta                JANUSOneDay_9_21_2016-fromXML.dat.meta                JANUSOneDayEEPAS1F_9_21_2016-fromXML.dat.targz       JANUSOneDayPPE_9_21_2016-fromXML.dat.targz.meta  JANUSOneDayTV_9_21_2016.xml
ETAS_DROneDayPPEMd2_9_21_2016.dat                  JANUSOneDay_9_21_2016-fromXML.dat.targz               JANUSOneDayEEPAS1F_9_21_2016-fromXML.dat.targz.meta  JANUSOneDayPPE_9_21_2016.xml                     JANUSOneDayTV_9_21_2016.xml.meta

Notice all of the forecasts on the operational system are accounted for, and the forecasts are correctly labeled in the database.

Evaluations

Warning, not yet verified. Every evaluation must be related to a forecast, so we show for each forecast group, therefore for each forecast, which evaluations are expected.

sqlite> select ForecastGroups.group_name, group_concat(distinct Forecasts.name), group_concat(distinct Evaluations.name) from Evaluations join Forecasts on Evaluations.forecast_id=Forecasts.forecast_id join ForecastGroups on ForecastGroups.forecastgroup_id=Forecasts.group_id group by ForecastGroups.group_name order by ScheduledEvaluations.date_time;
one-day-models|ETAS|N,L,CL,M,S,LW,RP,RD,RT,RTT,T,W,R
one-day-models-Md2-V12.10|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,ETAS_DROneDayPPEMd2,ETAS_DROneDayMd2,JANUSOneDay,JANUSOneDayEEPAS1F,JANUSOneDayTV,JANUSOneDayPPE,SE2OneDay|N,L,CL,M,S,LW,RP,RT,RTT,T,W
one-day-models-V10.10|STEPJAVA|N,L,CL,M,S,LW,RP,RT,RTT,T,W,R
one-day-models-V12.10|ETAS_HW_K3_AVERAGE_Md3,ETAS_HWMd3,K3Md3,ETAS_DROneDayMd3,ETAS_DROneDayPPEMd3|N,L,CL,M,S,LW,RP,RT,RTT,T,W,R
one-day-models-V12.7|ETASV1.1|N,L,CL,M,S,LW,RP,RT,RTT,T,W,R
one-day-models-V16.10|GSF_ISO,GSF_ANISO|N,L,CL,M,S,R,LW,RP,RT,RTT,T,W
one-day-models-V16.4|OneDayBayesianBMA,OneDayBayesianSeqBMA|N,L,S,CL,TX,WX
one-day-models-V16.7|ETASSYN_DROneDayMd2.95|NSIM,L,CL,M,S,N,TX,WX
one-day-models-V9.1|KJSSOneDayCalifornia|N,L,CL,M,S,LW,RP,RD,RT,RTT,T,W,R

Next we enumerate the number of missing evaluation for each forecast and evaluation type.

sqlite> select Forecasts.name, Evaluations.name, count(Evaluations.rowid) from Evaluations join Forecasts on Evaluations.forecast_id=Forecasts.forecast_id where Evaluations.status="Missing" and not Forecasts.status="Missing" group by Forecasts.name, Evaluations.name;
ETAS|CL|1147
ETAS|L|113
ETAS|LW|2038
ETAS|M|1592
ETAS|N|26
ETAS|R|603
ETAS|RD|3025
ETAS|RP|2038
ETAS|RT|2038
ETAS|RTT|2040
ETAS|S|1592
ETAS|T|1608
ETAS|W|1608
ETASSYN_DROneDayMd2.95|CL|2
ETASSYN_DROneDayMd2.95|L|2
ETASSYN_DROneDayMd2.95|M|2
ETASSYN_DROneDayMd2.95|N|614
ETASSYN_DROneDayMd2.95|NSIM|3
ETASSYN_DROneDayMd2.95|S|644
ETASSYN_DROneDayMd2.95|TX|684
ETASSYN_DROneDayMd2.95|WX|684
ETASV1.1|CL|633
ETASV1.1|L|633
ETASV1.1|LW|853
ETASV1.1|M|963
ETASV1.1|N|631
ETASV1.1|R|603
ETASV1.1|RP|853
ETASV1.1|RT|853
ETASV1.1|RTT|853
ETASV1.1|S|963
ETASV1.1|T|853
ETASV1.1|W|853
ETAS_DROneDayMd2|CL|641
ETAS_DROneDayMd2|L|641
ETAS_DROneDayMd2|LW|796
ETAS_DROneDayMd2|M|968
ETAS_DROneDayMd2|N|640
ETAS_DROneDayMd2|RP|796
ETAS_DROneDayMd2|RT|796
ETAS_DROneDayMd2|RTT|796
ETAS_DROneDayMd2|S|968
ETAS_DROneDayMd2|T|796
ETAS_DROneDayMd2|W|796
ETAS_DROneDayMd3|CL|644
ETAS_DROneDayMd3|L|644
ETAS_DROneDayMd3|LW|770
ETAS_DROneDayMd3|M|973
ETAS_DROneDayMd3|N|640
ETAS_DROneDayMd3|R|602
ETAS_DROneDayMd3|RP|770
ETAS_DROneDayMd3|RT|770
ETAS_DROneDayMd3|RTT|772
ETAS_DROneDayMd3|S|973
ETAS_DROneDayMd3|T|770
ETAS_DROneDayMd3|W|770
ETAS_DROneDayPPEMd2|CL|641
ETAS_DROneDayPPEMd2|L|641
ETAS_DROneDayPPEMd2|LW|796
ETAS_DROneDayPPEMd2|M|968
ETAS_DROneDayPPEMd2|N|640
ETAS_DROneDayPPEMd2|RP|796
ETAS_DROneDayPPEMd2|RT|796
ETAS_DROneDayPPEMd2|RTT|796
ETAS_DROneDayPPEMd2|S|968
ETAS_DROneDayPPEMd2|T|796
ETAS_DROneDayPPEMd2|W|796
ETAS_DROneDayPPEMd3|CL|644
ETAS_DROneDayPPEMd3|L|644
ETAS_DROneDayPPEMd3|LW|770
ETAS_DROneDayPPEMd3|M|973
ETAS_DROneDayPPEMd3|N|640
ETAS_DROneDayPPEMd3|R|602
ETAS_DROneDayPPEMd3|RP|770
ETAS_DROneDayPPEMd3|RT|770
ETAS_DROneDayPPEMd3|RTT|772
ETAS_DROneDayPPEMd3|S|973
ETAS_DROneDayPPEMd3|T|770
ETAS_DROneDayPPEMd3|W|770
ETAS_HWMd2|CL|45
ETAS_HWMd2|L|45
ETAS_HWMd2|LW|200
ETAS_HWMd2|M|372
ETAS_HWMd2|N|44
ETAS_HWMd2|RP|200
ETAS_HWMd2|RT|200
ETAS_HWMd2|RTT|200
ETAS_HWMd2|S|372
ETAS_HWMd2|T|200
ETAS_HWMd2|W|200
ETAS_HWMd3|CL|644
ETAS_HWMd3|L|644
ETAS_HWMd3|LW|770
ETAS_HWMd3|M|973
ETAS_HWMd3|N|640
ETAS_HWMd3|R|602
ETAS_HWMd3|RP|770
ETAS_HWMd3|RT|770
ETAS_HWMd3|RTT|772
ETAS_HWMd3|S|973
ETAS_HWMd3|T|770
ETAS_HWMd3|W|770
ETAS_HW_K3_AVERAGE_Md2|CL|45
ETAS_HW_K3_AVERAGE_Md2|L|45
ETAS_HW_K3_AVERAGE_Md2|LW|200
ETAS_HW_K3_AVERAGE_Md2|M|372
ETAS_HW_K3_AVERAGE_Md2|N|44
ETAS_HW_K3_AVERAGE_Md2|RP|200
ETAS_HW_K3_AVERAGE_Md2|RT|200
ETAS_HW_K3_AVERAGE_Md2|RTT|200
ETAS_HW_K3_AVERAGE_Md2|S|372
ETAS_HW_K3_AVERAGE_Md2|T|200
ETAS_HW_K3_AVERAGE_Md2|W|200
ETAS_HW_K3_AVERAGE_Md3|CL|639
ETAS_HW_K3_AVERAGE_Md3|L|639
ETAS_HW_K3_AVERAGE_Md3|LW|765
ETAS_HW_K3_AVERAGE_Md3|M|968
ETAS_HW_K3_AVERAGE_Md3|N|635
ETAS_HW_K3_AVERAGE_Md3|R|598
ETAS_HW_K3_AVERAGE_Md3|RP|765
ETAS_HW_K3_AVERAGE_Md3|RT|765
ETAS_HW_K3_AVERAGE_Md3|RTT|767
ETAS_HW_K3_AVERAGE_Md3|S|968
ETAS_HW_K3_AVERAGE_Md3|T|765
ETAS_HW_K3_AVERAGE_Md3|W|765
GSF_ANISO|CL|602
GSF_ANISO|L|602
GSF_ANISO|LW|602
GSF_ANISO|M|602
GSF_ANISO|N|602
GSF_ANISO|R|602
GSF_ANISO|RP|602
GSF_ANISO|RT|602
GSF_ANISO|RTT|602
GSF_ANISO|S|602
GSF_ANISO|T|602
GSF_ANISO|W|602
GSF_ISO|CL|602
GSF_ISO|L|602
GSF_ISO|LW|602
GSF_ISO|M|602
GSF_ISO|N|602
GSF_ISO|R|602
GSF_ISO|RP|602
GSF_ISO|RT|602
GSF_ISO|RTT|602
GSF_ISO|S|602
GSF_ISO|T|602
GSF_ISO|W|602
JANUSOneDay|CL|641
JANUSOneDay|L|641
JANUSOneDay|LW|796
JANUSOneDay|M|968
JANUSOneDay|N|640
JANUSOneDay|RP|796
JANUSOneDay|RT|796
JANUSOneDay|RTT|796
JANUSOneDay|S|968
JANUSOneDay|T|796
JANUSOneDay|W|796
JANUSOneDayEEPAS1F|CL|2047
JANUSOneDayEEPAS1F|L|2047
JANUSOneDayEEPAS1F|LW|2047
JANUSOneDayEEPAS1F|M|2047
JANUSOneDayEEPAS1F|N|2047
JANUSOneDayEEPAS1F|RP|2047
JANUSOneDayEEPAS1F|RT|2047
JANUSOneDayEEPAS1F|RTT|2047
JANUSOneDayEEPAS1F|S|2047
JANUSOneDayEEPAS1F|T|2047
JANUSOneDayEEPAS1F|W|2047
JANUSOneDayPPE|CL|2047
JANUSOneDayPPE|L|2047
JANUSOneDayPPE|LW|2047
JANUSOneDayPPE|M|2047
JANUSOneDayPPE|N|2047
JANUSOneDayPPE|RP|2047
JANUSOneDayPPE|RT|2047
JANUSOneDayPPE|RTT|2047
JANUSOneDayPPE|S|2047
JANUSOneDayPPE|T|2047
JANUSOneDayPPE|W|2047
JANUSOneDayTV|CL|2047
JANUSOneDayTV|L|2047
JANUSOneDayTV|LW|2047
JANUSOneDayTV|M|2047
JANUSOneDayTV|N|2047
JANUSOneDayTV|RP|2047
JANUSOneDayTV|RT|2047
JANUSOneDayTV|RTT|2047
JANUSOneDayTV|S|2047
JANUSOneDayTV|T|2047
JANUSOneDayTV|W|2047
K3Md2|CL|45
K3Md2|L|45
K3Md2|LW|200
K3Md2|M|372
K3Md2|N|44
K3Md2|RP|200
K3Md2|RT|200
K3Md2|RTT|200
K3Md2|S|372
K3Md2|T|200
K3Md2|W|200
K3Md3|CL|644
K3Md3|L|644
K3Md3|LW|770
K3Md3|M|973
K3Md3|N|640
K3Md3|R|602
K3Md3|RP|770
K3Md3|RT|770
K3Md3|RTT|772
K3Md3|S|973
K3Md3|T|770
K3Md3|W|770
KJSSOneDayCalifornia|CL|717
KJSSOneDayCalifornia|L|176
KJSSOneDayCalifornia|LW|1608
KJSSOneDayCalifornia|M|1082
KJSSOneDayCalifornia|N|115
KJSSOneDayCalifornia|R|603
KJSSOneDayCalifornia|RD|2536
KJSSOneDayCalifornia|RP|1608
KJSSOneDayCalifornia|RT|1608
KJSSOneDayCalifornia|RTT|1609
KJSSOneDayCalifornia|S|1082
KJSSOneDayCalifornia|T|1608
KJSSOneDayCalifornia|W|1608
OneDayBayesianBMA|CL|1286
OneDayBayesianBMA|L|1286
OneDayBayesianBMA|N|1286
OneDayBayesianBMA|S|1445
OneDayBayesianBMA|TX|1457
OneDayBayesianBMA|WX|1457
OneDayBayesianSeqBMA|CL|1286
OneDayBayesianSeqBMA|L|1286
OneDayBayesianSeqBMA|N|1286
OneDayBayesianSeqBMA|S|1445
OneDayBayesianSeqBMA|TX|1457
OneDayBayesianSeqBMA|WX|1457
SE2OneDay|CL|574
SE2OneDay|L|574
SE2OneDay|LW|729
SE2OneDay|M|901
SE2OneDay|N|573
SE2OneDay|RP|729
SE2OneDay|RT|729
SE2OneDay|RTT|729
SE2OneDay|S|901
SE2OneDay|T|729
SE2OneDay|W|729
STEPJAVA|CL|175
STEPJAVA|L|175
STEPJAVA|LW|1035
STEPJAVA|M|937
STEPJAVA|N|173
STEPJAVA|R|603
STEPJAVA|RP|1035
STEPJAVA|RT|1035
STEPJAVA|RTT|1035
STEPJAVA|S|937
STEPJAVA|T|1035
STEPJAVA|W|1035

We focus on the N-Test evaluations for the ETAS models, and print the days where this evaluation is missing.

sqlite> select Forecasts.name, Evaluations.name, ScheduledEvaluations.date_time from Evaluations join ScheduledEvaluations on Evaluations.schedule_id=ScheduledEvaluations.scheduled_evaluation_id join Forecasts on Evaluations.forecast_id=Forecasts.forecast_id where Evaluations.status="Missing" and Evaluations.name="N" and Forecasts.name="ETAS";
ETAS|N|2015-02-23 00:00:00
ETAS|N|2015-08-18 00:00:00
ETAS|N|2015-11-06 00:00:00
ETAS|N|2016-01-19 00:00:00
ETAS|N|2016-03-30 00:00:00
ETAS|N|2016-03-31 00:00:00
ETAS|N|2016-04-01 00:00:00
ETAS|N|2016-04-02 00:00:00
ETAS|N|2016-04-03 00:00:00
ETAS|N|2016-12-06 00:00:00
ETAS|N|2016-12-29 00:00:00
ETAS|N|2016-12-30 00:00:00
ETAS|N|2016-12-31 00:00:00
ETAS|N|2017-01-01 00:00:00
ETAS|N|2017-01-02 00:00:00
ETAS|N|2017-01-03 00:00:00
ETAS|N|2017-01-21 00:00:00
ETAS|N|2017-05-03 00:00:00
ETAS|N|2017-05-04 00:00:00
ETAS|N|2017-05-06 00:00:00
ETAS|N|2017-05-08 00:00:00
ETAS|N|2017-05-14 00:00:00
ETAS|N|2017-11-18 00:00:00
ETAS|N|2017-11-19 00:00:00
ETAS|N|2017-11-20 00:00:00
ETAS|N|2018-05-26 00:00:00

Determining which evaluations are missing is challenging because the evaluations could be saved in any forecast group where ETAS is listed. For example the ETAS forecast is listed in the following forecast groups meaning that these evaluations could be present in any of these directories. (However, in this particular case the first missing date could not occur in the V16.10 directory due to timing).

one-day-models | /home/csep/operations/SCEC-natural-laboratory/one-day-models/results
one-day-models-V9.1 | /home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1/results
one-day-models-V10.10 | /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results
one-day-models-V12.7 | /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results
one-day-models-V12.10 | /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10/results
one-day-models-V16.10 | /home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10/results

We will manually search these directories and show this model is not present.

[csep@csep-op results]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models/results/2015-02-23
ls: cannot access /home/csep/operations/SCEC-natural-laboratory/one-day-models/results/2015-02-23: No such file or directory

[csep@csep-op results]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2015-02-23
ls: cannot access /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2015-02-23: No such file or directory

[csep@csep-op results]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2015-02-23
ls: cannot access /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2015-02-23: No such file or directory

[csep@csep-op results]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10/results/2015-02-23
ls: cannot access /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10/results/2015-02-23: No such file or directory

[csep@csep-op results]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10/results/2015-02-23
ls: cannot access /home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10/results/2015-02-23: No such file or directory

Next, we verify the first 20 "Completed" simulations for an evaluation where the majority are missing.

sqlite> select Forecasts.name, ScheduledEvaluations.date_time from Evaluations join Forecasts on Forecasts.forecast_id=Evaluations.forecast_id join ScheduledEvaluations on Evaluations.schedule_id=ScheduledEvaluations.scheduled_evaluation_id where Evaluations.status="Complete" and Forecasts.name="ETAS" and Evaluations.name="LW" limit 20;
ETAS|2009-11-29 00:00:00
ETAS|2009-11-30 00:00:00
ETAS|2009-12-01 00:00:00
ETAS|2009-12-02 00:00:00
ETAS|2010-08-23 00:00:00
ETAS|2010-08-25 00:00:00
ETAS|2010-08-26 00:00:00
ETAS|2012-09-25 00:00:00
ETAS|2012-09-26 00:00:00
ETAS|2012-09-27 00:00:00
ETAS|2012-09-28 00:00:00
ETAS|2012-09-29 00:00:00
ETAS|2012-10-18 00:00:00
ETAS|2012-11-04 00:00:00
ETAS|2012-11-15 00:00:00
ETAS|2012-11-22 00:00:00
ETAS|2012-11-24 00:00:00
ETAS|2012-11-26 00:00:00
ETAS|2012-12-13 00:00:00
ETAS|2012-12-18 00:00:00

Here are the results from running find . -name "*LW-Test_ETAS*xml*" -print > ~/wsavran/LW-Test_ETAS-one-day-models-V9.1.txt ; sort ~/wsavran/LW-Test_ETAS-one-day-models-V9.1.txt > ~/wsavran/LW-Test_ETAS-one-day-models-V9.1-sorted.txt

./2009-11-29/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_29_2009-fromXML.xml.1407008088.841302.2
./2009-11-30/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_30_2009-fromXML.xml.1407011267.496842.2
./2009-12-01/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_1_2009-fromXML.xml.1407015914.721951.2
./2009-12-02/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_2_2009-fromXML.xml.1407018997.827789.2
./2010-08-23/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_8_23_2010-fromXML.xml.1407022122.738802.2
./2010-08-25/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_8_25_2010-fromXML.xml.1407025187.764500.2
./2010-08-26/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_8_26_2010-fromXML.xml.1407028336.569310.2
./2012-09-25/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_25_2012-fromXML.xml.1407031969.083919.2
./2012-09-26/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_26_2012-fromXML.xml.1407042083.107250.2
./2012-09-27/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_27_2012-fromXML.xml.1407051331.716230.2
./2012-09-28/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_28_2012-fromXML.xml.1407063925.933241.2
./2012-09-29/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_29_2012-fromXML.xml.1407092175.741659.2
./2012-10-18/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_10_18_2012-fromXML.xml.1407101243.736082.2
./2012-11-04/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_4_2012-fromXML.xml.1407120086.484133.2
./2012-11-15/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_15_2012-fromXML.xml.1407142495.333533.2
./2012-11-22/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_22_2012-fromXML.xml.1407180048.513000.2
./2012-11-24/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_24_2012-fromXML.xml.1407198649.649907.2
./2012-11-26/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_26_2012-fromXML.xml.1407217302.796640.2
./2012-12-13/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_13_2012-fromXML.xml.1407265079.436862.2
./2012-12-18/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_18_2012-fromXML.xml.1407284266.034296.2

Since the evaluations might be written in different directories, here are the results from the find command on the operational system.

For group one-day-models-V10.10

/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-09-25/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_25_2012-fromXML.xml.1407034485.064768.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-09-26/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_26_2012-fromXML.xml.1407044193.731013.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-09-27/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_27_2012-fromXML.xml.1407053484.363481.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-09-28/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_28_2012-fromXML.xml.1407066029.124641.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-09-29/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_29_2012-fromXML.xml.1407094241.989658.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-10-18/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_10_18_2012-fromXML.xml.1407103322.669393.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-11-04/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_4_2012-fromXML.xml.1407122141.539472.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-11-15/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_15_2012-fromXML.xml.1407144499.858849.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-11-22/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_22_2012-fromXML.xml.1407182120.590843.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-11-24/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_24_2012-fromXML.xml.1407200704.854854.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-11-26/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_26_2012-fromXML.xml.1407219350.492590.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-12-13/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_13_2012-fromXML.xml.1407267242.612174.3
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/results/2012-12-18/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_18_2012-fromXML.xml.1407286428.449104.3

For group one-day-models-V12.7

/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-09-25/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_25_2012-fromXML.xml.1407037982.879646.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-09-26/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_26_2012-fromXML.xml.1407046972.990417.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-09-27/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_27_2012-fromXML.xml.1407056315.647794.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-09-28/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_28_2012-fromXML.xml.1407068908.781897.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-09-29/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_9_29_2012-fromXML.xml.1407097022.806162.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-10-18/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_10_18_2012-fromXML.xml.1407106120.147435.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-11-04/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_4_2012-fromXML.xml.1407124959.788090.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-11-15/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_15_2012-fromXML.xml.1407147236.313998.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-11-22/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_22_2012-fromXML.xml.1407184916.610078.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-11-24/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_24_2012-fromXML.xml.1407203478.408629.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-11-26/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_11_26_2012-fromXML.xml.1407222190.625967.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-12-13/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_13_2012-fromXML.xml.1407270131.911237.4
/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/results/2012-12-18/scec.csep.DiagnosticsTest.dTest_LW-Test_ETAS_12_18_2012-fromXML.xml.1407289342.475311.4