CSEP1 Status
From SCECpedia
Contents
Introduction
The CSEP1 system has been in production for over a decade with dozens of models running prospectively during this time period. Currently, we do not have an understanding of the status of the operational system with respect to missing forecasts and their evaluations. This page provides the status of the CSEP1 system and outlines our work toward understanding the status of the CSEP1 operational system.
Data Model
CSEP1 simulation meta data represented as SQL schema. Extraction scripts can be found on GitHub at https://github.com/wsavran/csep_db/.
CREATE TABLE IF NOT EXISTS ScheduledForecasts (
scheduled_forecast_id INTEGER PRIMARY KEY,
date_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS ScheduledEvaluations (
scheduled_evaluation_id INTEGER PRIMARY KEY,
date_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS Dispatchers (
dispatcher_id INTEGER PRIMARY KEY,
script_name TEXT NOT NULL,
config_file_name TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS ForecastGroups (
forecastgroup_id INTEGER PRIMARY KEY,
group_name TEXT NOT NULL,
config_filepath TEXT NOT NULL,
dispatcher_id INTEGER NOT NULL,
FOREIGN KEY(dispatcher_id) REFERENCES Dispatchers
);
CREATE TABLE IF NOT EXISTS Forecasts (
forecast_id INTEGER PRIMARY KEY,
schedule_id INTEGER NOT NULL,
group_id INTEGER NOT NULL,
name TEXT NOT NULL,
filepath TEXT,
meta_filepath TEXT,
waiting_period TEXT,
logfile TEXT,
status TEXT,
FOREIGN KEY(schedule_id) REFERENCES ScheduledForecasts,
FOREIGN KEY(group_id) REFERENCES ForecastGroups
);
CREATE TABLE IF NOT EXISTS Evaluations (
evaluation_id INTEGER PRIMARY KEY,
scheduled_id INTEGER NOT NULL,
forecast_id INTEGER NOT NULL,
compute_datetime TEXT,
filepath TEXT,
name TEXT,
status TEXT,
FOREIGN KEY(scheduled_id) REFERENCES ScheduledEvaluations,
FOREIGN KEY(forecast_id) REFERENCES Forecasts
);
Verifying Database
In this section, we manually verify the integrity of the csep_db contents to ensure that the database is accurately representing the information present on the CSEP operational system. We show the verification results for a single dispatcher script.
Dispatcher Scripts
sqlite> select * from Dispatchers; 1|/usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.tcsh|31|/usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.init.xml 2|/usr/local/csep/cronjobs/dispatcher_ANSS1985_M2_95.tcsh|31|/usr/local/csep/cronjobs/dispatcher_ANSS1985_M2_95.init.xml 3|/usr/local/csep/cronjobs/dispatcher_ANSS1932_notFiltered_Md2_one_day.tcsh|31|/usr/local/csep/cronjobs/dispatcher_ANSS1932_notFiltered_Md2_one_day.init.xml
Contents of dispatcher_id=1
cat /usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.tcsh #!/bin/tcsh # dispatcher_ANSS1985.tcsh: # # Run daily Dispatcher for one-day and RELM forecasts models generation and evaluation. # All groups use ANSS catalog with start date of 1985-01-01 and Magnitude>=3.0. # source ~/.tcshrc set year=`date '+%Y'` # Don't pad integer values with zero's set month=`date '+%_m'` set day=`date '+%_d'` # Pad time values with zero's set hour=`date '+%H'` set min=`date '+%M'` set sec=`date '+%S'` # Capture all output produced by Dispatcher into the daily log file set logdir=$CSEP/dispatcher/logs/"$year"_"$month" mkdir -p "$logdir" set logfile="$logdir"/dailyANSS1985_"$year-$month-$day-$hour$min$sec" # Invoke Dispatcher for "today" (with 31 day delay for the testing date): nohup python3 $CENTERCODE/src/generic/Dispatcher.py --year="$year" --month="$month" --day="$day" --configFile=/usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.init.xml --waitingPeriod=31 --enableForecastXMLTemplate --enableForecastCompression --enableForecastMap --publishServer=csep-usc@cseppublishing.usc.edu --publishDirectory=/export/cseppublishing/csep/us/results/data/us/usc/california --logFile="$logfile" >& "$logfile" &
Forecast Groups
Here, we verify that the correct forecast groups were parsed from the Dispatcher configuration file, and the forecast group configuration files are properly represented in the database. Forecast groups that have no scheduled evaluation tests are not included in the database.
1. Group names and filepaths
sqlite> select group_name, group_path from ForecastGroups; one-day-models|/home/csep/operations/SCEC-natural-laboratory/one-day-models one-day-models-V9.1|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1 one-day-models-V10.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10 one-day-models-V12.7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7 one-day-models-V12.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10 one-day-models-V16.4|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.4 one-day-models-V16.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10 one-day-models-V16.7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.7 one-day-models-RELMTests-V16.7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-RELMTests-V16.7 one-day-models-Md2-V12.10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10
Note: Forecast groups with no evaluations scheduled are not included in this report.
[csep@csep-op ~]$ cat /usr/local/csep/cronjobs/dispatcher_ANSS1985_one_day.init.xml
<?xml version="1.0" encoding="utf-8"?>
<Dispatcher xmlns="http://www.scec.org/xml-ns/csep/0.1">
<directory>/home/csep/operations/dispatcher/runs</directory>
<email smtphost="email.usc.edu"
from="johnyu@usc.edu"
subjectPrefix="California one-day models evaluation (ANSS: 1985-01-01)">
fsilva@usc.edu maechlin@usc.edu wsavran@usc.edu
</email>
<!-- One day models -->
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models</forecastGroup>
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1</forecastGroup>
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10</forecastGroup>
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7</forecastGroup>
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10</forecastGroup>
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.4</forecastGroup>
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10</forecastGroup>
<forecastGroup>/home/csep/operations/SCEC-natural-laboratory/one-day-models-optimized</forecastGroup>
</Dispatcher>
Note: the one-day-models-optimized group does not contain any evaluations; hence it's not listed in the database.
2. Config Files
sqlite> select rowid, config_filepath from ForecastGroups; 1|/home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml 2|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1/forecast.init.xml 3|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml 4|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/forecast.init.xml 5|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.10/forecast.init.xml 6|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.4/forecast.init.xml 7|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.10/forecast.init.xml 8|/home/csep/operations/SCEC-natural-laboratory/one-day-models-V16.7/forecast.init.xml 9|/home/csep/operations/SCEC-natural-laboratory/one-day-models-RELMTests-V16.7/forecast.init.xml 10|/home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10/forecast.init.xml
Showing first 3 forecast group configuration files exist on the system.
[csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml /home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml [csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1/forecast.init.xml /home/csep/operations/SCEC-natural-laboratory/one-day-models-V9.1/forecast.init.xml [csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml [csep@csep-op ~]$ ls /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/forecast.init.xml /home/csep/operations/SCEC-natural-laboratory/one-day-models-V12.7/forecast.init.xml
3. Expected Forecasts in each group
select ForecastGroups.group_name, group_concat(distinct Forecasts.name) from ForecastGroups join Forecasts on ForecastGroups.forecastgroup_id=Forecasts.group_id group by ForecastGroups.group_name; 1|one-day-models|ETAS 2|one-day-models-V9.1|KJSSOneDayCalifornia 3|one-day-models-V10.10|STEPJAVA 4|one-day-models-V12.7|ETASV1.1 5|one-day-models-V12.10|ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3 6|one-day-models-V16.4|OneDayBayesianBMA,OneDayBayesianSeqBMA 7|one-day-models-V16.10|GSF_ISO,GSF_ANISO 8|one-day-models-V16.7|ETASSYN_DROneDayMd2.95 10|one-day-models-Md2-V12.10|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,ETAS_DROneDayMd2,ETAS_DROneDayPPEMd2,JANUSOneDay,JANUSOneDayPPE,JANUSOneDayEEPAS1F,JANUSOneDayTV,SE2OneDay
Note: In some cases forecasts are present in multiple forecasts groups, but are archived in the same location. In these cases, the models are associated with the earliest forecast group. Showing results for Groups (1, 3, 7).
cat /home/csep/operations/SCEC-natural-laboratory/one-day-models/forecast.init.xml
<ns0:ForecastGroup xmlns:ns0="http://www.scec.org/xml-ns/csep/0.1" name="One-day models">
<ns0:forecastDir>forecasts</ns0:forecastDir>
<ns0:resultDir>results</ns0:resultDir>
<ns0:catalogDir>observations</ns0:catalogDir>
<ns0:postProcessing>OneDayModel</ns0:postProcessing>
<ns0:entryDate>2007-08-01 00:00:00</ns0:entryDate>
<ns0:models filenameIncludesStartDate="True" files="ETAS_5_5_2018-fromXML.dat.targz">
ETAS
[csep@csep-op ~]$ cat /home/csep/operations/SCEC-natural-laboratory/one-day-models-V10.10/forecast.init.xml
<ns0:ForecastGroup xmlns:ns0="http://www.scec.org/xml-ns/csep/0.1" name="One-day models introduced in CSEP V10.10.0 due to submission of STEP_Java model by Matt Gerstenberger ">
<ns0:forecastDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/forecasts</ns0:forecastDir>
<ns0:resultDir>results</ns0:resultDir>
<ns0:catalogDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/observations</ns0:catalogDir>
<ns0:postProcessing>OneDayModel</ns0:postProcessing>
<ns0:entryDate>2010-10-01 00:00:00</ns0:entryDate>
<ns0:models filenameIncludesStartDate="True" files="ETAS_5_5_2018-fromXML.dat.targz KJSSOneDayCalifornia_5_5_2018-fromXML.dat.targz STEPJAVA_5_5_2018-fromXML.dat.targz">
ETAS KJSSOneDayCalifornia STEPJAVA
cat forecast.init.xml
<ns0:ForecastGroup xmlns:ns0="http://www.scec.org/xml-ns/csep/0.1" name="CSEP V12.10.0 new forecast group due to submission of ETAS_HW, K3 models by Agnes Helmstetter and Max Werner, and ETAS_DR model by David Rhoades">
<ns0:forecastDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/forecasts</ns0:forecastDir>
<ns0:resultDir>results</ns0:resultDir>
<ns0:catalogDir>/home/csep/operations/SCEC-natural-laboratory/one-day-models/observations</ns0:catalogDir>
<ns0:postProcessing>OneDayModel</ns0:postProcessing>
<ns0:entryDate>2012-10-01 00:00:00</ns0:entryDate>
<ns0:models filenameIncludesStartDate="True" files="ETASV1.1_2_5_2017-fromXML.dat.targz ETASV1.1_9_26_2016-fromXML.dat.targz ETAS_9_26_2016-fromXML.dat.targz ETAS_HWMd3_2_5_2017-fromXML.dat.targz ETAS_HWMd3_9_26_2016-fromXML.dat.targz K3Md3_9_26_2016-fromXML.dat.targz KJSSOneDayCalifornia_9_26_2016-fromXML.dat.targz STEPJAVA_9_26_2016-fromXML.dat.targz">
ETAS KJSSOneDayCalifornia STEPJAVA ETASV1.1 ETAS_HW K3 ETAS_DROneDay
<ns0:inputs>||||Md=3|Md=3|Md=3</ns0:inputs>
<ns0:hybridModel name="ETAS_HW_K3_AVERAGE_Md3">
<ns0:component>
<ns0:group weights="[0.5, 0.5]">ETAS_HW K3</ns0:group>
</ns0:component>
</ns0:hybridModel>
4. Start date of each forecast and the associated Forecast Group
sqlite> select ForecastGroups.group_name, Forecasts.name, min(ScheduledForecasts.date_time) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id join ForecastGroups on Forecasts.group_id=ForecastGroups.forecastgroup_id group by Forecasts.name; one-day-models|ETAS|2007-08-01 00:00:00 one-day-models-V9.1|KJSSOneDayCalifornia|2009-01-01 00:00:00 one-day-models-V10.10|STEPJAVA|2010-10-01 00:00:00 one-day-models-V12.7|ETASV1.1|2012-07-01 00:00:00 one-day-models-Md2-V12.10|ETAS_DROneDayMd2|2012-10-01 00:00:00 one-day-models-V12.10|ETAS_DROneDayMd3|2012-10-01 00:00:00 one-day-models-Md2-V12.10|ETAS_DROneDayPPEMd2|2012-10-01 00:00:00 one-day-models-V12.10|ETAS_DROneDayPPEMd3|2012-10-01 00:00:00 one-day-models-Md2-V12.10|ETAS_HWMd2|2012-10-01 00:00:00 one-day-models-V12.10|ETAS_HWMd3|2012-10-01 00:00:00 one-day-models-Md2-V12.10|ETAS_HW_K3_AVERAGE_Md2|2012-10-01 00:00:00 one-day-models-V12.10|ETAS_HW_K3_AVERAGE_Md3|2012-10-01 00:00:00 one-day-models-Md2-V12.10|JANUSOneDay|2012-10-01 00:00:00 one-day-models-Md2-V12.10|JANUSOneDayEEPAS1F|2012-10-01 00:00:00 one-day-models-Md2-V12.10|JANUSOneDayPPE|2012-10-01 00:00:00 one-day-models-Md2-V12.10|JANUSOneDayTV|2012-10-01 00:00:00 one-day-models-Md2-V12.10|K3Md2|2012-10-01 00:00:00 one-day-models-V12.10|K3Md3|2012-10-01 00:00:00 one-day-models-V16.4|OneDayBayesianBMA|2012-10-01 00:00:00 one-day-models-V16.4|OneDayBayesianSeqBMA|2012-10-01 00:00:00 one-day-models-Md2-V12.10|SE2OneDay|2012-10-01 00:00:00 one-day-models-V16.7|ETASSYN_DROneDayMd2.95|2016-07-01 00:00:00 one-day-models-V16.10|GSF_ANISO|2016-10-01 00:00:00 one-day-models-V16.10|GSF_ISO|2016-10-01 00:00:00
Forecasts
1. Missing Forecasts in Each Group
We will manually evaluate a subset of the forecasts and show inductively the data are correct. Add start_date to this query
ETAS|0 KJSSOneDayCalifornia|0 STEPJAVA|0 ETASV1.1|4 ETAS_HWMd3|5 ETAS_HW_K3_AVERAGE_Md3|10 K3Md3|5 ETAS_DROneDayPPEMd3|5 ETAS_DROneDayMd3|5 OneDayBayesianBMA|595 OneDayBayesianSeqBMA|595 GSF_ISO|1 GSF_ANISO|1 ETASSYN_DROneDayMd2.95|11 ETAS_HWMd2|601 ETAS_HW_K3_AVERAGE_Md2|601 K3Md2|601 ETAS_DROneDayMd2|17 ETAS_DROneDayPPEMd2|17 JANUSOneDay|17 JANUSOneDayPPE|17 JANUSOneDayEEPAS1F|17 JANUSOneDayTV|17 SE2OneDay|84
2. Missing Forecasts by Day
We can expand this list to show the missing forecasts grouped by day, which we manually verify for a few different forecasts. We start by selecting the first 10 days containing missing forecasts for all of the one-day-models found on the system.
sqlite> select ScheduledForecasts.date_time, group_concat(Forecasts.name) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id where status='Missing' and ScheduledForecasts.date_time < date('2018-05-10') group by ScheduledForecasts.date_time limit 10;
2016-09-20 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2
2016-09-21 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-22 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-23 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-24 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-25 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-26 00:00:00|ETAS_HW_K3_AVERAGE_Md3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-27 00:00:00|ETASV1.1,ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-28 00:00:00|ETASV1.1,ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-29 00:00:00|ETASV1.1,ETAS_HWMd3,ETAS_HW_K3_AVERAGE_Md3,K3Md3,ETAS_DROneDayPPEMd3,ETAS_DROneDayMd3,OneDayBayesianBMA,OneDayBayesianSeqBMA,ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
We take a subset of this group, namely one-day-models-Md2-V12.10 (id=10).
sqlite> select ScheduledForecasts.date_time, group_concat(Forecasts.name) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id where status='Missing' and ScheduledForecasts.date_time < date('2018-05-10') and Forecasts.group_id=10 group by ScheduledForecasts.date_time limit 10;
2016-09-20 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2
2016-09-21 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-22 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-23 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-24 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-25 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-26 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-27 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-28 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
2016-09-29 00:00:00|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,SE2OneDay
Notice, the SE2OneDay model does not show forecasts past 2016-09-20 in the archive dir /home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10/forecasts/archive/2016_9/.
[csep@csep-op 2016_9]$ pwd /home/csep/operations/SCEC-natural-laboratory/one-day-models-Md2-V12.10/forecasts/archive/2016_9 [csep@csep-op 2016_9]$ ls *SE2OneDay* SE2OneDay_9_10_2016-fromXML.dat.meta SE2OneDay_9_12_2016-fromXML.dat.targz.meta SE2OneDay_9_15_2016-fromXML.xml.meta SE2OneDay_9_19_2016-fromXML.dat.targz SE2OneDay_9_3_2016-fromXML.xml SE2OneDay_9_7_2016-fromXML.dat.meta SE2OneDay_9_10_2016-fromXML.dat.targz SE2OneDay_9_12_2016-fromXML.xml SE2OneDay_9_16_2016-fromXML.dat.meta SE2OneDay_9_19_2016-fromXML.dat.targz.meta SE2OneDay_9_3_2016-fromXML.xml.meta SE2OneDay_9_7_2016-fromXML.dat.targz SE2OneDay_9_10_2016-fromXML.dat.targz.meta SE2OneDay_9_12_2016-fromXML.xml.meta SE2OneDay_9_16_2016-fromXML.dat.targz SE2OneDay_9_19_2016-fromXML.xml SE2OneDay_9_4_2016-fromXML.dat.meta SE2OneDay_9_7_2016-fromXML.dat.targz.meta SE2OneDay_9_10_2016-fromXML.xml SE2OneDay_9_13_2016-fromXML.dat.meta SE2OneDay_9_16_2016-fromXML.dat.targz.meta SE2OneDay_9_19_2016-fromXML.xml.meta SE2OneDay_9_4_2016-fromXML.dat.targz SE2OneDay_9_7_2016-fromXML.xml SE2OneDay_9_10_2016-fromXML.xml.meta SE2OneDay_9_13_2016-fromXML.dat.targz SE2OneDay_9_16_2016-fromXML.xml SE2OneDay_9_20_2016-fromXML.dat.meta SE2OneDay_9_4_2016-fromXML.dat.targz.meta SE2OneDay_9_7_2016-fromXML.xml.meta SE2OneDay_9_11_2016-fromXML.dat.meta SE2OneDay_9_13_2016-fromXML.dat.targz.meta SE2OneDay_9_16_2016-fromXML.xml.meta SE2OneDay_9_20_2016-fromXML.dat.targz SE2OneDay_9_4_2016-fromXML.xml SE2OneDay_9_8_2016-fromXML.dat.meta SE2OneDay_9_11_2016-fromXML.dat.targz SE2OneDay_9_13_2016-fromXML.xml SE2OneDay_9_17_2016-fromXML.dat.meta SE2OneDay_9_20_2016-fromXML.dat.targz.meta SE2OneDay_9_4_2016-fromXML.xml.meta SE2OneDay_9_8_2016-fromXML.dat.targz SE2OneDay_9_11_2016-fromXML.dat.targz.meta SE2OneDay_9_13_2016-fromXML.xml.meta SE2OneDay_9_17_2016-fromXML.dat.targz SE2OneDay_9_20_2016-fromXML.xml SE2OneDay_9_5_2016-fromXML.dat.meta SE2OneDay_9_8_2016-fromXML.dat.targz.meta SE2OneDay_9_11_2016-fromXML.xml SE2OneDay_9_14_2016-fromXML.dat.meta SE2OneDay_9_17_2016-fromXML.dat.targz.meta SE2OneDay_9_20_2016-fromXML.xml.meta SE2OneDay_9_5_2016-fromXML.dat.targz SE2OneDay_9_8_2016-fromXML.xml SE2OneDay_9_11_2016-fromXML.xml.meta SE2OneDay_9_14_2016-fromXML.dat.targz SE2OneDay_9_17_2016-fromXML.xml SE2OneDay_9_2_2016-fromXML.dat.meta SE2OneDay_9_5_2016-fromXML.dat.targz.meta SE2OneDay_9_8_2016-fromXML.xml.meta SE2OneDay_9_1_2016-fromXML.dat.meta SE2OneDay_9_14_2016-fromXML.dat.targz.meta SE2OneDay_9_17_2016-fromXML.xml.meta SE2OneDay_9_2_2016-fromXML.dat.targz SE2OneDay_9_5_2016-fromXML.xml SE2OneDay_9_9_2016-fromXML.dat.meta SE2OneDay_9_1_2016-fromXML.dat.targz SE2OneDay_9_14_2016-fromXML.xml SE2OneDay_9_18_2016-fromXML.dat.meta SE2OneDay_9_2_2016-fromXML.dat.targz.meta SE2OneDay_9_5_2016-fromXML.xml.meta SE2OneDay_9_9_2016-fromXML.dat.targz SE2OneDay_9_1_2016-fromXML.dat.targz.meta SE2OneDay_9_14_2016-fromXML.xml.meta SE2OneDay_9_18_2016-fromXML.dat.targz SE2OneDay_9_2_2016-fromXML.xml SE2OneDay_9_6_2016-fromXML.dat.meta SE2OneDay_9_9_2016-fromXML.dat.targz.meta SE2OneDay_9_1_2016-fromXML.xml SE2OneDay_9_15_2016-fromXML.dat.meta SE2OneDay_9_18_2016-fromXML.dat.targz.meta SE2OneDay_9_2_2016-fromXML.xml.meta SE2OneDay_9_6_2016-fromXML.dat.targz SE2OneDay_9_9_2016-fromXML.xml SE2OneDay_9_1_2016-fromXML.xml.meta SE2OneDay_9_15_2016-fromXML.dat.targz SE2OneDay_9_18_2016-fromXML.xml SE2OneDay_9_3_2016-fromXML.dat.meta SE2OneDay_9_6_2016-fromXML.dat.targz.meta SE2OneDay_9_9_2016-fromXML.xml.meta SE2OneDay_9_12_2016-fromXML.dat.meta SE2OneDay_9_15_2016-fromXML.dat.targz.meta SE2OneDay_9_18_2016-fromXML.xml.meta SE2OneDay_9_3_2016-fromXML.dat.targz SE2OneDay_9_6_2016-fromXML.xml SE2OneDay_9_12_2016-fromXML.dat.targz SE2OneDay_9_15_2016-fromXML.xml SE2OneDay_9_19_2016-fromXML.dat.meta SE2OneDay_9_3_2016-fromXML.dat.targz.meta SE2OneDay_9_6_2016-fromXML.xml.meta
Next, will check the completed forecasts for 2016-09-20 to verify that all forecasts for this day were identified by the system.
sqlite> select ScheduledForecasts.date_time, group_concat(Forecasts.name) from Forecasts join ScheduledForecasts on Forecasts.schedule_id=ScheduledForecasts.scheduled_forecast_id where status='Complete' and ScheduledForecasts.date_time = datetime('2016-09-21 00:00:00') and group_id=10 limit 10;
2016-09-21 00:00:00|ETAS_DROneDayMd2,ETAS_DROneDayPPEMd2,JANUSOneDay,JANUSOneDayEEPAS1F,JANUSOneDayPPE,JANUSOneDayTV
We list the files in the archive directory to verify these files are present on the operational system.
[csep@csep-op 2016_9]$ ls *9_21_2016* ETAS_DROneDayMd2_9_21_2016.dat ETAS_DROneDayPPEMd2_9_21_2016.dat.meta JANUSOneDay_9_21_2016-fromXML.dat.targz.meta JANUSOneDayEEPAS1F_9_21_2016.xml JANUSOneDayPPE_9_21_2016.xml.meta ETAS_DROneDayMd2_9_21_2016.dat.meta ETAS_DROneDayPPEMd2_9_21_2016-fromXML.dat.targz JANUSOneDay_9_21_2016-fromXML.xml JANUSOneDayEEPAS1F_9_21_2016.xml.meta JANUSOneDayTV_9_21_2016.dat ETAS_DROneDayMd2_9_21_2016-fromXML.dat.targz ETAS_DROneDayPPEMd2_9_21_2016-fromXML.dat.targz.meta JANUSOneDay_9_21_2016-fromXML.xml.meta JANUSOneDayPPE_9_21_2016.dat JANUSOneDayTV_9_21_2016.dat.meta ETAS_DROneDayMd2_9_21_2016-fromXML.dat.targz.meta ETAS_DROneDayPPEMd2_9_21_2016.xml JANUSOneDayEEPAS1F_9_21_2016.dat JANUSOneDayPPE_9_21_2016.dat.meta JANUSOneDayTV_9_21_2016-fromXML.dat.targz ETAS_DROneDayMd2_9_21_2016.xml ETAS_DROneDayPPEMd2_9_21_2016.xml.meta JANUSOneDayEEPAS1F_9_21_2016.dat.meta JANUSOneDayPPE_9_21_2016-fromXML.dat.targz JANUSOneDayTV_9_21_2016-fromXML.dat.targz.meta ETAS_DROneDayMd2_9_21_2016.xml.meta JANUSOneDay_9_21_2016-fromXML.dat.meta JANUSOneDayEEPAS1F_9_21_2016-fromXML.dat.targz JANUSOneDayPPE_9_21_2016-fromXML.dat.targz.meta JANUSOneDayTV_9_21_2016.xml ETAS_DROneDayPPEMd2_9_21_2016.dat JANUSOneDay_9_21_2016-fromXML.dat.targz JANUSOneDayEEPAS1F_9_21_2016-fromXML.dat.targz.meta JANUSOneDayPPE_9_21_2016.xml JANUSOneDayTV_9_21_2016.xml.meta
Notice all of the forecasts on the operational system are accounted for, and the forecasts are correctly labeled in the database.
Evaluations
Warning, not yet verified. Every evaluation must be related to a forecast, so we show for each forecast group, therefore for each forecast, which evaluations are expected.
sqlite> select ForecastGroups.group_name, group_concat(distinct Forecasts.name), group_concat(distinct Evaluations.name) from Evaluations join Forecasts on Evaluations.forecast_id=Forecasts.forecast_id join ForecastGroups on ForecastGroups.forecastgroup_id=Forecasts.group_id group by ForecastGroups.group_name order by ScheduledEvaluations.date_time; one-day-models|ETAS|N,L,CL,M,S,LW,RP,RD,RT,RTT,T,W,R one-day-models-Md2-V12.10|ETAS_HWMd2,ETAS_HW_K3_AVERAGE_Md2,K3Md2,ETAS_DROneDayPPEMd2,ETAS_DROneDayMd2,JANUSOneDay,JANUSOneDayEEPAS1F,JANUSOneDayTV,JANUSOneDayPPE,SE2OneDay|N,L,CL,M,S,LW,RP,RT,RTT,T,W one-day-models-V10.10|STEPJAVA|N,L,CL,M,S,LW,RP,RT,RTT,T,W,R one-day-models-V12.10|ETAS_HW_K3_AVERAGE_Md3,ETAS_HWMd3,K3Md3,ETAS_DROneDayMd3,ETAS_DROneDayPPEMd3|N,L,CL,M,S,LW,RP,RT,RTT,T,W,R one-day-models-V12.7|ETASV1.1|N,L,CL,M,S,LW,RP,RT,RTT,T,W,R one-day-models-V16.10|GSF_ISO,GSF_ANISO|N,L,CL,M,S,R,LW,RP,RT,RTT,T,W one-day-models-V16.4|OneDayBayesianBMA,OneDayBayesianSeqBMA|N,L,S,CL,TX,WX one-day-models-V16.7|ETASSYN_DROneDayMd2.95|NSIM,L,CL,M,S,N,TX,WX one-day-models-V9.1|KJSSOneDayCalifornia|N,L,CL,M,S,LW,RP,RD,RT,RTT,T,W,R Next we enumerate the number of missing evaluation for each forecast and evaluation type. <pre> sqlite> select Forecasts.name, Evaluations.name, count(Evaluations.rowid) from Evaluations join Forecasts on Evaluations.forecast_id=Forecasts.forecast_id where Evaluations.status="Missing" and not Forecasts.status="Missing" group by Forecasts.name, Evaluations.name; ETAS|CL|1159 ETAS|L|125 ETAS|LW|2039 ETAS|M|1590 ETAS|N|38 ETAS|R|591 ETAS|RD|3940 ETAS|RP|2039 ETAS|RT|2039 ETAS|RTT|2042 ETAS|S|1591 ETAS|T|1609 ETAS|W|1609 ETASSYN_DROneDayMd2.95|CL|3 ETASSYN_DROneDayMd2.95|L|3 ETASSYN_DROneDayMd2.95|M|3 ETASSYN_DROneDayMd2.95|N|602 ETASSYN_DROneDayMd2.95|NSIM|3 ETASSYN_DROneDayMd2.95|S|633 ETASSYN_DROneDayMd2.95|TX|672 ETASSYN_DROneDayMd2.95|WX|672 ETASV1.1|CL|622 ETASV1.1|L|622 ETASV1.1|LW|842 ETASV1.1|M|951 ETASV1.1|N|620 ETASV1.1|R|591 ETASV1.1|RP|842 ETASV1.1|RT|842 ETASV1.1|RTT|842 ETASV1.1|S|951 ETASV1.1|T|842 ETASV1.1|W|842 ETAS_DROneDayMd2|CL|629 ETAS_DROneDayMd2|L|629 ETAS_DROneDayMd2|LW|784 ETAS_DROneDayMd2|M|956 ETAS_DROneDayMd2|N|628 ETAS_DROneDayMd2|RP|784 ETAS_DROneDayMd2|RT|784 ETAS_DROneDayMd2|RTT|784 ETAS_DROneDayMd2|S|956 ETAS_DROneDayMd2|T|784 ETAS_DROneDayMd2|W|784 ETAS_DROneDayMd3|CL|632 ETAS_DROneDayMd3|L|632 ETAS_DROneDayMd3|LW|758 ETAS_DROneDayMd3|M|961 ETAS_DROneDayMd3|N|628 ETAS_DROneDayMd3|R|590 ETAS_DROneDayMd3|RP|758 ETAS_DROneDayMd3|RT|758 ETAS_DROneDayMd3|RTT|760 ETAS_DROneDayMd3|S|961 ETAS_DROneDayMd3|T|758 ETAS_DROneDayMd3|W|758 ETAS_DROneDayPPEMd2|CL|629 ETAS_DROneDayPPEMd2|L|629 ETAS_DROneDayPPEMd2|LW|784 ETAS_DROneDayPPEMd2|M|956 ETAS_DROneDayPPEMd2|N|628 ETAS_DROneDayPPEMd2|RP|784 ETAS_DROneDayPPEMd2|RT|784 ETAS_DROneDayPPEMd2|RTT|784 ETAS_DROneDayPPEMd2|S|956 ETAS_DROneDayPPEMd2|T|784 ETAS_DROneDayPPEMd2|W|784 ETAS_DROneDayPPEMd3|CL|632 ETAS_DROneDayPPEMd3|L|632 ETAS_DROneDayPPEMd3|LW|758 ETAS_DROneDayPPEMd3|M|961 ETAS_DROneDayPPEMd3|N|628 ETAS_DROneDayPPEMd3|R|590 ETAS_DROneDayPPEMd3|RP|758 ETAS_DROneDayPPEMd3|RT|758 ETAS_DROneDayPPEMd3|RTT|760 ETAS_DROneDayPPEMd3|S|961 ETAS_DROneDayPPEMd3|T|758 ETAS_DROneDayPPEMd3|W|758 ETAS_HWMd2|CL|45 ETAS_HWMd2|L|45 ETAS_HWMd2|LW|200 ETAS_HWMd2|M|372 ETAS_HWMd2|N|44 ETAS_HWMd2|RP|200 ETAS_HWMd2|RT|200 ETAS_HWMd2|RTT|200 ETAS_HWMd2|S|372 ETAS_HWMd2|T|200 ETAS_HWMd2|W|200 ETAS_HWMd3|CL|632 ETAS_HWMd3|L|632 ETAS_HWMd3|LW|758 ETAS_HWMd3|M|961 ETAS_HWMd3|N|628 ETAS_HWMd3|R|590 ETAS_HWMd3|RP|758 ETAS_HWMd3|RT|758 ETAS_HWMd3|RTT|760 ETAS_HWMd3|S|961 ETAS_HWMd3|T|758 ETAS_HWMd3|W|758 ETAS_HW_K3_AVERAGE_Md2|CL|45 ETAS_HW_K3_AVERAGE_Md2|L|45 ETAS_HW_K3_AVERAGE_Md2|LW|200 ETAS_HW_K3_AVERAGE_Md2|M|372 ETAS_HW_K3_AVERAGE_Md2|N|44 ETAS_HW_K3_AVERAGE_Md2|RP|200 ETAS_HW_K3_AVERAGE_Md2|RT|200 ETAS_HW_K3_AVERAGE_Md2|RTT|200 ETAS_HW_K3_AVERAGE_Md2|S|372 ETAS_HW_K3_AVERAGE_Md2|T|200 ETAS_HW_K3_AVERAGE_Md2|W|200 ETAS_HW_K3_AVERAGE_Md3|CL|627 ETAS_HW_K3_AVERAGE_Md3|L|627 ETAS_HW_K3_AVERAGE_Md3|LW|753 ETAS_HW_K3_AVERAGE_Md3|M|956 ETAS_HW_K3_AVERAGE_Md3|N|623 ETAS_HW_K3_AVERAGE_Md3|R|586 ETAS_HW_K3_AVERAGE_Md3|RP|753 ETAS_HW_K3_AVERAGE_Md3|RT|753 ETAS_HW_K3_AVERAGE_Md3|RTT|755 ETAS_HW_K3_AVERAGE_Md3|S|956 ETAS_HW_K3_AVERAGE_Md3|T|753 ETAS_HW_K3_AVERAGE_Md3|W|753 GSF_ANISO|CL|590 GSF_ANISO|L|590 GSF_ANISO|LW|590 GSF_ANISO|M|590 GSF_ANISO|N|590 GSF_ANISO|R|590 GSF_ANISO|RP|590 GSF_ANISO|RT|590 GSF_ANISO|RTT|590 GSF_ANISO|S|590 GSF_ANISO|T|590 GSF_ANISO|W|590 GSF_ISO|CL|590 GSF_ISO|L|590 GSF_ISO|LW|590 GSF_ISO|M|590 GSF_ISO|N|590 GSF_ISO|R|590 GSF_ISO|RP|590 GSF_ISO|RT|590 GSF_ISO|RTT|590 GSF_ISO|S|590 GSF_ISO|T|590 GSF_ISO|W|590 JANUSOneDay|CL|629 JANUSOneDay|L|629 JANUSOneDay|LW|784 JANUSOneDay|M|956 JANUSOneDay|N|628 JANUSOneDay|RP|784 JANUSOneDay|RT|784 JANUSOneDay|RTT|784 JANUSOneDay|S|956 JANUSOneDay|T|784 JANUSOneDay|W|784 JANUSOneDayEEPAS1F|CL|2035 JANUSOneDayEEPAS1F|L|2035 JANUSOneDayEEPAS1F|LW|2035 JANUSOneDayEEPAS1F|M|2035 JANUSOneDayEEPAS1F|N|2035 JANUSOneDayEEPAS1F|RP|2035 JANUSOneDayEEPAS1F|RT|2035 JANUSOneDayEEPAS1F|RTT|2035 JANUSOneDayEEPAS1F|S|2035 JANUSOneDayEEPAS1F|T|2035 JANUSOneDayEEPAS1F|W|2035 JANUSOneDayPPE|CL|2035 JANUSOneDayPPE|L|2035 JANUSOneDayPPE|LW|2035 JANUSOneDayPPE|M|2035 JANUSOneDayPPE|N|2035 JANUSOneDayPPE|RP|2035 JANUSOneDayPPE|RT|2035 JANUSOneDayPPE|RTT|2035 JANUSOneDayPPE|S|2035 JANUSOneDayPPE|T|2035 JANUSOneDayPPE|W|2035 JANUSOneDayTV|CL|2035 JANUSOneDayTV|L|2035 JANUSOneDayTV|LW|2035 JANUSOneDayTV|M|2035 JANUSOneDayTV|N|2035 JANUSOneDayTV|RP|2035 JANUSOneDayTV|RT|2035 JANUSOneDayTV|RTT|2035 JANUSOneDayTV|S|2035 JANUSOneDayTV|T|2035 JANUSOneDayTV|W|2035 K3Md2|CL|45 K3Md2|L|45 K3Md2|LW|200 K3Md2|M|372 K3Md2|N|44 K3Md2|RP|200 K3Md2|RT|200 K3Md2|RTT|200 K3Md2|S|372 K3Md2|T|200 K3Md2|W|200 K3Md3|CL|632 K3Md3|L|632 K3Md3|LW|758 K3Md3|M|961 K3Md3|N|628 K3Md3|R|590 K3Md3|RP|758 K3Md3|RT|758 K3Md3|RTT|760 K3Md3|S|961 K3Md3|T|758 K3Md3|W|758 KJSSOneDayCalifornia|CL|718 KJSSOneDayCalifornia|L|177 KJSSOneDayCalifornia|LW|1609 KJSSOneDayCalifornia|M|1070 KJSSOneDayCalifornia|N|116 KJSSOneDayCalifornia|R|591 KJSSOneDayCalifornia|RD|2530 KJSSOneDayCalifornia|RP|1609 KJSSOneDayCalifornia|RT|1609 KJSSOneDayCalifornia|RTT|1610 KJSSOneDayCalifornia|S|1070 KJSSOneDayCalifornia|T|1609 KJSSOneDayCalifornia|W|1609 OneDayBayesianBMA|CL|1286 OneDayBayesianBMA|L|1286 OneDayBayesianBMA|N|1286 OneDayBayesianBMA|S|1445 OneDayBayesianBMA|TX|1457 OneDayBayesianBMA|WX|1457 OneDayBayesianSeqBMA|CL|1286 OneDayBayesianSeqBMA|L|1286 OneDayBayesianSeqBMA|N|1286 OneDayBayesianSeqBMA|S|1445 OneDayBayesianSeqBMA|TX|1457 OneDayBayesianSeqBMA|WX|1457 SE2OneDay|CL|562 SE2OneDay|L|562 SE2OneDay|LW|717 SE2OneDay|M|889 SE2OneDay|N|561 SE2OneDay|RP|717 SE2OneDay|RT|717 SE2OneDay|RTT|717 SE2OneDay|S|889 SE2OneDay|T|717 SE2OneDay|W|717 STEPJAVA|CL|176 STEPJAVA|L|176 STEPJAVA|LW|1036 STEPJAVA|M|925 STEPJAVA|N|174 STEPJAVA|R|591 STEPJAVA|RP|1036 STEPJAVA|RT|1036 STEPJAVA|RTT|1036 STEPJAVA|S|925 STEPJAVA|T|1036 STEPJAVA|W|1036
We focus on the N-Test evaluations for the ETAS models, and print the days where this evaluation is missing.
sqlite> select Forecasts.name, Evaluations.name, ScheduledEvaluations.date_time from Evaluations join ScheduledEvaluations on Evaluations.schedule_id=ScheduledEvaluations.scheduled_evaluation_id join Forecasts on Evaluations.forecast_id=Forecasts.forecast_id where Evaluations.status="Missing" and Evaluations.name="N" and Forecasts.name="ETAS"; ETAS|N|2010-07-05 00:00:00 ETAS|N|2010-07-17 00:00:00 ETAS|N|2010-08-05 00:00:00 ETAS|N|2010-08-07 00:00:00 ETAS|N|2010-08-18 00:00:00 ETAS|N|2010-08-19 00:00:00 ETAS|N|2010-09-03 00:00:00 ETAS|N|2010-09-09 00:00:00 ETAS|N|2010-09-10 00:00:00 ETAS|N|2010-09-21 00:00:00 ETAS|N|2012-05-28 00:00:00 ETAS|N|2015-02-23 00:00:00 ETAS|N|2015-08-18 00:00:00 ETAS|N|2015-11-06 00:00:00 ETAS|N|2016-01-19 00:00:00 ETAS|N|2016-03-29 00:00:00 ETAS|N|2016-03-30 00:00:00 ETAS|N|2016-03-31 00:00:00 ETAS|N|2016-04-01 00:00:00 ETAS|N|2016-04-02 00:00:00 ETAS|N|2016-04-03 00:00:00 ETAS|N|2016-12-06 00:00:00 ETAS|N|2016-12-29 00:00:00 ETAS|N|2016-12-30 00:00:00 ETAS|N|2016-12-31 00:00:00 ETAS|N|2017-01-01 00:00:00 ETAS|N|2017-01-02 00:00:00 ETAS|N|2017-01-03 00:00:00 ETAS|N|2017-01-21 00:00:00 ETAS|N|2017-05-03 00:00:00 ETAS|N|2017-05-04 00:00:00 ETAS|N|2017-05-06 00:00:00 ETAS|N|2017-05-08 00:00:00 ETAS|N|2017-05-14 00:00:00 ETAS|N|2017-11-18 00:00:00 ETAS|N|2017-11-19 00:00:00 ETAS|N|2017-11-20 00:00:00 ETAS|N|2018-05-14 00:00:00
