CSEP
The SCEC Collaboratory for the Study of Earthquake Predictability (CSEP) is developing a system for rigorous evaluation of Short Term Earthquake Forecasts.
CSEP Evaluation Testing
The plots show the information gain of the model named at the top compared to each of the models named at the side. For example, if we look at the plot headed "TripleS", we see that the information gain per earthquake of TripleS over STEP, based on 265 earthquakes, is negative 2.65, and relative to ETAS is negative 2.3. So TripleS is much less informative than either STEP or ETAS.
If we look at the plot headed "STEP", we see that the information gain per earthquake of STEP over ETAS is about 0.3. The fact that the error bar does not cross the zero line shows that STEP is significantly more informative than ETAS with 95% confidence, based on a T-test. If the average gain of 0.3 is borne out in the long run, it would represent quite a large advantage for STEP over ETAS, corresponding to an average probability gain of exp(0.3), or about 1.35 per earthquake. However, the 95% confidence interval for the information gain has a lower limit of about 0.03, so the experiment needs to be run for much longer to narrow down the uncertainty.
In all comparisons, the * indicates that the W-test also shows a significant differrence between the models.
We have a modified version of the plotting routine, which has the option of plotting "probability gain" rather than "information gain" on the x-axis, so you can produce the results in that form if you prefer it.
