Difference between revisions of "CSEP2 Storing Stochastic Event Sets"
| (9 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
__TOC__ | __TOC__ | ||
| − | = | + | = ZMAP Format = |
This page reflects the work on dealing with stochastic event sets within CSEP2. | This page reflects the work on dealing with stochastic event sets within CSEP2. | ||
| Line 7: | Line 7: | ||
In order to maintain inter-operability between models and the international testing centers we must adopt a standard format for stochastic event sets that contains the necessary and sufficient information needed to perform CSEP evaluations of the forecasts. | In order to maintain inter-operability between models and the international testing centers we must adopt a standard format for stochastic event sets that contains the necessary and sufficient information needed to perform CSEP evaluations of the forecasts. | ||
| − | The most straightforward approach would be to continue with a catalog format supported by CSEP1. The likely candidate for CSEP2 catalog format would be the so-called ZMAP format which has the following format: | + | The most straightforward approach would be to continue with a catalog format supported by CSEP1. The likely candidate for CSEP2 synthetic catalog format would be the so-called ZMAP format which has the following format: |
# Longitude [deg] | # Longitude [deg] | ||
# Latitude [deg] | # Latitude [deg] | ||
| − | # | + | # Year |
# Month | # Month | ||
# Day | # Day | ||
| Line 19: | Line 19: | ||
# Minute | # Minute | ||
# Second | # Second | ||
| + | # Catalog_ID | ||
The fields above can be represented on the computer using a number of different file formats. However, we will focus on a binary representation that aims to reduce the total file size. | The fields above can be represented on the computer using a number of different file formats. However, we will focus on a binary representation that aims to reduce the total file size. | ||
| Line 30: | Line 31: | ||
== UCERF3-ETAS == | == UCERF3-ETAS == | ||
| − | |||
| − | |||
'''Note''': java.io.DataOutputStream stores binary variables in Big Endian notation. | '''Note''': java.io.DataOutputStream stores binary variables in Big Endian notation. | ||
| + | === Binary File Format (Single Catalog) === | ||
| + | UCERF3-ETAS uses a binary format with the following configuration: | ||
# Header [6 bytes] | # Header [6 bytes] | ||
#* File Version: [2 byte short] | #* File Version: [2 byte short] | ||
| Line 52: | Line 53: | ||
#* Grid Node Index: [4 byte Integer] | #* Grid Node Index: [4 byte Integer] | ||
| − | ''' File Size Calculations | + | === Binary File Format (Stochastic Event Set) === |
| + | # Header [4 bytes] | ||
| + | #* Number of Catalogs in SES: [4 byte Integer] | ||
| + | #* ''M'' Binary Single Catalogs: [76 byte Struct] | ||
| + | |||
| + | === File Size Calculations === | ||
We can calculate the total file size of a single catalog using the following formula: | We can calculate the total file size of a single catalog using the following formula: | ||
| Line 61: | Line 67: | ||
We can compute the size of the stochastic event set using: | We can compute the size of the stochastic event set using: | ||
| − | NBYTES_SES: ''M'' * NBYTES_CATALOG | + | NBYTES_SES: 4B + ''M'' * NBYTES_CATALOG |
where ''M'' is the number of synthetic catalogs in the stochastic event set. | where ''M'' is the number of synthetic catalogs in the stochastic event set. | ||
| + | |||
| + | = File Formats for Observed Catalogs = | ||
| + | We would extend the ZMAP format to include metadata fields usable for reproducibility. These might include: | ||
| + | # event_id | ||
| + | # retrieval_time | ||
Latest revision as of 22:27, 24 April 2019
Contents
ZMAP Format
This page reflects the work on dealing with stochastic event sets within CSEP2.
In order to maintain inter-operability between models and the international testing centers we must adopt a standard format for stochastic event sets that contains the necessary and sufficient information needed to perform CSEP evaluations of the forecasts.
The most straightforward approach would be to continue with a catalog format supported by CSEP1. The likely candidate for CSEP2 synthetic catalog format would be the so-called ZMAP format which has the following format:
- Longitude [deg]
- Latitude [deg]
- Year
- Month
- Day
- Magnitude
- Depth [km]
- Hour
- Minute
- Second
- Catalog_ID
The fields above can be represented on the computer using a number of different file formats. However, we will focus on a binary representation that aims to reduce the total file size.
Internally, a stochastic event set will be represented as a collection of pandas DataFrames. Each column in the data frame will represent one of the 10 fields represented in the ZMAP format.
If the machines have sufficient memory, the data frames could be merged into a single large data structure that will support SQL-like queries. Note: Pandas data frames can also interface directly with an SQL database to allow for the possibility of storing simulation results in databases in the future.
File formats for Simulations
This section contains notes on how different forecasts store catalog formats. These notes will be used to implement the file readers.
UCERF3-ETAS
Note: java.io.DataOutputStream stores binary variables in Big Endian notation.
Binary File Format (Single Catalog)
UCERF3-ETAS uses a binary format with the following configuration:
- Header [6 bytes]
- File Version: [2 byte short]
- Catalog Size: [4 byte Integer]
- Rupture Start [70 bytes]
- Rupture Id: [4 byte Integer]
- Parent Id: [4 byte Integer]
- Generation: [2 byte Short]
- Origin Time: [8 byte Long]
- Latitude: [8 byte Double]
- Longitude: [8 byte Double]
- Depth: [8 byte Double]
- Magnitude: [8 byte Double]
- Distance to Parent: [8 byte Double]
- nth ERF Index: [4 byte Integer]
- FSS Index: [4 byte Integer]
- Grid Node Index: [4 byte Integer]
Binary File Format (Stochastic Event Set)
- Header [4 bytes]
- Number of Catalogs in SES: [4 byte Integer]
- M Binary Single Catalogs: [76 byte Struct]
File Size Calculations
We can calculate the total file size of a single catalog using the following formula:
NBYTES_CATALOG: 6B + N * 70B
where N is the number of events in the catalog
We can compute the size of the stochastic event set using:
NBYTES_SES: 4B + M * NBYTES_CATALOG
where M is the number of synthetic catalogs in the stochastic event set.
File Formats for Observed Catalogs
We would extend the ZMAP format to include metadata fields usable for reproducibility. These might include:
- event_id
- retrieval_time
