Page History
Introduction
Several AWI web-apps use the NRT The O2A components INGEST and DASHBOARD (and its Data Web Service) use a common data format to exchange time-series data: The NRT (Near Real Time) Standard Import Format to exchange scientific data time-series. Each data parameter needs to be defined and described in Sensor.awi.de. This results in a hierarchic sensor by a “parameter URN” (Unified Resource Name), example: vessel:heincke:ferrybox:temperature. Implementing web-apps are:Data Format.
It uses the SENSOR's parameter codes (URNs, Uniform Resource Names), e.g. vessel:mya_ii:dgps:lat to describe the data columns in the files.
The data is stored in a simple tabular plain-text format which is describe below.
If a sensor cannot provide data NRT Data Format but a binary or plain-text format instead, a driver (converter) will have to be programmed to convert the data to the NRT Data Format.
A small number of drivers is already available (see chapter Drivers).
The following O2A components are involved in / use NRT Data Format.
Web-app | Web-app’s task |
Sensor (sensor.awi.de) | Provides parameter code as URN, e. g. |
Auto-ingests NRT data and exports and stores sensor's raw data, executes drivers and forwards to Data-WS | |
Data-WS (dashboard.awi.de/data) | Data Web Service having all time-series data and mean values, |
Format version 2
Format version 2 has been introduced on 01-03-2019 and replaces format version 1. The following components only implement
The component Data-WS only uses format version 2 :
- Data-WS
why it is highly recommended to use this as the default format.
Format version 1 is deprecated. It is still supported by NRT Ingester by giving a --inputformat nrt1 attribute to the process which exports data to the NRT database.
Format version 2
Example
for old ingest jobs but should not be implemented in new projects.
Example
datetime vessel:polarstern:tsk1:salinity [psu] vessel:polarstern:tsk1:sbe38:temperature [°Cdatetime vessel:mya:dgps:lat [deg] vessel:mya:dgps:lon [deg]2019-02-28 15:50:00.000 56.00000 -49.05454000 34.1234 2.4432019-02-28 15:50:01.000 56.34300 -49.00122000 34.1345 2.5642019-02-28 15:50:02.000 56.452 -49.31232000 34.1456 2.544
Format
Plain-text table having columns for each parameter and rows for each individual time record.
| Field separator |
...
| Tabulator “\t” | |
| End of record marker |
...
| New line “\n” | |
| Header |
...
The header must consist of a column named “datetime” and 1-n columns having a valid parameter URN (from Sensor.awi.de) |
...
|
...
| |
| Header - Units | Units may be given in square brackets [], following each parameter URN, separated by a space “ “: |
...
|
...
Units are optional depending on the O2A component and application.
|
...
|
...
|
| Quality Flags |
...
Quality flags can be added to each data value. These are numbers from 0 to n defined in document |
...
not defined yet. |
...
|
...
|
...
|
| Date time format |
...
Expected time zone: UTC Allowed date formats:
|
...
|
...
|
| Decimal symbol |
...
| Dot |
...
| ". |
...
| " |
| Data values |
...
Decimal values, integer numbers or |
...
characters (without containing tabulators!) |
...
|
...
|
...
For use of Data-WS only: The Data-WS currently only accepts numbers as input. |
| Error values |
...
| Empty field (simply give a null string, but do not forget to add enough tabulators) |
Format version 1
Valid until Format version 1 is deprecated since 28-02-2019. It is still supported by NRT Ingester for old ingest jobs but should not be implemented in new projects.
Example
time;vessel:heincke:phins:latitude,[deg];vessel:heincke:tsg:salinity,[psu]
...
2019-02-28 15:50:02;56.452;-49.31232
Format
Plain-text table having columns for each parameter and rows for each individual time record.
...
Error values: Empty field (simply give a null string, but do not forget to add enough semicolons)
Changes from version 1 to version 2
- Field separator tabulator „\t“ instead of semicolon „;“
- Unit format changed from “parameterURN;[unit1]” to “parameterURN [unit1]”
- The column „datetime“ has to be the first column and has to be “datetime” instead of “time”
- Quality flag support added
Overview
Content Tools