Several AWI web-apps use the NRT The O2A components INGEST and DASHBOARD (and its Data Web Service) use a common data format to exchange time-series data: The NRT (Near Real Time) Standard Import Format to exchange scientific data time-series. Each data parameter needs to be defined and described in Sensor.awi.de. This results in a hierarchic sensor by a “parameter URN” (Unified Resource Name), example: vessel:heincke:ferrybox:temperature. Implementing web-apps are:Data Format.
It uses the SENSOR's parameter codes (URNs, Uniform Resource Names), e.g. vessel:mya_ii:dgps:lat
to describe the data columns in the files.
The data is stored in a simple tabular plain-text format which is describe below.
If a sensor cannot provide data NRT Data Format but a binary or plain-text format instead, a driver (converter) will have to be programmed to convert the data either to a intermediate NRT RAW Data format or to this NRT Data Format.
A number of drivers is already available (see subchapter Drivers).
The following O2A components are involved in / use NRT Data Format.
Web-app | Web-app’s task |
Sensor (sensor.awi.de) | Provides parameter code as URN, e. g. |
NRT Ingester (ingest.awihttp://ingestsrv2.dmawi.de/) | Auto-ingests NRT data and exports and stores sensor's raw data, executes drivers and forwards to Data-WS |
Data-WS (dashboard.awi.de/data/) | Data Web Service having all time-series data and mean values, |
Format version 2 has been introduced on 01-03-2019 and replaces format version 1. The following components only implement format version 2:
...
The component Data-WS only uses format version 2 why it is highly recommended to use this as the default format.
Format version 1 is deprecated. It is still supported by NRT Ingester by giving a --inputformat nrt1 attribute to the process which exports data to the NRT database.
for old ingest jobs but should not be implemented in new projects.
datetime vessel:polarstern:tsk1:salinity [psu] vessel:polarstern:tsk1:sbe38:temperature [°C
datetime vessel:mya:dgps:lat [deg] vessel:mya:dgps:lon [deg]
2019-02-28 15:50:00.
000 56.00000 -49.05454000 34.1234 2.443
2019-02-28 15:50:01.
000 56.34300 -49.00122000 34.1345 2.564
2019-02-28 15:50:02.
000 56.452 -49.31232000 34.1456 2.544
Plain-text table having columns for each parameter and rows for each individual time record.
A sample file is attached: nrtformat2-sample_adcp_20190217_235334.nrt
What | How |
---|---|
Character encoding | UTF-8 |
Field / column separator |
...
Tabulator “\t” | |
End of record marker |
...
New line “\n” | |
Header |
...
The header must consist of a column named “datetime” and 1-n columns having a valid parameter URN (from Sensor.awi.de) |
...
|
...
| |
Header - Units | Units may be given in square brackets [], following each parameter URN, separated by a space “ “: |
...
|
...
Units are optional depending on the O2A component and application.
|
...
|
...
|
Quality Flags |
...
Quality flags can be added to each data value. These are numbers from 0 to n defined in |
...
the flagging scheme. |
...
|
...
|
...
|
...
|
Date time format |
...
Expected time zone: UTC Allowed date formats:
|
...
|
...
|
Decimal symbol |
...
Dot |
...
". |
...
" |
Data values |
...
Decimal values, integer numbers or |
...
characters ( |
...
just no tabulators!) |
...
|
...
|
...
For use of Data-WS only: The Data-WS |
...
only accepts numbers as input |
...
for each column except when its unit is "text"! |
Error values |
...
Empty field (simply give a null string, but do not forget to add |
...
tabs as data field separator) |
29.06.2020
[text]
" as the column's unit to make it a text column.Valid until Format version 1 is deprecated since 28-02-2019. It is still supported by NRT Ingester for old ingest jobs but should not be implemented in new projects.
time;vessel:heincke:phins:latitude,[deg];vessel:heincke:tsg:salinity,[psu]
...
2019-02-28 15:50:02;56.452;-49.31232
Plain-text table having columns for each parameter and rows for each individual time record.
...
Error values: Empty field (simply give a null string, but do not forget to add enough semicolons)
Anchor | ||||
---|---|---|---|---|
|
Currently the following drivers are available to convert sensor's raw data to NRT Data Format:
Name | Sensor types | Language |
---|---|---|
adcpToNrt | RDI ADCP instruments (format pd0 data format such as contained in files .enc, .enx, ...) | Python |
ctdToNrt | Sea & Sun Technologies's CTD (format provided by STA acquisition software) | Python |
ferryboxToNrt | 4h-Jena Ferrybox | Python |
sbe38ToNrt | Seabird SBE38 temperature probe (format provided by probe itself via serial line) | Python |
snowbuoyToNrt | Python | |
toa5ToNrt | Python |
Drivers can be programmed and installed in the NRT Ingester environment. Python is currently the prefered language.