...
The O2A components INGEST and DASHBOARD (and its Data Web Service) use a common data format to exchange time-series data: The NRT (Near Real Time) Data Format.
It uses the SENSOR's parameter codes (URNs, Uniform Resource Names), e.g. vessel:mya_ii:dgps:lat
to describe the data columns in the files.
The data is stored in a simple tabular plain-text format which is describe below.
If a sensor cannot provide data NRT Data Format but a binary or plain-text format instead, a driver (converter) will have to be programmed to convert the data to the either to a intermediate NRT RAW Data format or to this NRT Data Format.
A number of drivers is already available (see subchapter Drivers).
The following O2A components are involved in / use NRT Data Format.
Web-app | Web-app’s task |
Sensor (sensor.awi.de) | Provides parameter code as URN, e. g. |
NRT Ingester (ingesthttp://ingestsrv2.dmawi.de/) | Auto-ingests and stores sensor's raw data, executes drivers and forwards to Data-WS |
Data-WS (dashboard.awi.de/data/) | Data Web Service having all time-series data and mean values, |
...
Plain-text table having columns for each parameter and rows for each individual time record.
A sample file is attached: nrtformat2-sample_adcp_20190217_235334.nrt
What | How |
---|---|
Character encoding | UTF-8The NRT Ingester can be manually configured to interprete other encodings, too, if necessary. |
Field / column separator | Tabulator “\t” |
End of record marker | New line “\n” |
Header | The header must consist of a column named “datetime” and 1-n columns having a valid parameter URN (from Sensor.awi.de)
|
Header - Units | Units may be given in square brackets [], following each parameter URN, separated by a space “ “:
Units are optional depending on the O2A component and application.
|
Quality Flags | Quality flags can be added to each data value. These are numbers from 0 to n defined in document not defined yetthe flagging scheme.
|
Date time format | Expected time zone: UTC Allowed date formats:
|
Decimal symbol | Dot "." |
Data values | Decimal values, integer numbers or characters (without containing just no tabulators!)
For use of Data-WS only: The Data-WS currently only accepts numbers as input . |
Error values | Empty field (simply give a null string, but do not forget to add tabs as data field separator) |
29.06.2020
[text]
" as the column's unit to make it a text column.Format version 1 is deprecated since 28-02-2019. It is still supported by NRT Ingester for old ingest jobs but should not be implemented in new projects.
...
Currently the following drivers are available to convert sensor's raw data to NRT Data Format:
Name | Sensor types | Language |
---|---|---|
adcpToNrt | RDI ADCP instruments (format pd0 data format such as contained in files .enc, .enx, ...) | Python |
ctdToNrt | Sea & Sun Technologies's CTD (format provided by STA acquisition software) | Python |
ferryboxToNrt | 4h-Jena Ferrybox | Python |
sbe38ToNrt | Seabird SBE38 temperature probe (format provided by probe itself via serial line) | Python |
snowbuoyToNrt | Python | |
toa5ToNrt | Python |
Drivers can be programmed and installed in the NRT Ingester environment. Python is currently the prefered language.