You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Introduction

The O2A components INGEST and DASHBOARD (and its Data Web Service) use a common data format to exchange time-series data: The NRT (Near Real Time) Data Format.
It uses the SENSOR's parameter codes (URNs, Uniform Resource Names), e.g. vessel:mya_ii:dgps:lat to describe the data columns in the files.
The data is stored in a simple tabular plain-text format which is describe below.
If a sensor cannot provide data NRT Data Format but a binary or plain-text format instead, a driver (converter) will have to be programmed to convert the data to the NRT Data Format.
A small number of drivers is already available (see chapter Drivers).

The following O2A components are involved in / use NRT Data Format.

Web-app

Web-app’s task

Sensor (sensor.awi.de)

Provides parameter code as URN, e. g. vessel:mya_ii:dgps:lat

NRT Ingester (ingest.dmawi.de)

Auto-ingests and stores sensor's raw data, executes drivers and forwards to Data-WS

Data-WS (dashboard.awi.de/data)

Data Web Service having all time-series data and mean values,
visualization frontend for time series (DASHBOARD) and REST interfaces for data in/output

Format version 2

Format version 2 has been introduced on 01-03-2019 and replaces format version 1.
The component Data-WS only uses format version 2 why it is highly recommended to use this as the default format.
Format version 1 is deprecated. It is still supported by NRT Ingester for old ingest jobs but should not be implemented in new projects.

Example

datetime    vessel:polarstern:tsk1:salinity [psu]    vessel:polarstern:tsk1:sbe38:temperature [°C]
2019-02-28 15:50:00.000    34.1234    2.443
2019-02-28 15:50:01.000    34.1345    2.564
2019-02-28 15:50:02.000    34.1456    2.544

Format

Plain-text table having columns for each parameter and rows for each individual time record.



Field separatorTabulator “\t”
End of record markerNew line “\n”
Header

The header must consist of a column named “datetime” and 1-n columns having a valid parameter URN (from Sensor.awi.de)
as column header field. The “datetime” column must always be the first column.

datetime    type:dev:parameter1    type:dev:parameter2

Header - Units

Units may be given in square brackets [], following each parameter URN, separated by a space “ “:

datetime    type:dev:parameter1 [°C]    type:dev:parameter2 []

Units are optional depending on the O2A component and application.

  • Data-WS exports the unit in a data request output (or [] if not defined).
  • Data-WS ignores a passed unit when importing data
  • NRT Ingester passes units to the Data-WS. Only if a new (not existing) parameter URN
    is created in the Data-WS the unit is added to that. Existing parameter URNs and their
    existing units are not updated by a change of the unit.
Quality Flags

Quality flags can be added to each data value. These are numbers from 0 to n defined in document not defined yet.
Each data column needs an additional column titled with the same parameter URN followed by “ (quality_flag)”:

datetime   vessel:mya:temp   vessel:mya:temp (quality_flag)
2019-02-28 15:50:00     56.00000    1
2019-02-28 15:50:01     566.0000    2

Date time format

Expected time zone: UTC

Allowed date formats:

yyyy-mm-dd HH:MM:SS.fff
yyyy-mm-dd HH:MM:SS
yyyy-mm-ddTHH:MM:SS.fff
yyyy-mm-ddTHH:MM:SS

Decimal symbolDot "."
Data values

Decimal values, integer numbers or characters (without containing tabulators!)

34.56556    23    334.43E-2    SAMPLE1

For use of Data-WS only: The Data-WS currently only accepts numbers as input.
Given characters or words (such as “inf”, “NaN”, “SAMPLE1”) will result in an import error.

Error valuesEmpty field (simply give a null string, but do not forget to add enough tabulators)


Format version 1

Format version 1 is deprecated since 28-02-2019. It is still supported by NRT Ingester for old ingest jobs but should not be implemented in new projects.

Example

time;vessel:heincke:phins:latitude,[deg];vessel:heincke:tsg:salinity,[psu]

2019-02-28 15:50:00;56.00000;-49.05454

2019-02-28 15:50:01;56.34300;-49.00122

2019-02-28 15:50:02;56.452;-49.31232


Example without full parameter URN:

In this case the NRT Ingester can prepend a URN prefix (set in the configuration, example “vessel:heincke”) to all parameter URNs  before exporting the data to the NRT Database.

time;phins:latitude,[deg];tsg:salinity,[psu]

2019-02-28 15:50:00;56.00000;-49.05454

2019-02-28 15:50:01;56.34300;-49.00122

2019-02-28 15:50:02;56.452;-49.31232


Format

Plain-text table having columns for each parameter and rows for each individual time record.

Field separator:             Semicolon “;”

End of record marker:   New line “\n”

Header:                        The header must consist of a column named “time” and 1-n columns having a valid parameter URN (from Sensor.awi.de) as column header field, or an incomplete parameter URN prepended by a URN prefix before importing it to NRT Database. A unit may be given as well.

time;type:dev:param1,[unit1];type:dev:param2,[unit2]

Date time format:          Expected time zone: UTC

                  yyyy-mm-dd HH:MM:SS

Decimal symbol:           Dot (.)

Data values:                 Decimal values, integer numbers or character (without containing tabulators)

                                    34.56556;23;334.43E-2;SAMPLE1

Error values:                 Empty field (simply give a null string, but do not forget to add enough semicolons)


Changes from version 1 to version 2

  • Field separator tabulator „\t“ instead of semicolon „;“
  • Unit format changed from “parameterURN;[unit1]” to “parameterURN [unit1]”
  • The column „datetime“ has to be the first column and has to be “datetime” instead of “time”
  • Quality flag support added
  • No labels