Introduction

Several AWI web-apps use the NRT (Near Real Time) Standard Import Format to exchange scientific data time-series. Each data parameter needs to be defined and described in Sensor.awi.de. This results in a hierarchic sensor by a “parameter URN” (Unified Resource Name), example: vessel:heincke:ferrybox:temperature. Implementing web-apps are:

Format version 2 has been introduced on 01-03-2019 and replaces format version 1. The following components only implement format version 2:

• Data-WS

Format version 1 is still supported by NRT Ingester by giving a --inputformat nrt1 attribute to the process which exports data to the NRT database.

Format version 2

Example

datetime    vessel:mya:dgps:lat [deg]     vessel:mya:dgps:lon [deg]

2019-02-28 15:50:00.000       56.00000    -49.05454

2019-02-28 15:50:01.000       56.34300    -49.00122

2019-02-28 15:50:02.000       56.452      -49.31232

Format

Plain-text table having columns for each parameter and rows for each individual time record.

Field separator:             Tabulator “\t”

End of record marker:   New line “\n”

Header:                        The header must consist of a column named “datetime” and 1-n columns having a valid parameter URN (from Sensor.awi.de) as column header field. The “datetime” column must always be the first column.

datetime  type:dev:parameter1       type:dev:parameter2

Units may be given in square brackets [], following each parameter URN, separated by a space “ “:

datetime  type:dev:parameter1 [°C]  type:dev:parameter2 []

Units are optional depending on the application.

• Data-WS exports the unit in a data request output (or [] of not defined).
• Data-WS ignores a passed unit when importing data
• NRT Ingester passes units to the Data-WS. Only if it creates a new (not existing) parameter URN in Data-WS the unit is added to that. Existing parameter URNs and their existing units are not updated by a change of the unit.

Quality Flags:               Quality flags can be added to each data value. These are numbers from 0 to n defined in document XYZ. Each data column needs an additional column titled with the same parameter URN followed by “ (quality_flag)”:

                                    Datetime   vessel:mya:temp   vessel:mya:temp (quality_flag)

                                    2019-02-28 15:50:00     56.00000    1

                                    2019-02-28 15:50:01     566.0000    2

Date time format:          Expected time zone: UTC

yyyy-mm-dd HH:MM:SS.fff

                  yyyy-mm-dd HH:MM:SS

yyyy-mm-ddTHH:MM:SS.fff

                  yyyy-mm-ddTHH:MM:SS

Decimal symbol:           Dot “.”

Data values:                 Decimal values, integer numbers or character (without containing tabulators)

                                    34.56556          23        334.43E-2         SAMPLE1

For use of Data-WS only: The Data-WS currently only accepts numbers as input. Given characters or words (such as “inf”, “NaN”, “SAMPLE1”) will result in an import error.

Error values:                 Empty field (simply give a null string, but do not forget to add enough tabulators)

Format version 1

Valid until 28-02-2019.

Example

time;vessel:heincke:phins:latitude,[deg];vessel:heincke:tsg:salinity,[psu]

2019-02-28 15:50:00;56.00000;-49.05454

2019-02-28 15:50:01;56.34300;-49.00122

2019-02-28 15:50:02;56.452;-49.31232

Example without full parameter URN:

In this case the NRT Ingester can prepend a URN prefix (set in the configuration, example “vessel:heincke”) to all parameter URNs  before exporting the data to the NRT Database.

time;phins:latitude,[deg];tsg:salinity,[psu]

2019-02-28 15:50:00;56.00000;-49.05454

2019-02-28 15:50:01;56.34300;-49.00122

2019-02-28 15:50:02;56.452;-49.31232

Format

Plain-text table having columns for each parameter and rows for each individual time record.

Field separator:             Semicolon “;”

End of record marker:   New line “\n”

Header:                        The header must consist of a column named “time” and 1-n columns having a valid parameter URN (from Sensor.awi.de) as column header field, or an incomplete parameter URN prepended by a URN prefix before importing it to NRT Database. A unit may be given as well.

time;type:dev:param1,[unit1];type:dev:param2,[unit2]

Date time format:          Expected time zone: UTC

                  yyyy-mm-dd HH:MM:SS

Decimal symbol:           Dot (.)

Data values:                 Decimal values, integer numbers or character (without containing tabulators)

                                    34.56556;23;334.43E-2;SAMPLE1

Error values:                 Empty field (simply give a null string, but do not forget to add enough semicolons)

Changes from version 1 to version 2

• Field separator tabulator „\t“ instead of semicolon „;“
• Unit format changed from “parameterURN;[unit1]” to “parameterURN [unit1]”
• The column „datetime“ has to be the first column and has to be “datetime” instead of “time”
• Quality flag support added