This specification extends the commonly known GeoCSV specification with
Its main purpose are single-source layers where data is coming in a harmonized table structure and vocabulary.
This specification comprises three different file types. The two Layer Definition Files play a role when using O2A GeoCSV files inside a standardised O2A SDI SOP dataflow They define the target structure of the database table fueling a WMS/WFS. Data Files and Join Files contain the actual data that should be provided as WMS/WFS. Join Files are an optional feature to reduce data redundancy during data exchange. Data Files can be chunked to organise your data. See the subsections, especially the examples for further understanding.
_layer_def.csv
_layer_def.csvt
<basename>.sdi.csv
(Base Usage)
<basename>@<chunk>.sdi.csv
(Chunked Usage)<basename>.sdi.join.csv
<basename>
and <chunk>
: alphanumeric, no special characters except underscore (_), dash (-), dot (.), hash symbol (#)a-zA-Z0-9
'), underscore, dotTo define the desired (layer) table structure, including colum types, two specific files need to be provided.
An empty CSV file, consisting of exactly one line with exactly all column headers the target layer should contain, separated by tabs.
A CSVT file, consisting of exactly one line with column types for exactly all columns specified in the _layer_def.csv
file, seperated by commas (,
). Supported column types are the following
Column Type | Explanation | Comment |
---|---|---|
Integer(Boolean) | boolean | Values need to be 0 (meaning false) or 1 (meaning true). |
Integer | natural number | -2147483648 to +2147483647 (integer in Postgres) |
Integer64 | -9223372036854775808 to +9223372036854775807 (bigint in Postgres) | |
Real | decimal numeral | 6 decimal digits precision (float in Postgres) |
Real(Float32) | 15 decimal digits precision (double precision in Postgres) | |
String | string/text | Could contain lists/dictionaries formatted as strings (see examples) |
DateTime | date with timestamp | |
WKT | geometry in WKT format |
Not every O2A GeoCSV file needs to contain all columns (except for those mandatory) of the target layer (as described in the Layer Definition Files). All unmentioned columns will hold null
values for all rows of O2A GeoCSV data files not containing them.
column type/group | column header | column data type | column+value mandatory? | description | example values |
---|---|---|---|---|---|
space/time metadata | date_time_start |
| yes | Date and time of data point in ISO 8601 format notation or start of time range. Since version 1.1: time zone specification (e.g. +00 for UTC) is mandatory but could be different from UTC. | valid:
|
date_time_end |
| End of time range of measurement(s) in ISO 8601 format notation, using UTC time zone. | see above | ||
elevation |
| Elevation in meter. A negative value means below sea level, while positive value means above sea level. See Pangaea Geocode definition. Note: This is not the height/depth of the measurement/observation (unless it's taken on earth's surface) but the topographical elevation at the lon/lat position. | valid: 200.1 invalid: | ||
z_value |
| Vertical position of the measurement, in meter (third spatial dimension). | |||
z_type |
| yes, if z_value is given | Pangaea Geocode to describe the type of z_value . | valid: "DEPTH, water" invalid: | |
geometry |
| yes | Geometry in WKT notation without third/vertical spatial dimension. The reference system needs to be EPSG:4326 and the unit decimal degrees. Longitude first, latitude second. The geometry type can be chosen freely. However, a simple POINT is usually the best choice. | valid: invalid: | |
data, other metadata | * | * | Arbitrary amount of data and/or metadata columns. For some metadata columns specific vocabulary needs to be used (see section). | column name examples: | |
join functionality | csv_join | none | Used to hold join keys for the use of join files. Must not appear in layer defintion files! |
|
This is a list of compulsory and encouraged vocabulary to use for column headers (if columns with corresponding meaning are used). Compulsory vocabulary does not mean that the column itself is compulsory. Some vocabulary is encouraged to foster consistency among AWI-hosted OWS. but may be deviated from (e.g. basis
instead of platform
).
metadata category | column vocabulary | vocabulary compulsory? | comment |
---|---|---|---|
space/time metadata | date_time_start | compulsory | |
date_time_end | compulsory | ||
elevation | compulsory | ||
z_value | compulsory | ||
z_type | compulsory | ||
geometry | compulsory | ||
acquisition/parameter metadata | expedition | encouraged (aka "Richtlinie") | see https://wiki.pangaea.de/wiki/Campaign |
event | encouraged | see https://wiki.pangaea.de/wiki/Event | |
platform | encouraged | see https://wiki.pangaea.de/wiki/Basis | |
method | encouraged | Method used for measuring (see https://wiki.pangaea.de/wiki/Method#2._Methods) | |
device | encouraged | Instrument used for measuring (see https://wiki.pangaea.de/wiki/Method#1._Instruments) | |
registry_uri | compulsory | Was "sensor_uri" before. Handles, not URNs! yes: https://hdl.handle.net/10013/sensor.d3360ba2-df00-44c3-a4b5-6af6ee2bb301 no: vessel:polarstern:gyrocompass | |
abstract metadata | citation | compulsory | citation string |
license | compulsory | license name (not license text) | |
data_url | compulsory | URL, pointing to original data source | |
metadata_url | compulsory | URL, pointing to (standard-compliant) metadata | |
sop_url | compulsory | URL, pointing to documented Standard Operating Procedures | |
doi_url | compulsory | URL, pointing to according DOI | |
join functionality | csv_join | compulsory | not exactly data or metadata, more of a helper column to add data from a join file to the data file. |
Basic relations between files are supported. For each data file (*.sdi.csv) a second CSV (*.sdi.join.csv) file can be joined using the column csv_join
to prevent redundancy in data exchange to a certain level. Both files will be joined during database ingest using the following statement. Join Files can hold arbitrary data and metadata columns (but not the geometry
colum). For details, see the example section.
# pseudo SQL select * from <datafile> left join <joinfile> on <datafile>.csv_join = <joinfile>.csv_join;
When multiple basic data files should share one join file, they can be chunked. The data files can have different table structures as long as they adhere to the specifications of basic data files. Join files cannot get chunked.
<basename>@<chunk>.sdi.csv
<basename>.sdi.join.csv
(see "Join Files" section)Beware that copy&paste won't work properly as there might be issues with tabs/spaces in Confluence (this website's software) and/or your text editor.
date_time_start | geometry | temperature |
---|---|---|
2023-06-28T00:00:00+00 | POINT(8.5 53.5) | 28.76 |
2023-06-28T01:00:00+00 | POINT(8.5 53.5) | 27.65 |
2023-06-28T02:00:00+00 | POINT(8.5 53.5) | 26.54 |
2023-06-28T03:00:00+00 | POINT(8.5 53.5) | 25.43 |
date_time_start | geometry | temperature | salinity | platform |
---|---|---|---|---|
2023-06-28T00:00:00+00 | POINT(8.5 53.5) | 28.76 | ||
2023-06-28T01:00:00+00 | POINT(8.5 53.5) | 27.65 | Black Pearl | |
2023-06-28T02:00:00+00 | POINT(8.5 53.5) | 26.54 | Flying Dutchman | |
2023-06-28T03:00:00+00 | POINT(8.5 53.5) | 25.43 |
date_time_start | geometry | people_dict | people_list |
---|---|---|---|
|
| {"Alice": {"age": 18, "favorite_colour": "blue"}, "Bob": {"age": 19, "favorite_colour": "pink"}}
| |
|
|
| ["Alice", "Bob", "Charlie"] |
date_time_start | geometry | name | institute | department | division | group | phone |
---|---|---|---|---|---|---|---|
2015-01-01T01:00:00+00 | POINT(8.5 53.5) | Andreas | AWI | Computing & Data Centre | DATA | Software Engineering | -1744 |
2019-08-01T01:00:00+00 | POINT(8.5 53.5) | Robin | AWI | Computing & Data Centre | DATA | Software Engineering | |
2019-11-01T00:00:00+00 | POINT(8.5 53.5) | Kono | AWI | Computing & Data Centre | DATA | Software Engineering | -2362 |
2020-01-01T00:00:00+00 | POINT(8.5 53.5) | Christopher | AWI | Computing & Data Centre | DATA | Software Engineering | |
2020-01-01T00:00:00+00 | POINT(8.5 53.5) | Max | AWI | Computing & Data Centre | DATA | Data Logistics Support | -2561 |
2021-01-01T00:00:00+00 | POINT(10.12 54.2) | Felix | GEOMAR | ||||
2020-01-01T00:00:00+00 | POINT(8.5 53.5) | Antje | AWI | Board of Directors |
Beispieldaten von wetterkontor.de.
date_time_start | date_time_end | geometry | precipitation |
---|---|---|---|
2022-01-01T00:00:00+00 | 2022-02-01T00:00:00+00 | POINT(8.5 53.5) | 54.9 |
2022-02-01T00:00:00+00 | 2022-03-01T00:00:00+00 | POINT(8.5 53.5) | 129.5 |
2022-03-01T00:00:00+00 | 2022-04-01T00:00:00+00 | POINT(8.5 53.5) | 24.2 |
2022-04-01T00:00:00+00 | 2022-05-01T00:00:00+00 | POINT(8.5 53.5) | 57.7 |
2022-05-01T00:00:00+00 | 2022-06-01T00:00:00+00 | POINT(8.5 53.5) | 76.1 |
2022-06-01T00:00:00+00 | 2022-07-01T00:00:00+00 | POINT(8.5 53.5) | 78.4 |
2022-07-01T00:00:00+00 | 2022-08-01T00:00:00+00 | POINT(8.5 53.5) | 60.1 |
2022-08-01T00:00:00+00 | 2022-09-01T00:00:00+00 | POINT(8.5 53.5) | 21.0 |
2022-09-01T00:00:00+00 | 2022-10-01T00:00:00+00 | POINT(8.5 53.5) | 170.9 |
2022-10-01T00:00:00+00 | 2022-11-01T00:00:00+00 | POINT(8.5 53.5) | 24.6 |
2022-11-01T00:00:00+00 | 2022-12-01T00:00:00+00 | POINT(8.5 53.5) | 47.8 |
2022-12-01T00:00:00+00 | 2023-01-01T00:00:00+00 | POINT(8.5 53.5) | 67.1 |
2023-01-01T00:00:00+00 | 2023-02-01T00:00:00+00 | POINT(8.5 53.5) | 93.0 |
2023-02-01T00:00:00+00 | 2023-03-01T00:00:00+00 | POINT(8.5 53.5) | 41.9 |
2023-03-01T00:00:00+00 | 2023-04-01T00:00:00+00 | POINT(8.5 53.5) | 96.1 |
2023-04-01T00:00:00+00 | 2023-05-01T00:00:00+00 | POINT(8.5 53.5) | 71.0 |
2023-05-01T00:00:00+00 | 2023-06-01T00:00:00+00 | POINT(8.5 53.5) | 14.8 |
2023-06-01T00:00:00+00 | 2023-07-01T00:00:00+00 | POINT(8.5 53.5) | 48.4 |
date_time_start | date_time_end | geometry | state | city | area_km² | population | unemployment_% |
---|---|---|---|---|---|---|---|
2000-01-01T00:00:00+00 | 2001-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 120822 | |
2001-01-01T00:00:00+00 | 2002-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 118701 | |
2002-01-01T00:00:00+00 | 2003-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 119111 | |
2003-01-01T00:00:00+00 | 2004-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 118276 | |
2004-01-01T00:00:00+00 | 2005-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 117281 | |
2005-01-01T00:00:00+00 | 2006-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 23.7 | |
2006-01-01T00:00:00+00 | 2007-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 20.7 | |
2007-01-01T00:00:00+00 | 2008-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 18.5 | |
2008-01-01T00:00:00+00 | 2009-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 16.7 | |
2009-01-01T00:00:00+00 | 2010-01-01T00:00:00+00 | POINT(8.54 53.54) | Bremen | Bremerhaven | 93.8 | 15.4 | |
2000-01-01T00:00:00+00 | 2001-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 278695 | |
2001-01-01T00:00:00+00 | 2002-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 275835 | 15.3 |
2002-01-01T00:00:00+00 | 2003-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 274926 | 16.0 |
2003-01-01T00:00:00+00 | 2004-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 273782 | 17.0 |
2004-01-01T00:00:00+00 | 2005-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 270109 | 18.0 |
2005-01-01T00:00:00+00 | 2006-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 268102 | 23.4 |
2006-01-01T00:00:00+00 | 2007-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 266772 | 20.1 |
2007-01-01T00:00:00+00 | 2008-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 264765 | 16.7 |
2008-01-01T00:00:00+00 | 2009-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 262063 | 15.2 |
2009-01-01T00:00:00+00 | 2010-01-01T00:00:00+00 | POINT(7.1 51.51) | Nordrhein-Westfalen | Gelsenkirchen | 302.9 | 259744 | 15.1 |
2000-01-01T00:00:00+00 | 2001-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 265609 | |
2001-01-01T00:00:00+00 | 2002-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 267197 | 6.7 |
2002-01-01T00:00:00+00 | 2003-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 268945 | 7.3 |
2003-01-01T00:00:00+00 | 2004-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 269579 | 7.8 |
2004-01-01T00:00:00+00 | 2005-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 270038 | 8.3 |
2005-01-01T00:00:00+00 | 2006-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 270868 | 9.1 |
2006-01-01T00:00:00+00 | 2007-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 272106 | 8.4 |
2007-01-01T00:00:00+00 | 2008-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 272951 | 7.1 |
2008-01-01T00:00:00+00 | 2009-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 273875 | 6.4 |
2009-01-01T00:00:00+00 | 2010-01-01T00:00:00+00 | POINT(7.63 51.96) | Nordrhein-Westfalen | Münster | 104.8 | 275543 | 6.4 |
temperature_degc
temperature
+ temperature_unit
28.76 °C
" instead of "28.76
"