Date: Fri, 29 Mar 2024 01:26:27 +0100 (CET) Message-ID: <2127077583.3205.1711671987487@confluencesrv1.awi.de> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_3204_1791314332.1711671987486" ------=_Part_3204_1791314332.1711671987486 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Measuring data can be explored in the database in multiple ways, here: w= e describe the way using the GUI way via https://dashboard.awi= .de/data-xxl/.
Reminder: The O2A near real-time data service provides data as-is. This = means no or only basic quality control procedures are applied. Thus use dat= a with caution.
On the left-hand side you can find several facets, which are an easy to =
use way to restrict your searching terms. For every facet applies: - if the=
re is a label that reads > show more...
there is more to be=
shown. - only available fields can be shown, means if you filter for a typ=
e (e.g. aircraft) and there is no specific parameter (e.g. pressu=
re) available it cannot be displayed in the facet - metadata that should be=
searched and filtered for need to exist!
First facet allows you to filter for a certain Type of = item, e.g. only measured parameters aboard of vessels. Then, only data= streams associated to a vessel (no matter which) are displayed.
Filtering for Parameter types directly corresponds to t= he entries made in sensor.awi.de at the parameter section (https://spaces.awi.de/x/1Ia= 0FQ).
Looking for specific events during a campaign or for a certain subset of=
data the =E2=80=98Actions=E2=80=99 (https://spaces.awi.de/x/0oa0) from sensor.awi.de<=
/a> can be facilitated. On the one hand this can be achieved by the facet <=
strong>Mission. This one incorporates all valid entries of the typ=
e mission from sensor.awi.de. On the other hand, if finer granularity i=
s needed or if you know a certain label title the facet Action should be clicked. These labels can have its origin from manual entrie=
s to sensor.awi.de or they correspond to the Dship https://spaces.awi.de/x/EQDnDw =
Action Log labels, e.g. PS122/1_1-220
.
Under Collection current as well as historical collecti= ons (once) defined in sensor.awi.de appear.
The facet Contact is a direct read from sensor.awi.de= a>. All available contacts with all contact roles are displayed here.
Additionally the field organisation is read per contact. The summary is = displayed in the facet Organisation and can be used to fil= ter according to institutions.
Remark: generally and in advance it might be useful to know what= you are searching for, e.g. by exploring relevant entries at sensor.a= wi.de, then it is much easier to formulate search queries.
The first field of the search bar allows the user to apply a full-text s= earch on metadata. It proved useful to search for item/parameter URNs. This= can be cut short by using wildcards (asterisk *) to meet several condition= s of your query.
Examples:
station:heluwobs:heluw1:adcp_awi_12868:adcp_temp
-> exa=
ctly one parameterstation:heluwobs:heluw1:adcp_awi_12868:*
-> meets all p=
arameters of adcp_awi_12868
station:heluwobs:heluw1:*
-> all items and parameters a=
t Helgoland Underwater Observatorystation:heluwobs:heluw1:adcp_awi_12868:dtb*
-> only the=
parameter current starting with =E2=80=98dtb=E2=80=99 (distance t=
o bottom)station:heluwobs:heluw1:*temperature
-> every parameter=
at Helgoland Underwater Node 1 that ends with temperature (as of today 18 =
parameters)station:heluwobs:heluw1:*temp
every parameter at Helgoland=
Underwater Node 1 that end with temperature (as of today 19 parameters)
Additionally it can be also searched on Action labels or Contact informa= tion or any information in the facets as well. We recommend to use a proper= search term and then restrict the query results by selecting facets.
The age of the incoming data streams can be restricted using the second = field. all literally means everything available in the database wi= ll be shown as long as it meets the restrictions provided by the user. The = age can be set for
By ticking the only QF field only data streams that undergone b= asic quality control (https://spaces.awi.de/x/22WjEw) during ingest will be displayed.= When the sorting is set to Code then a strict alphanumerical orde= r is kept. Selecting Relevance depends strongly on your search ter= m how exact the result meets your expectation.
The query results are displayed according to your favorite sorting (see = above). When ticked the selected data streams stays on top of the selection= , regardless of the pages you move to. The table consists of five columns. = The first one is just the selection square (with no column name).
The second column (Code) represents the full parameter = URN without unit (see https://spaces.awi.de/x/zYa0FQ at =E2=80=98Short names and URN s= yntax=E2=80=99 for more details). On the left-hand side of the parameter UR= N can be clicked to see more details about the paramet= er. By clicking =E2=80=98Detailed description=E2=80=99 you are forwarded to= the respective entry in sensor.awi.de. When there is a PI available fo= r the parameter=E2=80=99s item in sensor.awi.de (https://spaces.awi.de/x/0Ia0FQ) i= t is displayed here. It might be important to note, that this works only wi= th the contact role =E2=80=98PI=E2=80=99, otherwise the fields is blank. Fu= rthermore the explained properties are listed here (https://spaces.awi.de/x/2Ia0FQ), i= f they are set by the item editor.
<= /p>
On the right-hand side of the parameter URN the ico= n copies the full parameter URN to your clipboard. Mostly to the right of t= he copy tool a green icon indicates that the data stream origina= tes from our NRT database. Occasionally a blue denotes that th= e corresponding data stream is part of the AWI datapool, a hadoop cluster c= ontaining non-realtime mass data (more information about that project will = be announced here too). These data streams can be requested, but it takes (= in parts much) longer than the NRT-notated streams.
The third column gives the Age of the data stream. In o= ther words, what is the duration of the last timestamp from the ingested da= ta until now. Negative values, with a purple background color show that the= values are presumably erroneous. All valid values can have minutes, hours,= or days as time duration. Coming with a green background the age is < 1= 0 minutes, with an orange background color the age is < 60 minutes. Ever= y data stream older than one hour appears with red background.
The fourth column shows the last ingested Value plus un= it (for legacy reasons some data streams do not show units, but truly they = have).
The fifth column is reserved for the quality flag creat=
ed by autoQC
(see https://spaces.awi.de/x/2Ia0FQ). If no quality control =
was applied by autoQC
a 0 is printed, if yes, the last flag is=
printed here.
In the first field the user can optionally restricted the time span of d=
ata to be downloaded by the information incorporated in various Act=
ion labels. Technically this works in the same way as in the facet=
search. The only restriction is, that it only applies to actions of the ty=
pe =E2=80=98mission=E2=80=99 and =E2=80=98deployment=E2=80=99. By typing a =
search algorithm starts to query all events in sensor.awi.de. If a suit=
able event is found it can be selected and the start and end date timestamp=
s are filled in the fields From begin date (UTC) and
REMARKS:
The field Aggregation allows you aggregate data to minu= tes, hours or days. Choosing an aggregation type always refers to the (arit= hmetical) average value by default. If this is not the preferred statistica= l base, you can alternatively choose between
These functions are not available for higher resoluted data. That means,= when you choose seconds, milliseconds or microseconds the data will be off= ered to download as is.
REMARK: The data web service is limited to one million = values per call altogether. Hence, two parameters would divide the length o= f a time series in halves, three parameters would make roughly 300 000 rows= of data for each parameter and so on. That means very high resoluted data = needs to be called/downloaded in small chunks if the time span significantl= y overshoots one million values.
Under Quality you can specify if the data export shall = contains quality flags or not. By default this feature is disabled. You can= activate it by ticking . Please keep in mind that
Note on quality flags and aggregation: When data is aggregated (e.g.&nbs=
p;hourly values aggregated to daily values) and quality flags created from =
autoQC
(https://spaces.awi.de/x/22WjEw) are available only data values wi=
th a quality flag <=3D 3 are facilitated for aggregation. Values > 4 =
are omitted. The quality flags itself are aggregated as well. If a single d=
ata value has a quality flag of 0, the aggregated quality flag is set 0. Ot=
herwise the highest quality flag (in the sense of best quality available) w=
ill be used for the respective interval. The following synthetic value tabl=
es might clarify the procedure:
datetime | data | qf | note | |
---|---|---|---|---|
1 | 2021-06-24 01:00:00 | 3 | 2 | in |
2 | 2021-06-24 02:00:00 | 3 | 4 | out |
3 | 2021-06-24 03:00:00 | 4 | 1 | in |
4 | 2021-06-24 04:00:00 | 7 | 1 | in |
5 | 2021-06-24 05:00:00 | 11 | 3 | in |
6 | 2021-06-24 06:00:00 | 7 | 2 | in |
7 | 2021-06-24 07:00:00 | 14 | 3 | in |
8 | 2021-06-24 08:00:00 | 7 | 3 | in |
9 | 2021-06-24 09:00:00 | 6 | 3 | in |
10 | 2021-06-24 10:00:00 | 6 | 4 | out |
11 | 2021-06-24 11:00:00 | 4 | 2 | in |
12 | 2021-06-24 12:00:00 | 4 | 2 | in |
13 | 2021-06-24 13:00:00 | 11 | 1 | in |
14 | 2021-06-24 14:00:00 | 10 | 1 | in |
15 | 2021-06-24 15:00:00 | 13 | 2 | in |
16 | 2021-06-24 16:00:00 | 4 | 4 | out |
17 | 2021-06-24 17:00:00 | 9 | 3 | in |
18 | 2021-06-24 18:00:00 | 13 | 4 | out |
19 | 2021-06-24 19:00:00 | 10 | 1 | in |
20 | 2021-06-24 20:00:00 | 9 | 2 | in |
21 | 2021-06-24 21:00:00 | 11 | 3 | in |
22 | 2021-06-24 22:00:00 | 6 | 1 | in |
23 | 2021-06-24 23:00:00 | 6 | 1 | in |
24 | 2021-06-24 24:00:00 | 5 | 1 | in |
When all quality flags >3 are excluded from the table the following t= able remains to aggregate:
Foundation for aggregation=20datetime | data | qf | note | |
---|---|---|---|---|
1 | 2021-06-24 01:00:00 | 3 | 2 | in |
3 | 2021-06-24 03:00:00 | 4 | 1 | in |
4 | 2021-06-24 04:00:00 | 7 | 1 | in |
5 | 2021-06-24 05:00:00 | 11 | 3 | in |
6 | 2021-06-24 06:00:00 | 7 | 2 | in |
7 | 2021-06-24 07:00:00 | 14 | 3 | in |
8 | 2021-06-24 08:00:00 | 7 | 3 | in |
9 | 2021-06-24 09:00:00 | 6 | 3 | in |
11 | 2021-06-24 11:00:00 | 4 | 2 | in |
12 | 2021-06-24 12:00:00 | 4 | 2 | in |
13 | 2021-06-24 13:00:00 | 11 | 1 | in |
14 | 2021-06-24 14:00:00 | 10 | 1 | in |
15 | 2021-06-24 15:00:00 | 13 | 2 | in |
17 | 2021-06-24 17:00:00 | 9 | 3 | in |
19 | 2021-06-24 19:00:00 | 10 | 1 | in |
20 | 2021-06-24 20:00:00 | 9 | 2 | in |
21 | 2021-06-24 21:00:00 | 11 | 3 | in |
22 | 2021-06-24 22:00:00 | 6 | 1 | in |
23 | 2021-06-24 23:00:00 | 6 | 1 | in |
24 | 2021-06-24 24:00:00 | 5 | 1 | in |
The resulting aggregated single-day value would be:
Aggregation of ex= ample data to daily values using arithmetic mean=20datetime | data | qf |
---|---|---|
2021-06-24 | 7.85 | 1 |
Your selected data streams can have different flavors for downloading. A= s output Format it can be chosen between JSON or = tab-delimited (CSV).
Example Data =E2=80=93 JSON type
{
"beginDate": "2021-06-08T13:00:00.000",
"endDate": "2021-06-08T13:00:30.000",
"qualityFlags": [],
"withQualityFlags": false,
"sensors": [
"station:heluwobs:heluw1:ctd_awi_578:chlorophyll_a_03"
],
"data": [
[
"2021-06-08T13:00:00.000",
20.58
],
[
"2021-06-08T13:00:01.000",
24.77
],
[
"2021-06-08T13:00:02.000",
21.3
],
[
"2021-06-08T13:00:03.000",
33.83
],
[
"2021-06-08T13:00:04.000",
23.88
],
[
"2021-06-08T13:00:05.000",
31.34
]
]
}
Example Data =E2=80=93 CSV type
datetime station:heluwobs:heluw1:ctd_awi_578:c=
hlorophyll_a_03 [=C2=B5g/l]
2021-06-08T13:00:00.000 20.58
2021-06-08T13:00:01.000 24.77
2021-06-08T13:00:02.000 21.3
2021-06-08T13:00:03.000 33.83
2021-06-08T13:00:04.000 23.88
2021-06-08T13:00:05.000 31.34
Finally can be clicked and the request is p=
rocessed. Then a list is generated. The first line is a suggestion on how t=
o cite the data. If the item has a declared PI in sensor.awi.de she/he =
is included in the citation as first author. Otherwise the generic citation=
is used: O2A Data Services (2021): Data from provided by O2A Data Se=
rvices. Alfred-Wegener-Institut, https://dashboard.awi.de/data-xxl
B=
elow the citation three options are shown:
The data set will be =E2=80=93 with re= gard to your choice =E2=80=93 downloaded after clicking.
The download link will be copied to your clipbo= ard so you can paste it elsewhere.
A short summary of the data set can be generated= , annotated with some links to more information about the parameter. This i= s just an excerpt from sensor.awi.de.
// The following metadata is prepared based on se=
nsor.awi.de descriptions.
// Always quote the principal investigator and this data service when using=
data! The licence is CC BY 4.0.
// See O2A documentation: https://spaces.awi.de/display/DM/
station:heluwobs:heluw1:ctd_awi_578:chlorophyll_a_03
- name: Chlorophyll A
- unit: =C2=B5g/l
- type: chlorophyll a
- principal investigator(s):
Fischer, Philipp <philipp.fischer@awi.de>
- resources:
JSON representation [https://sensor.awi.de/rest/sensors/item/getDetailedI=
tem/3833?includeChildren=3Dtrue]
SensorML representation [https://sensor.awi.de/rest/sensors/item/getItemA=
sSensorML/3833]
Web page [https://sensor.awi.de/?id=3D3833]