Quality Control and beyond
During ingest (spaces.awi.de/x/VSIUEg) several elaborated plausibility and quality checks (autoQC, see spaces.awi.de/x/22WjEw) are applied to the incoming data. This is done for various purposes: These tests shall ensure that each observation is properly flagged when it enters the database. In our opinion estimates about quality, plausibility, and integrity of the data play a significant role in the metadata. Estimates on data quality help the user to evaluate her/his input data as well as possible data products.
Some tests of autoQC are more or less self-propelling, such as device or timestamp identification, since it is data-inherent information which is used by these tests. But in order to offer near-real time quality controlled data some assistance by the sensor's editor is required.
registry.awi.de offers a wide range of properties. In parts, they can be set per whole item as well as per parameter. Thus, if for example a lower and upper threshold like operation range is available for a whole device, it should be set per item. If the property is valid for a specific parameter exclusively, set the measurement property per parameter.
Available properties
| systemName | id | description | uuid |
|---|---|---|---|
| Gradient_Threshold | 402 | a threshold supplied to the Gradient Test as part of the autoQC routine at AWI | bf962581-627e-48b5-a85d-d03a3aedddc5 |
| Manufacturer_Range | 208 | the manufacturer's suggested/proclaimed minimum and maximum value to be measured, below/above accuracy and precision is not guaranteed | 29943ecd-2d9a-4db5-9202-949360ca0fbd |
| Measurement_Accuracy | 310 | gives the accuracy of the sensor's measurements in units | c25a833f-d46b-4a4f-a1f9-3b7ce3acd418 |
| Measurement_Depth | 20 | The depth of a measurement | 21f470c9-0d58-4822-ae76-9021e6220883 |
| Measurement_Drift | 185 | describes the stability of a sensor's measurements in units over time | 84e8d781-cb9c-463c-abd1-35517beb3f52 |
| Measurement_Frequency | 19 | is the declared repetition of measurements in units | 6dab7c89-e745-4394-9d9a-13f1740b0e14 |
| Measurement_Precision | 18 | states the expected precision of the sensor's measurements in units | 412b712a-1ace-47b3-9ab2-51df20d837f0 |
| Measurement_Resolution | 178 | The resolution of a measurement | 7ba1a063-b831-4e0d-b766-eeb32d1dcb84 |
| Operation_Range | 32 | Operation Range | fc5f18e6-0ed4-429a-b21c-022f29ded578 |
| Operation_Temperature | 21 | gives the minimum and maximum tolerable environment temperature of the sensor to be operated | 271d0474-9a2b-4228-90b4-d83f45b58b85 |
| Spike_Threshold | 403 | a threshold supplied to the Spike Test as part of the autoQC routine at AWI | a2f788a8-3b56-4067-8755-c1dcdc58471d |
Most of the tests in autoQC are derived from or inspired by Wong et al. (2021) and its implementation in O2A is described in Silva et al. (2020) and Silva et al. (2019). Some details can be found at spaces.awi.de/x/22WjEw as well.
This feature is implemented to serve in a very generic way. It might be that not all properties (and the resulting tests) apply for your specific parameter or item.
Beyond the above mentioned tests as part of autoQC during ingest (spaces.awi.de/x/VSIUEg) some more tests are applied:
- consecutive date test
- duplicate date test
- empty file test
|
|---|
| Reminder: precision vs. accuracy |
|
|---|
| example of fully set properties of Turbidity Sensor 12912 Seapoint at Svalbard Underwater Node 2 |
Setting Properties
Adding new properties works literally identical for both per parameter and per item.
p a r a m e t e r |
|
|
|
| At 'Parameters' under 'Tools' the properties can be displayed → | and if missing, they can be added. | All fields are mandatory in this setup. Units typically refer to those ones the parameter itself is measured. If there is no lower and upper bound, but a steady factor, please enter it in both fields. | |
i t e m |
|
|
|
| At the tab 'Properties' the properties can be displayed → | and if missing, they can be added. | All fields are mandatory in this setup. Units typically refer to those ones the parameter itself is measured. If there is no lower and upper bound, but a steady factor, please enter it in both fields. |
- Silva, Brenner, Najmeh Kaffashzadeh, Erik Nixdorf, Sebastian Immoor, Philipp Fischer, Norbert Anselm, Peter Gerchow, Angela Schäfer, and Roland Koppe. 2020. “Automatic Quality Control and Quality Control Schema in the Observation to Archive.” EGU2020-15961. Copernicus Meetings. https://doi.org/https://doi.org/10.5194/egusphere-egu2020-15961.
Silva, Brenner, Roland Koppe, Antonie Haas, Christian Schäfer-Neth, Philipp Fischer, Sebastian Immoor, Peter Gerchow, Bernadette Fritzsch, and Stephan Frickenhaus. 2019. “Automatic Data Quality Control for Understanding Extreme Climate Event.” In EPIC32nd International REKLIM Conference, Berlin, 2019-09-23-2019-09-26Bremerhaven, Germany, REKLIM - Helmholtz Climate Initiative. Bremerhaven, Germany: REKLIM - Helmholtz Climate Initiative. https://www.reklim-conference-2019.de/.
Wong, Annie, Robert Keeley, Thierry Carval, and Argo Data Management Team. 2021. “Argo Quality Control Manual for CTD and Trajectory Data,” May. https://archimer.ifremer.fr/doc/00228/33951/.
Overview
Content Tools