During ingest (spaces.awi.de/x/VSIUEg) several elaborated plausibility and quality checks (autoQC
, see spaces.awi.de/x/22WjEw) are applied to the incoming data. This is done for various purposes: These tests shall ensure that only data with a certain degree of quality enters the database. In our opinion estimates about quality, plausibility, and integrity of the data is significant part of the metadata. Estimates on data quality help the user to evaluate her/his input data as well as possible data products.
Some tests of autoQC
are more or less self-propelling, such as device or timestamp identification, since it is data-inherent information which is used by these tests. But in order to offer data in near-real time, which is checked for quality and plausibility, on a high level some assistance by the sensor editors is required.
sensor.awi.de offers a wide range of properties. In parts, they can be set per whole item as well as per parameter. Thus, if for example a lower and upper threshold like operation range
is available for a whole device, it should be set per item. If the property is valid for a specific parameter exclusively, set the measurement property per parameter.
Available properties
systemName | id | description | uuid |
---|---|---|---|
Gradient_Threshold | 402 | a threshold supplied to the Gradient Test as part of the autoQC routine at AWI | bf962581-627e-48b5-a85d-d03a3aedddc5 |
Manufacturer_Range | 208 | the manufacturer's suggested/proclaimed minimum and maximum value to be measured, below/above accuracy and precision is not guaranteed | 29943ecd-2d9a-4db5-9202-949360ca0fbd |
Measurement_Accuracy | 310 | gives the accuracy of the sensor's measurements in units | c25a833f-d46b-4a4f-a1f9-3b7ce3acd418 |
Measurement_Depth | 20 | The depth of a measurement | 21f470c9-0d58-4822-ae76-9021e6220883 |
Measurement_Drift | 185 | describes the stability of a sensor's measurements in units over time | 84e8d781-cb9c-463c-abd1-35517beb3f52 |
Measurement_Frequency | 19 | is the declared repetition of measurements in units of time | 6dab7c89-e745-4394-9d9a-13f1740b0e14 |
Measurement_Precision | 18 | states the expected precision of the sensor's measurements in units | 412b712a-1ace-47b3-9ab2-51df20d837f0 |
Measurement_Resolution | 178 | The resolution of a measurement | 7ba1a063-b831-4e0d-b766-eeb32d1dcb84 |
Operation_Range | 32 | Operation Range | fc5f18e6-0ed4-429a-b21c-022f29ded578 |
Operation_Temperature | 21 | gives the minimum and maximum tolerable environment temperature of the sensor to be operated | 271d0474-9a2b-4228-90b4-d83f45b58b85 |
Spike_Threshold | 403 | a threshold supplied to the Spike Test as part of the autoQC routine at AWI | a2f788a8-3b56-4067-8755-c1dcdc58471d |
Most of the tests in autoQC
are derived from or inspired by Wong et al. (2021) and its implementation in O2A is described in Silva et al. (2020) and Silva et al. (2019). Some details can be found at spaces.awi.de/x/22WjEw as well.
This feature is implemented to serve in a very generic way. It might be that not all properties (and the resulting tests) apply for your specific parameter or item.
Beyond the above mentioned tests as part of autoQC
during ingest (spaces.awi.de/x/VSIUEg) some more tests are applied:
Adding new properties works literally identical for both per parameter and per item.
p a r a m e t e r | |||
At 'Parameters' under 'Tools' the properties can be displayed → | and if missing, they can be added. | All fields are mandatory in this setup. Units typically refer to those ones the parameter itself is measured. If there is no lower and upper bound, but a steady factor, please enter it in both fields. | |
i t e m | |||
At the tab 'Properties' the properties can be displayed → | and if missing, they can be added. | All fields are mandatory in this setup. Units typically refer to those ones the parameter itself is measured. If there is no lower and upper bound, but a steady factor, please enter it in both fields. |
Silva, Brenner, Roland Koppe, Antonie Haas, Christian Schäfer-Neth, Philipp Fischer, Sebastian Immoor, Peter Gerchow, Bernadette Fritzsch, and Stephan Frickenhaus. 2019. “Automatic Data Quality Control for Understanding Extreme Climate Event.” In EPIC32nd International REKLIM Conference, Berlin, 2019-09-23-2019-09-26Bremerhaven, Germany, REKLIM - Helmholtz Climate Initiative. Bremerhaven, Germany: REKLIM - Helmholtz Climate Initiative. https://www.reklim-conference-2019.de/.
Wong, Annie, Robert Keeley, Thierry Carval, and Argo Data Management Team. 2021. “Argo Quality Control Manual for CTD and Trajectory Data,” May. https://archimer.ifremer.fr/doc/00228/33951/.