• Save
Enabling Quality Control of SensorWeb Observations
Upcoming SlideShare
Loading in...5
×
 

Enabling Quality Control of SensorWeb Observations

on

  • 340 views

The rapid development of sensing technologies had led to the creation of large volumes of environmental observation data. Data quality control information informs users how it was gathered, processed, ...

The rapid development of sensing technologies had led to the creation of large volumes of environmental observation data. Data quality control information informs users how it was gathered, processed, examined. Sensor Web is a web-centric framework that involves observations from various providers. It is essential to capture quality control information within the framework to ensure that observation data are of known and documented quality. In this paper, we present a quality control framework covering different environmental observation data, and show how it is implemented in the TERENO data infrastructure. The infrastructure is modeled after the OGC’s Sensor Web Enablement (SWE) standards.

Statistics

Views

Total Views
340
Views on SlideShare
340
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
3

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Enabling Quality Control of SensorWeb Observations Enabling Quality Control of SensorWeb Observations Presentation Transcript

  • Mitglied der Helmholtz-Gemeinschaft Enabling Quality Control of Sensor Web Observations 7th January 2014 | 3rd International Conference on Sensor Networks (SENSORNETS 2014) Anusuriya Devaraju, Ralf Kunkel, Juergen Sorg, Heye Bogena, Harry Vereecken
  • Presentation Outline 1 2 • Introduction • Motivation • Research Questions & Solutions 4 • Summary and Ongoing Work Mitglied der Helmholtz-Gemeinschaft 3 2
  • Presentation Outline 1 2 • Introduction • Motivation • Research Questions & Solutions 4 • Summary and Ongoing Work Mitglied der Helmholtz-Gemeinschaft 3 3
  • 1. Quality Control (QC) “…. started with activities whose purpose is to control the quality of products or services by finding problems and defects..”1 1http://www.iso9001consultant.com.au/QA.html The goal of QC of observation data is to identify problems Mitglied der Helmholtz-Gemeinschaft within the data, fixing or eliminating them, and documenting the details involved. 4
  • 2. Sensor Web Mitglied der Helmholtz-Gemeinschaft Common standards for structuring sensor information and its exchange. 5
  • Mitglied der Helmholtz-Gemeinschaft OGC Sensor Web Enablement (SWE) An overview of the OGC’s Sensor Observation Service (SOS) *Source: http://52north.org 6
  • Mitglied der Helmholtz-Gemeinschaft 3. Terrestrial Environmental Observatories (TERENO) 7
  • Mitglied der Helmholtz-Gemeinschaft The Eifel/Lower Rhine Valley Observatory 8
  • TERENO Data Infrastructure (Juelich) 4. Publication 5. Administration Mitglied der Helmholtz-Gemeinschaft 3. Standardized Access 1. Data Importing & Processing 2. Storage
  • Presentation Outline 1 2 • Introduction • Motivation • Research Questions & Solutions 4 • Summary and Ongoing Work Mitglied der Helmholtz-Gemeinschaft 3 10
  • Observation Data Processed at Each Local Observatory Eifel/Lower Harz/Centr Rhine al Lowland Climate, soil, water Bavarian Alps and Prealps HMGU IMK/IFU 589 stations 980000 obs/d 75 stations 125000 obs/d 179 stations 320000 obs/d 95 stations 848000 obs/d 8 stations 52128 obs/d 7 stations 133000000 obs/d 3 stations 57000000 obs/d 3 stations 57000000 obs/d 1 station 1900000 obs/d 4 stations 76000000 obs/d Weather radar 2 devices 576 rasters/d 1 device 288 rasters/d SoilCan 36 lysimeters 285000 obs/d 30 lysimeters 238000 obs/d EC flux data Mitglied der Helmholtz-Gemeinschaft Northeastern Lowland 1 device 288 rasters/d 12 lysimeters 95000 obs/d 6 lysimeters 47500 obs/d 42 lysimeters 333000 obs/d
  • Mitglied der Helmholtz-Gemeinschaft We are buried in data!! How can we uncover good and bad observation data?! 12
  • Key Aspect of QC Information How are data series quality checked? Which quality tests are applied? Mitglied der Helmholtz-Gemeinschaft What leads to problems within data? Where the quality control is performed? Who checks the data? What are the quality levels of the data? When the quality control procedure is performed? 13
  • Research Goals Mitglied der Helmholtz-Gemeinschaft The goals are to capture QC information of various observation data systematically and make the information accessible via the Sensor Web. 14
  • Presentation Outline 1 2 • Introduction • Motivation • Research Questions & Solutions 4 • Summary and Ongoing Work Mitglied der Helmholtz-Gemeinschaft 3 15
  • Research Questions Q1. How are raw data gathered and processed into qualitycontrolled observation data? Mitglied der Helmholtz-Gemeinschaft Q2. How the key aspects of data quality control can be modeled and be related to existing observational information? How can QC information be made available via the Sensor Web? 16
  • Mitglied der Helmholtz-Gemeinschaft Different Ways of Importing Data 1. Data series are quality controlled externally via proprietary tools and then imported into the data infrastructure 2. Data series are imported automatically from sensors and then quality controlled internally (within the TEODOOR data infrastructure). 17
  • Data Processing Status (Level) Level Descriptions QC Data Editing Availability Raw Data No No Internal* 2a Externally quality controlled data; approval is pending Yes No, flagging only (except human observations) Internal* 2b Internally quality controlled data with automatic QC procedures Yes No, flagging only Internal* 2c Externally quality controlled data with approval Yes No, flagging only Public 2d Mitglied der Helmholtz-Gemeinschaft 1 Internally quality controlled data with combined QC procedures (automatic and human) Yes No, flagging only Public 3 Derived data Yes Allowed Public *on request 18
  • Quality Flags (Qualifiers) Quality Flags GENERIC FLAGS unevaluated ok baddata suspicious gapfilled SPECIFIC FLAGS moderatequality Mitglied der Helmholtz-Gemeinschaft goodquality extrapolated minerror interpolated badqualityquality isolatedspike 19
  • Externally QC Data (from level 2a to 2c) Start Manually-uploaded, externally quality controlled data e.g., eddy-covariance series fail Send an email alert of resubmission of data Data importing pass Perform flags mapping no Mitglied der Helmholtz-Gemeinschaft Processing level: Level 2a (quality controlled data without approval) Set processing level: Level 2c (externally quality controlled data with approval) Update approver information Publish data via TEODOOR Approval yes End 20
  • Internally QC Data (from level 2b to 2d) Start Automatically-uploaded data e.g., air temperature series fail Send an email alert to the responsible scientist / field technician DATA IMPORT Raw data processing pass fail Set processing level: Level 2b Set generic flag: e.g., suspicious Set specific flag: e.g., minerror (value below detection) Automatic quality checks Visual Inspection Mitglied der Helmholtz-Gemeinschaft pass Set processing level: Level 2b Set generic flag: ok Set specific flag: passedautochecks Set processing level : Level 2d (quality controlled data with automated procedures and visual inspections) Update specific flags and evaluator information Publish data via TEODOOR End 21
  • Research Questions Q1. How are raw data gathered and processed into qualitycontrolled observation data? Mitglied der Helmholtz-Gemeinschaft Q2. How the key aspects of data quality control can be modeled and be related to existing observational information? How can QC information be made available via the Sensor Web? 22
  • Observational Data Model (ODM) sites PK objectid U2,U1 code definition elevation_m foi geom latitude localx localy longitude name posaccuracy_m remarks latlondatumid localprojectiondatumid verticaldatumid sources PK U2,U1 qualifiers variables objectid PK objectid PK objectid address administrativearea citation city code country definition email firstname link organization phone surname zipcode metadataid U1 code definition U1 U2 abbreviation code definition datatypeid offeringid samplemediumid timeunitid unitid valuetypeid propertyid qualifiergroups PK objectid FK1 FK2 groupid qualifierid processingstati PK PK U1 code definition link manufacturer model type version terenodata objectid FK1 FK7 FK3 I1 FK4 methods objectid U1 Mitglied der Helmholtz-Gemeinschaft PK code definition link organization FK6 I2 FK5 FK2 objectid U1 sensors code definition shortdesc U2 timestampto processingstatusid siteid variableid The existing observational data model has been modified to support quality control descriptions • Qualifiers (quality flags) • Data processing status • Source • Method..etc. objectid datavalue datavalueaccuracy offsetvalue timestampfrom censorcodeid importid methodid offsettypeid qualifierid sampleid sourceid validationsourceid derivedfrom binobject binobjecttypeid usersitevariablepermissions PK objectid U1 U1 FK1,U1 groupsetid siteid sourceid variableid loggervariables PK sensorcomponents PK objectid U1 code definition functionid methodid sensorid sensortypeid FK1,U1 FK2,U1 U1 FK1,U1 FK3 FK4,U1 FK2,U1 U1 logger objectid PK objectid allowedmaxvalue allowedminvalue importfactor loggerfilecolumnname loggerfilecolumnnumber loggerid processingstatusid sampletypeid sensorcomponentid variableid sensorinstanceid U1 code definition technicalwarningdays timestampfrom timestampto datatableclassid filetypeid sourceid timezone siteid notify U1 23
  • QC-Enabled SOS Quality Flags Observation Values Mitglied der Helmholtz-Gemeinschaft Data Processing Status Each value is accompanied with a reference combining quality flag id and data processing 24 status id
  • Mitglied der Helmholtz-Gemeinschaft Sensor Web Client – Quality Flagging An Online Quality Flagging Tool is developed based on the 52N Sensor Web Client 25
  • Mitglied der Helmholtz-Gemeinschaft TEODOOR Front End 26
  • Presentation Outline 1 2 • Introduction • Motivation • Research Questions & Solutions 4 • Summary and Ongoing Work Mitglied der Helmholtz-Gemeinschaft 3 27
  • Summary A common quality control framework for processing and assessing time series from various sensing applications of TERENO infrastructure. The framework consists of: A common QC workflow covering various sensor data • An extensible quality flag classification • Changes applied to existing observational data model • QC-Enabled SOS • Sensor Web Client(s) delivering quality controlled observation data. Mitglied der Helmholtz-Gemeinschaft • 28
  • What’s Next? Extend the observation request of the SOS with QCbased filters 1. Mitglied der Helmholtz-Gemeinschaft 1. Incorporate descriptions about operation and maintenance sensing systems in the Sensor Web 29
  • Thank you. Mitglied der Helmholtz-Gemeinschaft For more information, please visit: http://teodoor.icg.kfa-juelich.de 30