Enabling Quality Control of SensorWeb Observations

Mitglied der Helmholtz-Gemeinschaft

Enabling Quality Control of
Sensor Web Observations
7th January 2014 | 3rd International Conference on Sensor Networks (SENSORNETS 2014)
Anusuriya Devaraju, Ralf Kunkel, Juergen Sorg, Heye Bogena, Harry Vereecken

Presentation Outline

1

2

• Introduction

• Motivation
• Research Questions & Solutions

4

• Summary and Ongoing Work


3

2


1

2

• Introduction

• Motivation

4



3

3

1. Quality Control (QC)

“…. started with activities whose purpose is to control the quality
of products or services by finding problems and defects..”1
1http://www.iso9001consultant.com.au/QA.html

The goal of QC of observation data is to identify problems


within the data, fixing or eliminating them, and documenting
the details involved.

4

2. Sensor Web


Common standards
for structuring sensor
information and its
exchange.

5


OGC Sensor Web Enablement (SWE)

An overview of the OGC’s Sensor Observation Service (SOS)

*Source: http://52north.org

6


3. Terrestrial Environmental Observatories
(TERENO)

7


The Eifel/Lower Rhine Valley Observatory

8

TERENO Data Infrastructure (Juelich)
4. Publication

5. Administration


3. Standardized
Access

1. Data Importing & Processing

2. Storage


1

2

• Introduction

• Motivation

4



3

10

Observation Data Processed at Each
Local Observatory
Eifel/Lower Harz/Centr
Rhine
al Lowland

Climate,
soil, water

Bavarian Alps and
Prealps
HMGU
IMK/IFU

589 stations
980000 obs/d

75 stations
125000 obs/d

179 stations
320000 obs/d

95 stations
848000 obs/d

8 stations
52128 obs/d

7 stations
133000000
obs/d

3 stations
57000000
obs/d

3 stations
57000000
obs/d

1 station
1900000
obs/d

4 stations
76000000
obs/d

Weather
radar

2 devices
576 rasters/d

1 device
288 rasters/d

SoilCan

36 lysimeters
285000 obs/d

30 lysimeters
238000 obs/d

EC flux
data


Northeastern
Lowland

1 device
288 rasters/d

12 lysimeters
95000 obs/d

6 lysimeters
47500 obs/d

42 lysimeters
333000 obs/d


We are buried in data!!
How can we uncover good and bad observation data?!
12

Key Aspect of QC Information
How are data
series quality
checked? Which
quality tests are
applied?


What leads to
problems
within data?

Where the quality
control is
performed?

Who checks the
data?

What are the
quality levels
of the data?

When the quality
control procedure
is performed?
13

Research Goals


The goals are to capture QC information of various observation
data systematically and make the information accessible via
the Sensor Web.

14


1

2

• Introduction

• Motivation

4



3

15

Research Questions
Q1. How are raw data gathered and processed into qualitycontrolled observation data?


Q2. How the key aspects of data quality control can be modeled
and be related to existing observational information? How can
QC information be made available via the Sensor Web?

16


Different Ways of Importing Data

1. Data series are quality
controlled externally via
proprietary tools and then imported
into the data infrastructure

2. Data series are imported
automatically from sensors and
then quality controlled internally
(within the TEODOOR data
infrastructure).
17

Data Processing Status (Level)
Level

Descriptions

QC

Data Editing

Availability

Raw Data

No

No

Internal*

2a

Externally quality controlled
data; approval is pending

Yes

No, flagging only
(except human
observations)

Internal*

2b

Internally quality controlled
data with automatic QC
procedures

Yes

No, flagging only

Internal*

2c

Externally quality controlled
data with approval

Yes

No, flagging only

Public

2d

1

Internally quality controlled
data with combined
QC procedures (automatic
and human)

Yes

No, flagging only

Public

3

Derived data

Yes

Allowed

Public
*on request
18

Quality Flags (Qualifiers)
Quality Flags
GENERIC FLAGS
unevaluated

ok

baddata

suspicious

gapfilled

SPECIFIC FLAGS
moderatequality


goodquality

extrapolated

minerror

interpolated
badqualityquality

isolatedspike

19

Externally QC Data (from level 2a to 2c)
Start

Manually-uploaded, externally quality
controlled data
e.g., eddy-covariance series

fail

Send an email alert of
resubmission of data

Data importing

pass
Perform flags mapping
no


Processing level: Level 2a (quality controlled data without approval)

Set processing level: Level 2c (externally quality controlled data with approval)
Update approver information

Publish data via
TEODOOR

Approval

yes

End

20

Internally QC Data (from level 2b to 2d)
Start

Automatically-uploaded data
e.g., air temperature series
fail

Send an email alert to the responsible
scientist / field technician

DATA IMPORT

Raw data processing

pass

fail

Set processing level: Level 2b
Set generic flag: e.g., suspicious
Set specific flag: e.g., minerror (value below detection)

Automatic quality checks

Visual Inspection


pass

Set processing level: Level 2b
Set generic flag: ok
Set specific flag: passedautochecks

Set processing level : Level 2d (quality controlled data with automated procedures and visual inspections)
Update specific flags and evaluator information

Publish data via TEODOOR

End

21

Research Questions
Q1. How are raw data gathered and processed into qualitycontrolled observation data?


Q2. How the key aspects of data quality control can be modeled
and be related to existing observational information? How can
QC information be made available via the Sensor Web?

22

Observational Data Model (ODM)
sites
PK

objectid

U2,U1

code
definition
elevation_m
foi
geom
latitude
localx
localy
longitude
name
posaccuracy_m
remarks
latlondatumid
localprojectiondatumid
verticaldatumid

sources
PK

U2,U1

qualifiers

variables

objectid

PK

objectid

PK

objectid

address
administrativearea
citation
city
code
country
definition
email
firstname
link
organization
phone
surname
zipcode
metadataid

U1

code
definition

U1
U2

abbreviation
code
definition
datatypeid
offeringid
samplemediumid
timeunitid
unitid
valuetypeid
propertyid

qualifiergroups
PK

objectid

FK1
FK2

groupid
qualifierid

processingstati
PK

PK
U1

code
definition
link
manufacturer
model
type
version

terenodata

objectid
FK1
FK7
FK3

I1
FK4
methods
objectid

U1


PK

code
definition
link
organization

FK6
I2
FK5
FK2

objectid

U1

sensors

code
definition
shortdesc

U2

timestampto
processingstatusid
siteid
variableid

The existing observational data model
has been modified to support quality
control descriptions
• Qualifiers (quality flags)
• Data processing status
• Source
• Method..etc.

objectid
datavalue
datavalueaccuracy
offsetvalue
timestampfrom
censorcodeid
importid
methodid
offsettypeid
qualifierid
sampleid
sourceid
validationsourceid
derivedfrom
binobject
binobjecttypeid

usersitevariablepermissions

PK

objectid

U1
U1
FK1,U1

groupsetid
siteid
sourceid
variableid

loggervariables
PK

sensorcomponents
PK

objectid

U1

code
definition
functionid
methodid
sensorid
sensortypeid

FK1,U1
FK2,U1
U1

FK1,U1
FK3
FK4,U1
FK2,U1
U1

logger

objectid

PK

objectid

allowedmaxvalue
allowedminvalue
importfactor
loggerfilecolumnname
loggerfilecolumnnumber
loggerid
processingstatusid
sampletypeid
sensorcomponentid
variableid
sensorinstanceid

U1

code
definition
technicalwarningdays
timestampfrom
timestampto
datatableclassid
filetypeid
sourceid
timezone
siteid
notify

U1

23

QC-Enabled SOS
Quality Flags

Observation Values


Data Processing Status

Each value is accompanied with a reference
combining quality flag id and data processing
24
status id


Sensor Web Client – Quality Flagging

An Online Quality Flagging Tool is developed based on the
52N Sensor Web Client

25


TEODOOR Front End

26


1

2

• Introduction

• Motivation

4



3

27

Summary
A common quality control framework for processing and assessing
time series from various sensing applications of TERENO
infrastructure. The framework consists of:
A common QC workflow covering various sensor data

•

An extensible quality flag classification

•

Changes applied to existing observational data model

•

QC-Enabled SOS

•

Sensor Web Client(s) delivering quality controlled observation
data.


•

28

What’s Next?
Extend the
observation request
of the SOS with QCbased filters

1.


1.

Incorporate
descriptions about
operation and
maintenance
sensing systems in
the Sensor Web

29

Thank you.


For more information, please visit:

http://teodoor.icg.kfa-juelich.de
30

Enabling Quality Control of SensorWeb Observations

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to Enabling Quality Control of SensorWeb Observations

Similar to Enabling Quality Control of SensorWeb Observations (20)

More from Anusuriya Devaraju

More from Anusuriya Devaraju (18)

Recently uploaded

Recently uploaded (20)

Enabling Quality Control of SensorWeb Observations