Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Assessing the quality of volunteered weather observations to provide high-resolution weather maps

8 views

Published on

Slides used during my presentation at the 12th EUMETNET Data Management Workshop, held during 6-8 November 2019 at the Dutch Met Office (KNMI).

Published in: Science
  • Be the first to comment

  • Be the first to like this

Assessing the quality of volunteered weather observations to provide high-resolution weather maps

  1. 1. Irene Garcia-Marti Gerard van der Schrier Jan-Willem Noteboom 7th November 2019 Assessing the quality of volunteered weather observations to provide high-resolution weather maps
  2. 2. › Weather observations are crucial! › Spatial sparsity is a challenge for high-res weather forecasts › Increasing number of weather-related citizen science projects – WOW, Wunderground, Netatmo, Meteoclimatic Motivation 6 november 2019 Koninklijk Nederlands Meteorologisch Instituut 2
  3. 3. 1st September 2019: 1,400 million observations and 17K stations worldwide
  4. 4. › 2015: KNMI partner of WOW › Contributors: 400+ CWS › Data NL+BE: 3.7M obs/month › Devices: semi-professional – Manufacturers: Davis, Oregon scientific, Ventus, Alecto… – Expected “reasonable” quality of the observations WOW-NL 4
  5. 5. › Quality not only related to device: – Good with respect to what? What variables are (not) properly monitored? – Local processes: radiation, shadowing, siting › Classical challenges of citizen science data: – Gaps in data – Noisy observations WOW-NL 5
  6. 6. WOW-NL 6 › Quality not only related to device: – Good with respect to what? What variables are (not) properly monitored? – Local processes: radiation, shadowing, siting › Classical challenges of citizen science data: – Discretization – Scale of the phenomena
  7. 7. 7
  8. 8. What is the quality of WOW-NL data? 1. Quality control 2. Interpolated maps 8
  9. 9. Preprocessing WOW json csv 11.6M observations 65 features SVF Feature engineering
  10. 10. 6 november 2019 Koninklijk Nederlands Meteorologisch Instituut 10
  11. 11. Quality control 11 › Narrowing down variables – Not reinventing the wheel: ▪ Temperature: • (Napoly, 2018) 🡪 • (Meier, 2017) • (Lussana, Titan project) – QC based on (Napoly, 2018) ▪ Feasible to implement on WOW-NL ▪ Levels have been compacted
  12. 12. Each of the 11.6M observations go through this workflow and is labeled with a quality level
  13. 13. Overview of the quality of WOW-NL (Each square represents 10K observations) M0: incorrect metadata (not shown) M1: insufficient Z-score (presence outliers) M2: insufficient day/mon coverage M3: insufficient (Pearson) correlation M4: OK
  14. 14. › Setting up a baseline: – Kriging (or GP) ▪ Ordinary: no {external drift, trend} ▪ Temporal resolution: hourly Interpolation 14 Radiation bias?
  15. 15. 16
  16. 16. › If good enough: – Open the door for new research: ▪ Fine-grained interpolated layers ▪ Nowcasting / hi-res weather ▪ Gridded weather variables ▪ Study local phenomena – Governance level: ▪ Lower cost for administration ▪ Better weather forecast for underrepresented areas ▪ Bottom-up initiatives might work in developing regions Why the quality of citizen science weather data is important? 17
  17. 17. › Met offices have enough knowledge to develop QC’s – Does it make sense? – Collaboration: “the more, the merrier” › Re-use pieces of software: – Corrections: radiation, air pressure – QC’s for other variables: precipitation, wind › Ideally: work together to build a pipeline with mechanistic, corrective, and statistical filters Not reinventing the wheel 18
  18. 18. Challenges and opportunities ahead 19 Big data problem! We are here Imperfect data, but volume is difficult to ignore › Big Data problem – Paradigm shift: code 🡪 data – Adoption of cloud technologies › Data analysis: – Machine learning for QC: Outlier detection, add external drifts – Coupling volunteered data with weather models – Study local patterns: urban wind dynamics, urban rainfall, effect of SVF on outliers – Assess other quality control schemas
  19. 19. Questions? ☺ Thanks! 6 november 2019 Koninklijk Nederlands Meteorologisch Instituut 20

×