The ESD Tool for handling Big Data

1. Brussels 2015-06-15
Climate analysis & Big data
Rasmus E. Benestad
Abdelkader Mezghani, & Kajsa M. Parding
esd for retrieving, processing, dissecting, analysing, and visualisation

2. How do I find useful climate
information
in large volumes of data?

3. Information & Data & Knowledge
● Information
"...in its most restricted technical sense, is a sequence of
symbols that can be interpreted as a message".
(wikipedia)
● Data are measurements, observations, calculations
● Knowledge is expectation about causalities,
dependencies, and why?

4. Norwegian Meteorological Institute
The ultimate objective is to
find the answers.
The data is the source and
the means finding answers.
Analytical tools designed for
data analysis/statistics.
What is the question?

5. Big data - simple algorithms
Fast “distillation” of information

6. Open source tool ‘esd’
http://github.com/metno/esd

7. Norwegian Meteorological Institute
esd: open-source and free

8. Data access
Source of information

9. Temperature
ECA&D:
~1.1 Gb
-temperature
-precipitation

10. Temperature
ECA&D +
GHCN +
MET
Norway
-temperature

11. Precipitation

12. Reanalyses and satellite data

13. Climate model results

14. Norwegian Meteorological Institute
Make use of information hidden
in vast archives of climate
model results and observations

15. Example
Extracting information embedded in global climate models
and observations

16. Climate model results

17. Norwegian Meteorological Institute

18. Norwegian Meteorological Institute

19. Norwegian Meteorological Institute
What is hidden behind the results?

20. Norwegian Meteorological Institute
Building block
Empirical-statistical
downscaling
Dependencies &
connections
Redundancy

21. Norwegian Meteorological Institute
Extension of the results to regions

22. Norwegian Meteorological Institute
High one-in-twenty mean
summer temperature in 2100

23. Infographics
Exposing different aspects

24. Different sides to information

25. Changes in rainfall statistics?

26. Quick look at temperatures

27. Maps
Correlation

28. Analysis
Patterns of behaviour

29. Anomaly with respect to latitude

30. How similar are the reanalyses? Correlation maps

31. How good are the models?

32. Using mathematics to make
sense of the data
Is there a change in the precipitation?

33. Norwegian Meteorological Institute
How much does it rain?

34. Norwegian Meteorological Institute
Global rain gauge data
= huge
volume
35,000 rain
gauges with
daily data
Best way to
mine hidden
information?

35. Norwegian Meteorological Institute
Understand the data
Exponential
distribution?
Vast number
of data points
on top of each
other

36. Norwegian Meteorological Institute
Extract the essence (cleverly)
Principal
component
analysis:
two main
characteristics

37. Norwegian Meteorological Institute
E.g. The wet-day 95-percentile
for 24-hr precipitation
Calculated Observed

38. Norwegian Meteorological Institute
Test results

39. Norwegian Meteorological Institute
Changing rainfall patterns

40. Norwegian Meteorological Institute
Changing rainfall patterns

41. Norwegian Meteorological Institute
Aggregate
Principal Component Analysis (PCA)
Regression analysis
Quick search
Combination of sources
Mapping/gridding
Statistical distributions
Main instruments

42. Norwegian Meteorological Institute
Tricks for efficiency
Structured and standardised metadata
Common information model (CIM)
Data reference syntax (DRS)
classes and ‘S3’ methods
Fast algorithms making use knowns

43. Norwegian Meteorological Institute
• Facilitate intercomparisons
• Sharing of generic methods
• Traceability and replicability
• Promotes community building & discussions
Benefits of common standards &
structures

44. Norwegian Meteorological Institute
Summary
●esd - “easy and simple data” or empirical-
statistical downscaling
●Statistics - information from the data
●Aim to address specific questions

45. Meteorologisk institutt
Thanks for your attention!