BigDataEurope, 1st workshop of the Societal Challenge 5

1. June 18, 2015
Big Data in ECMWF
Ioannis Mallas
Forecast Department

2. role is to address the critical
and most difficult research problems in
medium-range NWP that no one country
could tackle on its own
European cooperation at its best
2EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

3.  Global numerical weather forecasts
 Composition of the atmosphere:
monitoring and forecasting
 Climate re-analysis: monitoring
 Supercomputing & data archiving
 Education programme
European cooperation at its best:
Deliverables and research
3EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

4.  34 member and co-operating states
 270 staff
 30 countries
 Partnerships around the world …
European with a global reach
4EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

5. Mission-driven science
5EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
 What does all this cooperation provide?
 First of all it gets our member and co-operating states global numerical weather forecasts
 How does that work?
 We get observations–now mainly from satellites, which give us the present state, and we use
laws of physics and maths to program on the supercomputer.
 What this cooperation also achieves is economy of scale with a supercomputer that is
owned jointly by all our member states, of which 50% are used for research purposes
here, 25% are used to deliver our operational forecasts, and 25% are used by our
member states. Some examples of what our member states use their share of those 25% for
are that the Met Office currently uses it for their regional climate runs; Meteo France runs its
seasonal forecast system, Austria runs its operational model. Generally speaking, I’d say
that NMSs tend to use it as a back up, which has proven to be very helpful, like in the case of
Denmark who facing a major computing issue a couple of years back, had to use our system
to produce their operational forecast.
 Our HPC also allows us to host the largest meteorological archive in the world

6. Observation Life cycle
IngestionConversion
AEOLUSDecoding
EUMETCAST
TIGGETIGGE -
LAMEFAS
Acquisition
Observations flow
Pre -
Processing
Extractions
MARS
Internet
ECFS
Private
Line(s)RMDCN
Input to
Forecast
Model
MACC
Operations
Quality
ControlEncoding

7. Data Acquisition
7EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Preprocessing, CAMS, TIGGE, EFAS-GEO
ECPDSecpds
dm1
ecpds
dm2
mspds
dm3
ecpds
dm3
mspds
dm2
mspds
dm1
LANpds
dm1
LANpds
dm2
ECPDS
MASTER
MASTER1
MASTER2
MASTER3
MONI-
TORING
RMDCN –
Private Line(s)Internet
MONI-
TORING
MONI-
TORING
LAN
MONI-
TORING
EUMETCAST

8. Data Acquisition
8EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
 Acquisition
 329 Destinations
 19 different
countries
 Data formats
 TAC, BUFR,GRIB
 NetCDF, HDF, ASCII
 More that 530.000.000
Observations
 More 30 Gbytes / per day

9. EumetCast Data Acquisition
9EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
 EUTELSAT-10A (DVB-S2)
 Basic 50.0 Mbps data rate
 Max 77.0 Mbps data rate
 2015 (Last quarter) - start of
operational High Volume
service
 data file volume ~900
Mbytes

10. Traffic Volume Trends Report
10EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
0
10
20
30
40
50
60
70
80
90
100
Internet LAN RMDCN
2007
2008
2009
2010
2011
2012
2013
2014

11. 11EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
A basic description of our models
OOPS
IFS
Product Generation
Data Storage
Encoding + Caching
Processing
Observations
+ Visualisation
+ Web services

12. 12EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Major assimilated datasets
Surface
stations
Radiosonde
balloons
Polar,
infrared
Polar,
microwave
Geostationary, IR
Aircraft
Receive 530 million observations
from more that 300 sources daily.

13. Meteorological Fields
13EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Operational models produce:
13 millions fields daily
Totalling 8 TB/day

14. ECMWF products
14EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
77 million products
disseminated ever day,
totalling 6 TB.
Interpolate output fields into user
required grids
Product generation is also subject to
a dissemination schedule (time
critical)
Products also served via web
visualisation services

15. October 29, 2014
Models Volume
15EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

16. Product Dissemination
16EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
 Dissemination
 309 Destinations
 60 different
countries

17. MECMWF’s Meteorological Archival and Retrieval System
17EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
 A managed archive, not a file system
 Users not aware of the location of the data
 Retrievals expressed in meteorological terms
 Data is kept forever:
 Dataset becomes more useful once enough data
has been accumulated
 Deleting old data in an exponentially growing
archive is meaningless
 Consists of 3 layers:
 FDB - cache at the HPC level (~80% hit ratio)
 DHS - HDD cache (~80% hit ratio)
 HPSS Tape system

18. MECMWF’s Meteorological Archival and Retrieval System
18EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
 Fully distributed (migrated 2012)
 15 servers for metadata and data
movers
 40 PB primary archive
 1 PB of disk cache (2.5%)
 110 billion fields in 8.5 million files
 200 million objects/65 TB added
daily
 7000 registered users
 650 daily active users
 100 TB retrieved per day, in 1.5
million requests

19. Useful links
19EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS