The document provides information about the Finnish Meteorological Institute (FMI) including: - FMI has roughly 650 full-time employees split between research and operational services related to public safety, commercial, and other sectors. - FMI's software development team consists of 40-50 developers working across several units, with all new development done as open source. - In 2013, FMI began openly providing its data in machine-readable formats through an open data portal.

1. FMI Information
Management System
RoopeTervo
LeadArchitect
Finnish Meteorological Institute

2. If not stated otherwise, omages by FMI. Licence CC4BY
Finnish Meteorological Institute
2
• Roughly 650 FTE
• Research Institute
• Operative services
• Safety / Public / B2B
• Roughly half of the employees on operative side
https://www.youtube.com/watch?v=WzIVAEvLH2E

3. If not stated otherwise, omages by FMI. Licence CC4BY
Software Development at FMI
• ~40 - 50 developers in 5-7 units
• + contractors
• In principle, all new development as open source
• Many projects in collaboration (i.e. MetCoOp, Namcon, WFS3…)
3

4. If not stated otherwise, omages by FMI. Licence CC4BY
FMI Open Data
• Finnish Meteorological Institute opened its data in 2013
• Basically everything that FMI has property rights was opened
• Data is provided in freely in machine readable format
• https://en.ilmatieteenlaitos.fi/open-data
4

5. If not stated otherwise, omages by FMI. Licence CC4BY
5

6. If not stated otherwise, omages by FMI. Licence CC4BY
Distribution
Product
Human
intervention
Post
processing
Raw data
SATELLITES
• Polar
• Geostationary
RADAR NETWORK
• National
• Baltrad (Norhern Europe)
• Opera (Europe)
NUMERICAL MODELS
• Atmospheric / Ocean / Road
• Local 2.5 km / Global 25 km
• Ensembles
OBSERVATIONS
• Global Network
• Surface / Balloons / Airplanes
LIGHTNNING
DETECTION NETWORK
• Northern Europe
CROWDSOURCING
• Social sensor networks
• Smartphones
• 112 call data
KALMAN MOS LAPS KRIGING
WWW MOBILE TV NEWSPAPERS B2B OPENDATA SAFETY
VISUALIZA
TION
MODEL
SELECTIO
N
POST
PROCESSI
NG
FIELD
MODIFICA
TION
QUALITY
CONTROL
DATASERVERS
WFS
JSON
XML
MAPSERVERS
WMS
WCS
APPLICATIONS
ILMANET
B2B
MOBILE
6

7. If not stated otherwise, omages by FMI. Licence CC4BY
7
Observations
FMI operated -- operational
Automatic surface stations 184
Rainfall measurement sites 91
Sounding stations 3
Air quality stations 28
Buoys 16
Antennas for lightning detection 8
Mareographs 14
Solar 13
Total 481
Automation 95%

8. If not stated otherwise, omages by FMI. Licence CC4BY
8
External air quality 83
Skandinavian lightning detection 29
Baltic mareographs 24
Road weather stations 29
Runway weather stations 63
Foreign surface weather stations 13736
External water measurement stations 63
Road weather stations 629
NetAtmo ~100k
Observations
External

9. If not stated otherwise, omages by FMI. Licence CC4BY
9
Observations
Process
1 / 10
minute
aggregation
LAN / 3G Oracle

10. If not stated otherwise, omages by FMI. Licence CC4BY
Radars
10
• 10 radars
• Range 120-250 km
• Typical file formats HDF5, GeoTiff, PNG
• Sending power 250 000 W
• Receiving power
0,00000000001 – 0,00000001 W

11. If not stated otherwise, omages by FMI. Licence CC4BY
Radars
Products
• Single radars
• Reflectivity
• Cappi (‘rain intensity’)
• Doppler speed (‘wind’)
• Echotop (‘cloud base’)
• Echotop (‘thunder risk’)
• Composite
• Reflectivity
• Rain intensity
• Rain accumulation (1h, 3h, 6h…)
• Cross sections
11

12. If not stated otherwise, omages by FMI. Licence CC4BY
Satellites
Earth Observation
• In weather, used to observe:
• cloud, pressure, wind, temperature...
• Operated by EUMETSAT and NOAA
• Reception, processing and filing of data
• Geostationary satellites
• Received in Germany
• Delivered to Finland via telecommunication satellite and
land link
• Polar satellites
• Received in Helsinki and Sodankylä
12

13. If not stated otherwise, omages by FMI. Licence CC4BY
• Multicast technology
(DVB-S2)
• EUMETCast Europe, data rate
46.215 GB/hour, 1109.16
GB/day
Satellites
Data Transfer
13
Image: FMI

14. If not stated otherwise, omages by FMI. Licence CC4BY
Lightning Detection
• 9 sensor in Finland, 34 in NORDLIS
• Practically all ground strikes detected with < 1km accuracy
• Also part of cloud strikes
• Stored to climate database
• Can be delivered to clients in
approximately 20 seconds
14

15. If not stated otherwise, omages by FMI. Licence CC4BY
Numerical Weather Prediction (NWP)
• Physical modelling of atmospheric
• Models developed in international consortiums
• Typical programming languages: Fortran, C++
• Input: observations, boundaries from other model, satellites
• Output: grid from several surfaces
• Surface, pressure levels, model levels
• Typical data formats:
• GRIB1, GRIB2
(and NetCDF in marine models)
15

16. If not stated otherwise, omages by FMI. Licence CC4BY
Weather
forecasts are
calculated to a
regular grid
16
Products (like at
https://ilmatieteenlaitos.fi) are
often interpolations between
grid points

17. If not stated otherwise, omages by FMI. Licence CC4BY
Numerical Weather Prediction (NWP)
ECMWF
European Centre for Medium-Range
Weather Forecasts (ECMWF) to
create products that support weather
service
ECMWF model gridded data in
Meteorological Archival and
Retrieval System (MARS). It is the
repository of meteorological data at
ECMWF (storage size of PBs of
data)
17
Model ECMWF ECMWF EPS
Spatial
Resolution
15km 15 km
Coverage Global Global
Temporal
Resolution
1-3h 1-3h
Time range 10 days 10 days
Output data Surface,
pressure
levels
Surface 51
members
Update
interval
6h 6h

18. If not stated otherwise, omages by FMI. Licence CC4BY
Numerical Weather
Prediction (NWP)
18
Model Hirlam Harmonie
Spatial
Resolution
7,5 km 2,5 km
Coverage Europe Skandinavia
Temporal
Resolution
1h 15 min
Time range 48h 48h
Update
interval
6h 6h
Output data Surface,
pressure levels,
model levels
Surface,
pressure levels,
model levels

19. If not stated otherwise, omages by FMI. Licence CC4BY
Human Intervention
• Meteorologists selects the best source
model and refines forecast
• The forecast is updated as often as
necessary
• Output is used as primary data for
Scandinavian area
• Software ‘SmartMet’ is done at FMI (C++)
• Meteorologists draws an analysis maps,
fronts, jets, etc…
• Software ‘Mirwa’ is done at FMI (Java)
• The data is handled as objects
19

20. If not stated otherwise, omages by FMI. Licence CC4BY
Human Intervention
• Meteorologists generates warnings
• Software ‘SmartMet Alert’ is done at FMI
(Java)
• Meteorologists does aviation forecasts
• METARS, TAFs, SWC charts…
• Software ‘SmartMet Aviation’ is done at
FMI (Java)
• Several other tasks
• Writing texts, generating ice advisories,
consulting about weather
• Several web based tools
20

21. If not stated otherwise, omages by FMI. Licence CC4BY
Storing Data
• FMI total capacity: 6 PB
• Ceph (Sodankylä satellite center): 1.5 PB
• Ceph (Main site): 2.5 PB  coming 2020
• Fast file system: 300 TB (Sodankylä) + 300 TB
(main site)
• Archive: 1.5 PB (disk) + 2 PB (tape)
• Operational databases: Oracle, PostgreSQL,
MySQL and Redis
• FMI utilizes ECMWF data storage in daily
operations
• ECMWF MARS
• Archive capacity: 225 PB
• Current daily increase: 225 TB
• Data size grows exponentially in near future
21
71.7
287
929
0
100
200
300
400
500
600
700
800
900
1000
2017 2020 2025
Global Numerical Weather
Model Output
Model output per day (TiB)

22. AI inWeather
Forecasting
Examples of Operational Systems and
Possibilities
22

23. If not stated otherwise, omages by FMI. Licence CC4BY
ML can be used in many parts of the process
• Making observations (machine vision)
• Quality Check (unsupervised learning)
• Radar and satellite data (machine vision)
• Weather predictions (i.a. neural networks)
• Post-prosessing (traditional supervised
learning, neural networks)
• Impact analysis
23

24. If not stated otherwise, omages by FMI. Licence CC4BY
MachineVision in Observations
24
https://www.visivis.at/

25. If not stated otherwise, omages by FMI. Licence CC4BY
Precipitation NowCast (Radar)
https://arxiv.org/abs/1912.12132
25
2 years history of radar images
256km
U-Net
Image: Mehrdad Yazdani, Wikipedia, Licence CC BY-SA 4.0
Prediction (i.e. t +2h)

26. If not stated otherwise, omages by FMI. Licence CC4BY
Global SyntheticWeather Radar
Poster:
https://www.star.nesdis.noaa.gov/star/documents/meetings/2019AI/
posters/P2.3_Mattioli.pdf
26
Convolutional Neural Network to create a global
synthetic weather radar based on satellites, lightning
data and numerical weather prediction data.
Image:Mattioli presentation

27. If not stated otherwise, omages by FMI. Licence CC4BY
Level 1
• Post-processing
• Downscaling
Level 2
• Emulate existing
parametrization
scheme
• Correct existing
schemes
Level 3
• Learn new
parametrization
scheme from
simulations or data
Level 4
• Model replacement
Weather Models and Post-processing
27
Credits: Stephan Rasp, slides

28. If not stated otherwise, omages by FMI. Licence CC4BY
Post-processing with Blend

29. If not stated otherwise, omages by FMI. Licence CC4BY
 Method used: “Exponentially weighted moving average” / “Moving average” (Cui
et al. 2012)
 BCt = (1-α)BCt-1 + α(FCSTt-1 – OBSt-1)
 MAEt = (1-α)MAEt-1 + α|FCSTt-1 – OBSt-1|
BC = bias correction, α = decaying weight (0.05), OBS = observation, FCST = model forecast,
MAE = mean absolute error
 BC and MAE are calculated for each forecast model, analysis time and lead time
separately
 Bias corrected forecasts are used to calculate MAE values and calculating the
latest Blend forecast
Bias correction (BC) and MAE

30. If not stated otherwise, omages by FMI. Licence CC4BY
Biascorrection
How large effect will the error have and for how long with different alpha values:
equation: BCt = (1-α)BCt-1 (started with BC value 1)
Decaying weight, alpha

31. If not stated otherwise, omages by FMI. Licence CC4BY
 Verification results based on MAE are used to produce weights for each model
(Woodcock and Engel, 2005):
W1 = (1/MAE1) / (1/MAE1+1/MAE2+...+1/MAEn)
WhereW = weight, n= total number of models
 The Blend is calculated using weights and bias corrected model forecasts:
Blend =W1*BCFCST1 +W2*BCFCST2+...+Wn*BCFCSTn
Where BCFCST = Latest Bias corrected model forecast
Calculating weights and Blend

32. If not stated otherwise, omages by FMI. Licence CC4BY
Neural networks for post-processing ensemble
weather forecasts
Image: ECMWF
Image: Cecbur, source: Wikipedia., License: CC BY-SA 4.0
Loss:
where Φ and φ denote CDF and PDF of a
standardGaussian distribution
Features (X):
• Several weather parameters
• Embedded station location
Label (y): observations
Ouput:
normal distribution mean μ
and standard deviation σ

33. If not stated otherwise, omages by FMI. Licence CC4BY
PredictingWeather Using Neural Networks is
Under Research
Jonathan Weyn:
Can Machines Learn to Predict Weather? Using Deep Learning to Predict
Gridded 500‐hPa Geopotential Height From Historical Weather Data:
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019MS001705
https://github.com/jweyn/DLWP
Peter Düben:
Deep learning for weather and climate (excellent presentation about the
topic):
https://ossg.bcs.org/wp-content/uploads/04-19-dueben-weather.pdf
Challenges and design choices for global weather and climate models based
on machine learning:
https://www.researchgate.net/publication/326043857_Challenges_and_de
sign_choices_for_global_weather_and_climate_models_based_on_machi
ne_learning
Image: Mehrdad Yazdani, Wikipedia, Licence CC BY-SA 4.0

34. TRAINS
Predicting Train Delays Inflicted by Weather
Photo by Kalevi Lehtonen 1955. Not published until Commons in 2014.
https://fi.wikipedia.org/wiki/Tiedosto:Finnish_class_Dm4_locomotive_number_1607_in_the_year_1955.jpg

35. If not stated otherwise, omages by FMI. Licence CC4BY
Predicting Weather Inflicted Train Delays
Data Liikennevirasto (CC4)
Delay between stations
• Passenger trains
• 514 stations
Weather observations
• 19 parameters

36. If not stated otherwise, omages by FMI. Licence CC4BY
Image: Venkata Jagannath. License: CC4-BY
Image: CC0
Image: BiObserver. License: CC4-BY
Results
RFR
RMSE: 5.37
MAE: 3.21
BSS: 0.11
LSTM
RMSE: 4.35
MAE: 2.75
BSS: 0.01
LR
RMSE: 5.59
MAE: 3.11
BSS: 0.08
𝐵𝑆𝑆 = 1 −
𝑅𝑀𝑆𝐸
𝑅𝑀𝑆𝐸𝑟𝑒𝑓
,
where 𝑅𝑀𝑆𝐸𝑟𝑒𝑓 denotes root mean square error
calculated with a mean value over the whole dataset

37. If not stated otherwise, omages by FMI. Licence CC4BY
LSTM shows no real skill
0
25
50
Delay(minutes)
Time
02/2011
Time
06/2016
Time
02/2017
Predicted vs. true delay, case Ahvenus
Predicted delay
True delay

38. If not stated otherwise, omages by FMI. Licence CC4BY
RFR works relatively well
0
50
100
Delay(minutes)
Time
02/2011
Time
06/2016
Time
02/2017
Predicted vs. true delay, average over all stations
Predicted delay
True delay

39. SASSE
Forecasting Electricity Outages Caused by
Convective Storms
Photo by Bryan Alexander. CC BY 2.0

40. If not stated otherwise, omages by FMI. Licence CC4BY
• Trained with observed
outages (Random
Forest Classifier)
• Several input features
• Few years of old data
required
• Based on storm cell
severity and
knowledge about
clients electricity grid
• For example category
2 storm cell is
expected to cause
problems in 10-50 %
of grid nodes
Turn
outages
to money
Power companies
has to give
compensation for
outages
Forecasting Electricity Outages
Extract storm cells
from radar data
Classify the storm
cells
Predict future
outages
Turn outages to
money

41. Resources
General:
https://ilmatieteenlaitos.fi/avoin-data
http://www.slideshare.net/tervo/
http://ilmatieteenlaitos.fi/tekniikka
http://www.ecmwf.int/
GRIB usage examples:
https://polar.ncep.noaa.gov/waves/examples/usingpython.shtml
https://confluence.ecmwf.int/pages/viewpage.action?pageId=57443489
41

42. If not stated otherwise, omages by FMI. Licence CC4BY
Most Important DataTypes
Data Data Type Software
Weather forecasts GRIB1, GRIB2, (NetCDF) PanoPly, IDV, ECMWF GRIB
tools, WGRIB, WGRIB2, gdal,
SmartMet Server, pangeo
Marine forecasts, air quality
forecasts…
NetCDF PanoPly, IDV, gdal, pangeo,
GeoServer
Observations XML, JSON, csv any
BUFR ecCodes, NCEP
Aviation products TAF, METAR, SIGMET… any
IWXXM fmi-avi-message-converter-iwxxm
Radar GeoTiff, HDF5 gdal, GeoServer (see also:
https://en.wikipedia.org/wiki/Hiera
rchical_Data_Format)
Satellites GeoTiff, NetCDF gdal, GeoServer, PyTroll
42