Meteorological and Aviation Weather Open Data implementation utilising OGC standards. Presentation held in FOSS4G Europe

1. Meteorological and Aviation Weather Open
Data implementation utilising OGC
standards
Finnish Meteorological Institute
Finnish Meteorological Institute
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

2. Finnish Meteorological Institute opened its data in 2013.
Basically everything that FMI has property rights was opened.
Data is provided in freely in machine readable format.
17.7.2015 Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
2
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data

3. Data set Description Time
Interval
Estimated
publish date
Weather
Observations
Temperature, Wind,
Humidity, Ground
Temperature…
10 min Open,
older data to be
added
Sun Radiation UV, Short and Long
Term Radiation…
1 min Open
Marine
Observations
Waves, Sea
Temperature, Sea
Level…
1 h Open
Weather Radars Precipitation Rate,
Precipitation Amount…
5 min Open,
older data to be
added
Lightning Thunder Strikes in
Finland
5 min Open
Example of Data Sets
17.7.2015 3Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

4. Example of Data Sets
Data set Description Time Interval Estimated
publish date
Real Time
Observations
Real Time Observations from
specific location(s)
AWS 2010 –
Soundings 1959 –
Flashes 1998 –
Sea Level 1971 –
Waves 2005 –
Open
older data will
be added
Climatological
Observations
Dayly and monthly
temperature mean and
extreme values from weather
stations
1959 - Open
Climatological
Observations
Monthly temperature and
precipitation rate mean
values interpolated to grid
1961 - Open
Climatological
Reference
Climatological Reference.
Temperature, humidity,
pressure, precipitation
amount and snow depth.
Reference seasons:
1971-2000 1981-
2010
Open
17.7.2015 4Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

5. Example of Data Sets
Data set Description Time Interval Estimated
publish date
Weather forecast
model HIRLAM RCR
Point forecasts and grid
data
Latest model
run
(4 times a day)
0…54 h
Open
Sea forecast models Sea level point
forecasts, Wave (WAM)
and current (HBM) as
grid data
Latest model
run
(4 times a day)
0...54 h
Open
Environmental
Monitoring Facilities
Weather observation
stations, radars…
2015
Aviation
Observations
METAR 30 min open
Ground & mast
observations
Special observations
from ground and masts
2016 /Open
17.7.2015 5Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

6. Example of Data Sets
Data set Description Time Interval Estimated
publish date
Air Quality
Observations
Air Quality Observations 1h 2015-2016
Silam Model Dispersion Model for Air
Quality, Forest Fire and
Pollen
Latest model
run (once a day)
0…96h
2015
HELMI Ice Model Ice forecast model Latest model
run
(4 times a day)
0...54 h
open
Soundings Temperature, Humidity,
Pressure, Wind from
ground to 25 km height
2 times a day 2015
Road Weather
Observations (LIVI)
Road Weather
Observations
10 min open
17.7.2015 6Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

7. FMI Open Data Portal follows INSPIRE requirements.
FMI Open Data Portal
Meta data
Data
Models
Services
The very same data portal works as Open Data and
INSPIRE portal.
17.7.2015 7Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

8. Catalog Service
(CSW)
o Based on GeoNetwork
View Service (WMS)
o Based on GeoServer
o Only the most common layers
published
17.7.2015 8Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

9. Download Service
(WFS 2.0)
o Web Feature Service (WFS) 2.0
Simple Profile
o Based on stored queries
o Predefined data sets with
possibility for additional
parameters (i.e. time and
area)
o In-house production
17.7.2015 9Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

10. Registration
o Registration is required to use View and Download
Services
o Working email address is the only mandatory
information
o After registration the user gets an API key which have to
be added into all requests
o GET parameter fmi-apikey=…&
o Header fmi-apikey; …
o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?
o One can create several API keys with one email
17.7.2015 10Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

11. Usage Limits
With one API key it’s allowed to
o do at most 20 000 requests per day to Download Service
o do at most 10 000 requests per day to View Service
o do at most 600 requests per 5 minutes to both services
o If all observations from one time step is calculated to as one,
little over 17 000 new data sets are published daily
o So, with one API key it’s allowed load everything once
o View service can be used for testing but can not be used as a
back end for popular clients
17.7.2015 11Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

12. Open Data Service
Cluster
S1 S2 S3
Client Data Service
Cluster
S1 S2 S3
Load Balancer
Configuration
Data
(NFS)
Configuration
(NFS)
Database

13. MetoLib
o Open source JavaScript library produced by Finnish
Meteorological Institute
o Helps users to load and use the data
o Supports multi point coverage data format
o Python version is on the list
Easy
requests Cache
Parse the data
to as JSON
17.7.2015 13Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

14. Data Models
o Observations and point
forecasts as GML
o The same data is published in:
o MultiPointCoverage
o MeasurementTimeSeries
o SimpleFeature
o Gridded data is provided in
appropriate binary format (Grib,
NetCDF, GeoTiff…)
o WFS members contains the
metadata ‘envelope’ with a link
to a actual data
17.7.2015 14Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

15. Data Models
gmlcov:MultiPointCoverage
17.7.2015 15
gml:rangeSet
gml:doubleOrNilReasonTupleList
The data is listed for every
point defined in domain set.
gml:domainSet
gmlcov:simpleMultiPoint
The coverage is
defined as a list of
points in 4
dimensional grid (lat,
lon, height, time).
gmlcov:rangeType
The parameters
listed in range set
are defined in
separate element.
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

16. Cons
- Not intuitive
- No natural
structure of XML
 XSLT and
Xpath don’t work
Pros
+ Compact
+ Efficient
+ Small file size
+ Works for many
data types
17.7.2015 16
Data Models
gmlcov:MultiPointCoverage
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

17. Data Models
wml2:MeasurementTimeseries
17.7.2015 17
wml2:MeasurementTimeseries
One member contains time
series for one parameter
and one location
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

18. Cons
- Lots of repetition
- Large file size
- Heavy for DOM-
based parsers
- Don’t work i.e. for
thunder strikes
Pros
+ Intuitive
+ Easy to use
+ XSLT & XPath
works
17.7.2015 18
Data Models
wml2:MeasurementTime
series
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

19. Data Models
SimpleFeature
17.7.2015 19
SimpleFeature
One member contains one
time, one parameter and
one location
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

20. Cons
- Lots of repetition
- Very large file size
- Heavy for DOM-
based parsers
Pros
+ Intuitive
+ Easy to use
+ XSLT & XPath
works
+ Ready client
support
17.7.2015 20
Data Models
SimpleFeature
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

21. • Aviation weather reposts are delivered as IWXXM
• New data model coming into use in aviation
• Consists of the same elements than other messages
• om:phenomenonTime, om:procedure, om:featureOfInterest,
om:result
• Content of the METAR is in om:result part as
• extracted into XML elements
• original, “old fashion”, METAR string
Data Models
aviation observations IWXXM / METARS
17.7.2015 21Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

22. Data Type Data Format
Observations wml2:MeasurementTimeseries
gmlcov:MultiPointCoverage
SimpleFeature
Point Forecasts wml2:MeasurementTimeseries
gmlcov:MultiPointCoverage
SimpleFeature
Lighting Observations gmlcov:MultiPointCoverage
SimpleFeature
Grid Forecasts XML Envelope + Grib2/NetCDF
Radar Images GeoTiff / PNG images
METAR IWXXM
17.7.2015 22Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

23. 17.7.2015 23
Data Models File size Comparison
81.7
52.9
1.81.3 1.2 0.2
0
10
20
30
40
50
60
70
80
90
Document Size
[MB]
Compressed
DocumentSize
[MB]
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

24. 17.7.2015 24
Data Models Popularity
Comparison
80
19.8
0.2
0
10
20
30
40
50
60
70
80
90
Downloads[%]
Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

25. And a little over
430 000 data
downloads
per day
(5 req/s)
At the moment
about 7200
registered users
Some Experiences
17.7.2015 25Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

26. Practically
no client
supports
complex
features
Although standards
are followed, there’s
a gap between
provided data model
and clients’
capabilities
Some Experiences
17.7.2015 26Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

27. GeoServer is
modified to support
stored queries in
WFS 2.0 (released
in version 2.7)
Also simple features
had to be supported
Some Experiences
17.7.2015 27Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

28. Industry is
happy to use
standardized
services
Amateur and
freelancer coders
would prefer simple
JSON API
Some Experiences
17.7.2015 28Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

29. www.fmi.fi
http://www.slideshare.net/tervo/
https://en.ilmatieteenlaitos.fi/open-data