Why we need open data and how should we provide it. FMI provides the same data in its own open data portal but also in AWS public dataset program. Different use cases require different services and channels.

1. Open data distributed on Amazon’s cloud service
Roope Tervo, Mikko Visa
Finnish Meteorological Institute

2. Motivation
The world is changing
A role of NMHS is also changing
Challenges:
Ensure authoritative voice in warnings
More efficiency in development and operations
Ensure the impact of produced information
9/4/2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko
Rauhala
3

3. Motivation
Streamline collaboration with partners
• Open data and open source software can boost the development
• Open data lower usability barrier and thus increase data usage
• Collaboration is open and easy
• No long and burden negotiations – just evaluate, use, develop
• Using Open Source Software prevents from vendor locks
9/4/2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko
Rauhala
4
ApplicationsObservations
Collaboration and OSS Proprietary software

4. Motivation
Increase appropriability of
weather and climate data
• Maximal coverage requires
several different channels
and services
• One organization can’t handle
them all
9/4/2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko
Rauhala
5

5. Motivation
Support Research
• Open data and open source empowers research as
well
• Easy and open methods to access and analyze the data
• Methods are repeatable when anyone can access the tools
9/4/2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko
Rauhala
6

6. • Finnish Meteorological Institute
opened its data in 2013.
• Basically everything that FMI has
property rights was opened.
• Both (near) real-time and historical
and climatological data.
• Data is provided in freely in machine
readable format.
9/4/2018 7
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
Open data distributed on Amazon’s cloud service

7. FMI Open Data Portal follows INSPIRE requirements.
FMI Open Data
Data Portal
Meta data
Services
The very same data portal works as Open Data and
INSPIRE portal.
9/4/2018 8
ISO19115 WFS WMS
CSW
Grid Series
Observations
Time Series
Observations
Data
Models O&M
Simple
Feature
GRIB
NetCDF GeoTiff
Open data distributed on Amazon’s cloud service

8. FMI Open Data
Registration
o Registration is required to use View and Download
Services
o Working email address is the only mandatory
information
o After registration the user gets an API key which have to
be added into all requests
o GET parameter fmi-apikey=…&
o Header fmi-apikey; …
o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?
o One can create several API keys with one email
9/4/2018 9Open data distributed on Amazon’s cloud service

9. And a little over
830 000 data
downloads
per day
(9,6 req/s)
At the moment
about 11 700
registered users
Some Experiences
9/4/2018 10Open data distributed on Amazon’s cloud service

10. Motivation
Increase appropriability of
weather and climate data
• Data is valuable only when it
reach relevant audience
• Maximal coverage requires
several different channels
and services
 FMI joins Amazon’s Public
Datasets
9/4/2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko
Rauhala
11

11. FMI OpenData on AWS
• FMI OpenData is also distributed on
Amazon Web Services (AWS) Cloud platform
• 2-years pilot
• Started in May 2017
• Hirlam surface and pressure levels in the first stage
• The objective is to
• increase the utility and effective use of weather and climate data
• support public-private-partnership
• Specially convenient for users who need the whole model data
• i.e for post-processing or generating map visualizations
• Licence: CC BY 4.0
9/4/2018 Open data distributed on Amazon’s cloud service 12

12. FMI OpenData on AWS
• Content
• Hirlam Surface
• Hirlam Pressure Levels
• Coverage: Europe
• Grid resolution: 7,5 km
• Updates: 4 times a day
• Time range: 54 hours (from model
run start)
• Time step: 1 hour
• Archive kept during the pilot
• Parameters:
http://en.ilmatieteenlaitos.fi/open-
data-on-aws-s3
9/4/2018 Open data distributed on Amazon’s cloud service 13

13. FMI OpenData on AWS
• Access through buckets:
• Surface data: fmi-opendata-rcrhirlam-surface-grib
• Pressure level data: fmi-opendata-rcrhirlam-pressure-grib
• Browse bucket content:
• http://fmi-opendata-rcrhirlam-surface-grib.s3-website-eu-west-
1.amazonaws.com/
• http://fmi-opendata-rcrhirlam-pressure-grib.s3-website-eu-west-
1.amazonaws.com/
• Public Amazon SNS topics are available for every new object added to
the Amazon S3:
• arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-surface-grib
• arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-pressure-
grib
9/4/2018 Open data distributed on Amazon’s cloud service 14

14. FMI OpenData on AWS
See documentation:
http://en.ilmatieteenlaitos.fi/open-data-on-aws-s3
9/4/2018 Open data distributed on Amazon’s cloud service 15

15. Requests by Data Format
in FMI Open Data Portal
9/4/2018 Open data distributed on Amazon’s cloud service 16

16. Hirlam Requests in binary
format by Channel
9/4/2018 Open data distributed on Amazon’s cloud service 17

17. Would mean
200 – 300 users who
fetch whole data
operatively
At the moment
6000 - 46000
downloads per day
Some Experiences
9/4/2018 18Open data distributed on Amazon’s cloud service

18. Point requests are
still by far the most
popular type
For binary data
downloaders S3 is
the most popular
channel
Some Experiences
9/4/2018 19Open data distributed on Amazon’s cloud service

19. www.fmi.fi
https://github.com/fmidev
https://en.ilmatieteenlaitos.fi/open-data
http://en.ilmatieteenlaitos.fi/open-data-on-aws-s3
http://roopetervo.com
http://www.slideshare.net/tervo