CONNECT. TRANSFORM. AUTOMATE.
Jakob Lagerstedt, Mats Högström
Constructing a Federated Data Distribution Service for Unive...
CONNECT. TRANSFORM. AUTOMATE.
Overview
Background
Previous Attempts
A modern solution
Some tricks learned
Results
CONNECT. TRANSFORM. AUTOMATE.
Background
We are from the
Swedish University of Agricultural Sciences
CONNECT. TRANSFORM. AUTOMATE.
Background
Data and researchers in Sweden
CONNECT. TRANSFORM. AUTOMATE.
Background
Not free
Image courtesy of KJGarbutt (CC BY 2.0)
CONNECT. TRANSFORM. AUTOMATE.
Background
Strange formats
Image courtesy of Toronto Public Library
Special Collections (CC ...
CONNECT. TRANSFORM. AUTOMATE.
Previous attempts
Distribution was difficult
Required authorization and data preparation
Res...
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade ()
Do not panic
All the code is in the handouts if you are interested!
I...
CONNECT. TRANSFORM. AUTOMATE.
FME server Rest API
Token service /fmetoken/service/generate
HTTP get /fmerest/repositories/...
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (Shutdown Python script)
To enable unsupported functions
Zip & Email & R...
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (External script example 1 handout)
Use an external script to enable fas...
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite example 2 handout)
Wonderful format for mixed environments
W...
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
However, FME does not read as efficiently as other formats
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
does not set spatial index
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
Can fix
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (Google Fusion Tables)
Fusion tables are nice for reporting
But often wr...
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (KML network link)
KMLs are nice for reporting
CONNECT. TRANSFORM. AUTOMATE.
Results
23 Universities
100K distributed data sets ( 400 * total area of country)
6000 Uniqu...
Thank You!
 Questions?
 Handout link
http://pluto.slu.se/drop/public.php?
service=files&t=2e0c13550e0c71e963e6f6bb2c5256...
Upcoming SlideShare
Loading in...5
×

Constructing a Federated Data Distribution Service for Universities

467

Published on

The GET application enables students and staff at all Swedish universities to download selected areas of basemaps from Lantmäteriet (the Swedish Land Survey).

The presentation will contain a technical overview of the application and lessons learned.

See more presentations from the FME User Conference 2014 at: www.safe.com/fmeuc

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
467
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Constructing a Federated Data Distribution Service for Universities

  1. 1. CONNECT. TRANSFORM. AUTOMATE. Jakob Lagerstedt, Mats Högström Constructing a Federated Data Distribution Service for Universities
  2. 2. CONNECT. TRANSFORM. AUTOMATE. Overview Background Previous Attempts A modern solution Some tricks learned Results
  3. 3. CONNECT. TRANSFORM. AUTOMATE. Background We are from the Swedish University of Agricultural Sciences
  4. 4. CONNECT. TRANSFORM. AUTOMATE. Background Data and researchers in Sweden
  5. 5. CONNECT. TRANSFORM. AUTOMATE. Background Not free Image courtesy of KJGarbutt (CC BY 2.0)
  6. 6. CONNECT. TRANSFORM. AUTOMATE. Background Strange formats Image courtesy of Toronto Public Library Special Collections (CC BY 2.0)
  7. 7. CONNECT. TRANSFORM. AUTOMATE. Previous attempts Distribution was difficult Required authorization and data preparation Researchers would share pieces of data on USB drives (sneakernets) Image from http://japanese.engadget.com/2005/11/12/hdd-circa-1975/
  8. 8. CONNECT. TRANSFORM. AUTOMATE. A modern solution Geographic Extraction Tool
  9. 9. CONNECT. TRANSFORM. AUTOMATE. A modern solution Geographic Extraction Tool
  10. 10. CONNECT. TRANSFORM. AUTOMATE. A modern solution Geographic Extraction Tool
  11. 11. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade () Do not panic All the code is in the handouts if you are interested! Image courtesy of Nico Kaiser (CC BY 2.0)
  12. 12. CONNECT. TRANSFORM. AUTOMATE. FME server Rest API Token service /fmetoken/service/generate HTTP get /fmerest/repositories/" + your_fme_repo_name + ".json Tricks of the trade (Servlet) Image courtesy of abraham.williams (CC BY 2.0)
  13. 13. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (Shutdown Python script) To enable unsupported functions Zip & Email & Remove files that have been downloaded, Anything you want!
  14. 14. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (External script example 1 handout) Use an external script to enable faster debugging Monkey patch sys or import something from path (yes we actually use the linux version)
  15. 15. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (SpatiaLite example 2 handout) Wonderful format for mixed environments We have Qgis, Arcmap, FME, they all read it! Image courtesy of juicyverve (CC BY 2.0)
  16. 16. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (SpatiaLite) However, FME does not read as efficiently as other formats
  17. 17. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (SpatiaLite) does not set spatial index
  18. 18. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (SpatiaLite) Can fix
  19. 19. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (Google Fusion Tables) Fusion tables are nice for reporting But often writes fail (we do not know why, reported the issue)
  20. 20. CONNECT. TRANSFORM. AUTOMATE. Tricks of the trade (KML network link) KMLs are nice for reporting
  21. 21. CONNECT. TRANSFORM. AUTOMATE. Results 23 Universities 100K distributed data sets ( 400 * total area of country) 6000 Unique users Image courtesy of saamiblog (CC BY 2.0)
  22. 22. Thank You!  Questions?  Handout link http://pluto.slu.se/drop/public.php? service=files&t=2e0c13550e0c71e963e6f6bb2c52569c  For more information:  jakob.lagerstedt | mats.hogstrom @slu.se  Swedish University of Agricultural Sciences CONNECT. TRANSFORM. AUTOMATE.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×