Vancouver Uses FME to Open
Data to the World




Meng Li, GIS Systems Analyst, City of Vancouver

Jonathan Mark, GIS Manag...
We’ll Talk About


  Overview
    The Open Data Initiative and ICIS support
    The Road Ahead Website Upgrade
    Saf...
The Open Data Initiative


  The Open3 Council motion of May 2009
      Open data
      Open standards
      Open sour...
Open Data Website

  September 2009 launch
    About 50 layers, many hard to find
    Mostly static datasets, limited f...
The Road Ahead Website


    Map in AJAX Viewer of MGE using SDF3 files
    Atom and GeoRSS feed data generated
    Sou...
Safe Software in Use


  FME Desktop
    The City has gradually increased its pool of
     licenses to 10
    Being use...
Open Data Infrastructure


  Requirements of source and target data:
    Public VanMap consistency
       layers/groups...
Open Data—Processes/workflow




outputs




          VM source                          Windows
                        ...
Open Data: one Workbench script




                         shape




                                       KML




    ...
Open Data—Automation


  Scheduler Tasks for ETL Scripts (run weekly)
Open Data - Deployment


  All files in KML/KMZ format uploaded weekly
   to the Open Data web server
  All files in oth...
Open Data- Challenges


  Data Catalogue Design:       groupings and classification



  Required Data Groupings and Fil...
The Road Ahead Website
The Road Ahead—New Features


   Hourly update replacing nightly update:
     FME ETL job runs hourly from 7am to 7pm
  ...
The Road Ahead—Data Extract


  Source Data
     SQL Server--point & street-based projects
     Oracle Spatial--street ...
Road Ahead— Workbench file
The Road Ahead-FME Server Deployment



    The Road Ahead FME script workspace
     published to our FME Server (2009)

...
The Road Ahead-FME Server
Interface




   http://gisweb9/fmedatastreaming/
    Roadahead_dataextract/
    prd_roadahead_...
The Road Ahead- Challenges


  FME Date Formatting: customized transformer

  Feed format (RSS or Atom) and specificatio...
Summary


  FME is very good solution for both Open Data
   and Road Ahead (and lots of other things!)

  FME Desktop an...
Thank You!


  Questions?

  For more information:
      meng.li@vancouver.ca
      jonathan.mark@vancouver.ca
      ...
Upcoming SlideShare
Loading in …5
×

Vancouver Uses FME to Open Data to the World

1,201 views

Published on

"Attend this session to learn how the City of Vancouver uses FME to:
• Provide data for both their Open Data and Road Ahead initiatives.
• Improve the City's Road Ahead map with more current information and open formats. (Learn how FME is being used to retrieve the source data from SQL Server and Oracle Spatial, and to generate SDF3/SDF2 files and a GeoRSS/RSS feed integrated with Google Street View, which is accessible from the Road Ahead home page. Plus, see how the Road Ahead FME workspace is also published to FME Server where the relevant data format files can be generated on the fly via data streaming services.)
• Prepare datasets for the City's OpenData website on a weekly basis. "

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,201
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Vancouver Uses FME to Open Data to the World

  1. 1. Vancouver Uses FME to Open Data to the World Meng Li, GIS Systems Analyst, City of Vancouver Jonathan Mark, GIS Manager, City of Vancouver March 29, 2010
  2. 2. We’ll Talk About   Overview   The Open Data Initiative and ICIS support   The Road Ahead Website Upgrade   Safe software in use at the City   Open Data Infrastructure Platform   Workflow, ETL scripts, and deployment   The Road Ahead Website Upgrade Project   ETL, Atom, GeoRSS, data streaming experience
  3. 3. The Open Data Initiative   The Open3 Council motion of May 2009   Open data   Open standards   Open source software   More sharing with ICIS   Directive to make City’s data freely accessible subject to   Security concerns   Privacy concerns   Terms of use
  4. 4. Open Data Website   September 2009 launch   About 50 layers, many hard to find   Mostly static datasets, limited formats   FME used for most of initial creation   January 2010 launch   About 100 layers, all easy to find   Most in DWG, KML, SHP with VanMap groupings   Weekly updating (thanks to FME software)   Can display in Google Maps or Bing Map easily   ICIS-specific data groups for GIS & AddressBC   Demo
  5. 5. The Road Ahead Website   Map in AJAX Viewer of MGE using SDF3 files   Atom and GeoRSS feed data generated   Source data: SQL Server and Oracle Spatial   FME ETL process to update SDF3, SDF2 & feed data hourly 7am-7pm Mon to Fri   The Road Ahead feed data being widely used   Feed data with Street View via data streaming
  6. 6. Safe Software in Use   FME Desktop   The City has gradually increased its pool of licenses to 10   Being used for IMS, Open Data, Road Ahead, and a variety of day-to-day data ETL needs   FME Server   Acquired in November 2009   Important foundation component of our GIS software infrastructure   Moving Open Data and Road Ahead to Server
  7. 7. Open Data Infrastructure   Requirements of source and target data:   Public VanMap consistency   layers/groups specifications (filtering, grouping, naming)   source formats: MDB, DWG, SDF, SQL Server, Oracle   Target Data Catalogue in OpenData Website   formats: DWG, KML/KMZ, SHAPE, CSV, XLS   naming convention: low casing and no spaces   file structure: data folders for FTP and Web servers   flexible for data packaging: via published parameters   ICIS specific data specifications   Automated Execution of the Data Extraction Scripts via Windows scheduler tasks
  8. 8. Open Data—Processes/workflow outputs VM source Windows Scheduler Task ETL Workspace and scripts loaded with Python pre- processing, filtering, routing & post-processing logic
  9. 9. Open Data: one Workbench script shape KML DWG
  10. 10. Open Data—Automation   Scheduler Tasks for ETL Scripts (run weekly)
  11. 11. Open Data - Deployment   All files in KML/KMZ format uploaded weekly to the Open Data web server   All files in other formats uploaded weekly to the public FTP server   FME ETL Scripts (now Desktop) Updates for Flexible Deployment
  12. 12. Open Data- Challenges   Data Catalogue Design: groupings and classification   Required Data Groupings and File Structure   Naming Conventions: machine and user friendly   KML Specification Compliance/Validation Issue   Data fitness of use: attribute requests & selection
  13. 13. The Road Ahead Website
  14. 14. The Road Ahead—New Features   Hourly update replacing nightly update:   FME ETL job runs hourly from 7am to 7pm weekdays   update SDF3/SDF2 (for VanMap) and feeds data   MapGuide Enterprise AJAX Viewer uses SDF3   Atom and Geo RSS feed data generated   The Road Ahead feed data replaced an email list and is being widely used (including EOC)   Feed data with Street View via data streaming
  15. 15. The Road Ahead—Data Extract   Source Data   SQL Server--point & street-based projects   Oracle Spatial--street network reference data   Output files:   SDF3 file of point & line features with 3 Road Ahead categories   Atom xml file, and a GeoRSS xml file   SDF2 files for existing VanMap Road Ahead layers   Feed xml file with Google Street View (not in production)   FME script data extract job automation
  16. 16. Road Ahead— Workbench file
  17. 17. The Road Ahead-FME Server Deployment   The Road Ahead FME script workspace published to our FME Server (2009)   The same ETL script job is run hourly against the FME Server:   "D:FMEServerClientsFMEServerConsolefmeserverconsole.exe" RUN_WORKSPACE Roadahead_dataextract/ uat_roadahead_sdf3_georss_sdf2_lp.fmw   Feed data (Atom and GeoRSS) integrated with Google Street View is accessible live via data streaming services from FME Server
  18. 18. The Road Ahead-FME Server Interface   http://gisweb9/fmedatastreaming/ Roadahead_dataextract/ prd_roadahead_georss_lp_sv.fmw Feed data:
  19. 19. The Road Ahead- Challenges   FME Date Formatting: customized transformer   Feed format (RSS or Atom) and specification issues:   unique ID for feed data,   duplicate feed entries on disjoin hundred block streets   FME Server user account: ID required cross domain access   Authentication limitation: use FME 2010!
  20. 20. Summary   FME is very good solution for both Open Data and Road Ahead (and lots of other things!)   FME Desktop and Server have greatly improved the openness and interoperability of City’s GIS infrastructure platform   FME Server 2010 will expand our abilities in supporting GIS, Open3, Road Ahead and other initiatives going forward
  21. 21. Thank You!   Questions?   For more information:   meng.li@vancouver.ca   jonathan.mark@vancouver.ca   IT Department   City of Vancouver

×