Using FME to Automate Region-Wide Data Sharing


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Using FME to Automate Region-Wide Data Sharing

  1. 1. Using FME to AutomateRegion‐Wide Data SharingSteve Grisé – Vertex3 April 10, 2013
  2. 2. A Perspective Based on recent project work  Working on data models  Defining processes  New editing tools  Getting agreement And a bit of experience  ArcGIS  FME  …
  3. 3. Recent Projects Region of Waterloo  Regional data model for Roads and Addresses  Editing tools  Data exchange with Cities York Region  Parcels, Addresses and Roads data model  All Pipes Data Model  Implementation in partnership with Timmons Group
  4. 4. Current Situation Most of the data shared between organizations is almost the same  Different Names, same data Differences Roads Addresses  Dataset names ROADSEGMENTID STREET_NAME HOUSE_NUMBER UNIT_NUMBER  ROAD vs. Roads LEFT_FROM STREET_NAME  Field names LEFT_MUNI ... ADDRESS_POINT_ID ...  ROADID vs. GeoID GeoID HouseNumber  Field values FullStreetName UnitNumber LeftFrom FullStreetName  CAM vs. Cambridge LeftMunicipality AddressID ... ...  DI vs. Ductile Iron
  5. 5. Region of Waterloo ExampleCity Extract, Transform, and Load (ETL), QA QA and Update to Region-wide Database Regional Systems ETL, QA c c Regional update using sources Cambridge Regional Change Sync ETL, QA k k QA Detector Kitchener Region-wide Internal & Database External ETL, QA w w Databases Waterloo QA Issues fixed at source Partner read only access to Region-wide Database ETL can be done on local systems or regional Region-wide Update done by Regional Staff, server. Partners manage their own ETL and QA semi-automated processes to Region-wide model
  6. 6. Our options haven’t changed
  7. 7. A new idea We need a new user experience Arcgis and/or FME on the desktop Source-target mapping vs. programming Eventually a knowledge-based system that does 80% of the work for you FME becomes the engine that drives your data movement
  8. 8. Gizinta.com1.0 – process and some UI2.0 – knowledge-based ETL
  9. 9. Approach Gizinta.fmw Extract Sources Transform Jdbdjbjdb Jdbdjdj Jnjdbjn Add fields Map values Jndjdbjnj nGizinta Project Calculate values Load Multiple options
  10. 10. Why Use FME? Faster Larger datasets You prefer it More challenging extract and load needs Use a bit of FME Server…
  11. 11. Demo Initial workbench Move Cambridge roads to regional data model
  12. 12. Next Steps July 1.0 release Exploring team and approach for 2.0 Free downloads on  ArcGIS Galleries  FME Store Premium subscription options Looking for a few good projects over the next several months
  13. 13. Longer Term Ideas FME cloud implementation  Build and test locally  Submit data and Gizinta project to online service  Download or distribute via Gizinta site Benefits  Pay-for-use model  Reduced processing time
  14. 14. Thank You! Questions? For more information:  Steve Grisé   226-444-0323