No pressure, but the Workspace needs fixing so the coworker can go have her baby...


Published on

Presented by Bruce Harold, Esri
See more FME World Tour 2014 presentations at

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • FME technology use at Esri:Who wants a report and who wants a story?Bad luck for anyone wanting a report, here is my story…
  • Hollywood script buyers always look for these aspects of a story, so if you’re looking to retire early by selling a movie script make sure you include them.FME professionals will recognize these plot lines in their everyday work. You get a tap on the shoulder, grudgingly launch into a project, battle the ghouls of Bad Data, find a way to beat them in their hollowed out volcano lair, achieve the satisfaction of success, the banality of documenting your workflow, and emerge with a functioning workspace with which to make the (customer’s) world a better place.
  • The Greek ideals of persuasion (in our case to use Workbench)
  • More specifically, FME use cases tend to crop up in patterns. Here are some that are format-driven.An example of a ‘weird’ structure would be 4kB-wide fixed record length data from a vehicle data aggregator.The baby situation is just bad planning…
  • Here are some that are functionality-driven drivers for using FME technology.By ‘feature granularity’ I mean taking individual features down different tracks based on some test.‘ETL ease of use’ would be one theme I can pick out for why we elect to use FME technology ahead of (say) scripted workflows.
  • More specifically, FME use cases tend to crop up in patterns that aren’t functionality drivenThe baby situation is just bad planning…
  • Esri has 900 geoprocessing tools, plus change. You would think we covered it all, and we do, but sometimes at the cost of coding.FME technology is used by Esri when it makes sense, to plug gaps, and in an easily developed and documented environment – Workbench.
  • Esri’s core GP software does have field mapping and casting capabilities. See the Field Map control here, it crops up in several tools. It is tricky to script.Model Builder is of course the Esri environment analogous to Workbench.
  • The baby story is true, I got a call late one day from a co-worker who had to complete a job for an expectant mother being induced that evening.By that I mean the baby was coming that evening AND the project had to complete that evening. Who plans these things?The co-worker didn’t know Workbench well, but we made the deadline (and baby arrived).
  • The first project I’ll outline that depends on FME technology is Community Maps
  • If you are the curator for these types of useful data WE WANT IT.We will cook it into tiled map services and serve it to the World.
  • We do eat our own dog food, see here Models, script tools and of course an ETL tool handling the critical change detection function.
  • In fact it is all scripted after importing the models and tools…which IMHO is a better approach than the executable script approach native to FME.I think SAFE should re-expose fmeobjects as a Python module, but that is a discussion for another day.
  • Community TIGER is another major project where we use FME technology.TIGER is the US Census Bureau street centreline dataset they make publicly available.The general idea is to streamline aggregating street and address data for Census purposes.
  • Speaking of Census’ purposes, the URL isn’t really a workflow Esri does, Census build the geocoding service with our outputs. So that’s the end game.AFAIK this is the first announcement at any event of the existence of Census’ geocoder!
  • Note the ETL tools.
  • DI is handy for bulk 3D work, here I made some 3D parcel data for geocoding development work, for the real estate market.We will support 3D data in our new US Apartment locator style due out in Q3.
  • At the extreme end we have this example from the Chamber of Horrors – AIXM file processing.Aeronautical team process these very large XML files for military customers.
  • We find many businesses communicate in lowest common denominator formats – Excel, Dbase, CSV etc.You can script for these in Python but why go to the trouble?Workbench delivers easy development and documentation for business-friendly formats, in and out.
  • Nothing fancy here, just handling Dbase as we get it from the customer…
  • When I get bored I do research.I leveraged something I found in a Python list I’m on and was pleasantly surprised to see the approach worked in a PythonCaller.What is happening here is a pool of worker processes is doing something with geometry in parallel.SAFE will be able to tell you if this should work generally, and if it breaks their license agreement .
  • I just want to call out Python again here; Esrilites tend to be Python people, personally I use PythonCaller a lot, it gets you out of jail frequently.My joke hypothesis in this slide that with generic formats and dynamic schema, plus PythonCaller, you can do anything.
  • No pressure, but the Workspace needs fixing so the coworker can go have her baby...

    1. 1. CONNECT. TRANSFORM. AUTOMATE. No pressure, but the Workbench needs fixing so the co-worker can go have her baby... Bruce Harold Product Manager, Esri
    2. 2. Where does Esri use FME technology? Report or Story? Show of hands please
    3. 3. Your Next 20 19 Minutes Journey  Ordinary World  Call to Adventure  Refusal of the Call  Meeting with the Mentor  Crossing the Threshold  Tests, Allies and Enemies  Approach the Innermost Cave  Ordeal  Reward  The Road Back  Resurrection  Return with the Elixir  Better World
    4. 4. Ethos, Logos, Pathos  Project an image of credibility  Then go prove it  Win hearts and minds  Get more work to do….
    5. 5. Patterns in FME Use Cases - Formats  Spreadsheets  DBase  OGC Formats  CAD  CSV (the optimists spreadsheet)  Custom (weird) structures  XML (what are they thinking?)  JSON  RSS  Directories of Data  URLs  The baby is coming
    6. 6. Patterns in FME Use Cases - Functions  Cross Database Joins  Casting Data Types  Lists & Cardinality  Vertical Datum  Coordinate Rounding  NULL handling  SDE Updating  Feature Granularity  Binary Attributes  Change Detection  Concatenation  Expressions  REST/URL handling
    7. 7. The Dreaded Functionality Gap  Change is guaranteed  Change always meets resistance  It is hard to design for change  Luckily, our software is complete and perfect  Except for…
    8. 8. Sometimes it’s a judgment call Field Map Control Modelbuilder
    9. 9. Baby Story Baby needed TestFilter
    10. 10. Community Maps
    11. 11. Community Maps  Crowd to Cloud  Authoritative Data for:  Streets, Topography, Imagery, Hydrography  Esri supply the data model  Contributor sends data regularly  Contributor data is Change Detected (FME!)  Adds, Deletes go to SDE Version, gets posted  Map tile cache refreshed
    12. 12. Community Maps Processing Contributor ID: {B0BEE756-A683-4700-ACCE-9585A07559C4}
    13. 13. Community Maps Processing
    14. 14. Community TIGER  Census Bureau project to harvest:  Streets with Address Ranges  Address Points (Multiple Dwelling Units)  Local Government Boundaries  39,000 Local Governments  Quality Control Steps – Data Reviewer  Interesting combination of DR workspace & DI  Change Detect, De-dupe, Compress, Load to SDE
    15. 15. Community TIGER Workflows
    16. 16. Community TIGER ETL Tools
    17. 17. 3D Work  Bulk 3D Geometry Creation, Editing  3D Geometry Comparisons  Next Generation 3D Geocoding
    18. 18. Aeronautical Charting Example  FMW is 36Mb  110 Bookmarks  1000+Transformers
    19. 19. Professional Services  Handling Excel, DBase, CSV without stress  Eliminate need for scripting in Python Join Excel Read CSV
    20. 20. Professional Services Cont’d  Example Multiple DBase Join 7 x Dbase JoinersRead File GDB
    21. 21. On the Slab in the Lab Thread Pool
    22. 22. The Perfect Workspace  Generic Format + Dynamic Schema + Python
    23. 23. Thank You!  Questions?  For more information:  Bruce Harold,  Esri Inc. 