Turbocharging FME: How to Improve the Performance of Your FME Workspaces

4,436 views

Published on

Getting the best performance out of FME is as important to us as it is to you. In this webinar, you’ll get tips from three FME experts - Mark Ireland (FME Evangelist), David Eagle (FME Certified Trainer and Professional) and Dale Lutz (Safe Software Co-founder). They’ll share easy-to-apply advice on: querying databases efficiently, making the most of FME's new multiprocessing capabilities, and simple techniques to speed up your workflows.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,436
On SlideShare
0
From Embeds
0
Number of Embeds
814
Actions
Shares
0
Downloads
35
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Turbocharging FME: How to Improve the Performance of Your FME Workspaces

  1. 1. Turbocharging FME:How to Improve the Performance of YourFME WorkspacesHost:Ken BraggPanel:Mark Ireland, David Eagle, Dale Lutz December 12, 2012 12/12/12
  2. 2. Host  Ken Bragg European Services Manager Safe Software Questions are Encouraged!
  3. 3. Our Panel  Mark Ireland (iMark) Product Evangelist, Safe Software Killarney, Manitoba, Canada  David A. Eagle (The Eagle) Principal Consultant, 1Spatial Group Ltd. Cambridge, United Kingdom  Dale Lutz (The Founder) Co-Founder & VP of Development Safe Software Vancouver, BC, Canada
  4. 4. 100 exuberant employees Secret HQ: Vancouver Thousands of happy customers, certified professionals, trainers, system integrators, and value added resellers; all around the globe!
  5. 5. FME Capabilities Transform Data to Use and Share  Convert spatial data between hundreds of formats  Transform spatial data into the precise data model you need  Integrate multiple different data types into a single data model  Share spatial data with people where, when and how they need it
  6. 6. FME Workbench:Graphical Data Flow Authoring
  7. 7. FME : Our PromiseFaster than a Helicopter?Faster than ever!
  8. 8. Turbocharge your Workspaces!
  9. 9. Poll How Long do your Workspaces Take?
  10. 10. iMark – Mark IrelandWorkbench Performance
  11. 11. Workspace Performance The key to workspace performance is to reduce the amount of data being processed.
  12. 12. Feature vs GroupSome transformers work on one feature at a time; others work on agroup of features.
  13. 13. Feature BasedMeasuring individual heights...Feature-based transformers work on one feature at a time.They are LESS resource intensive. 5’ 8” 7” 9”
  14. 14. Group BasedMeasuring average heights for each gender...Group-based transformers work on many Average = 5’ 9”features at a time. They are MORE resourceintensive. 5’ 7” 6’ 0” 5’ 8” 5’ 9”
  15. 15. Transformers Some group-based transformers have settings to turn them into feature-based • AttributeAccumulator • Aggregator • Clipper • NeighborFinder
  16. 16. First Writer Order your writers so the one receiving the largest amount of data is first in the list.http://evangelism.safe.com/fmeevangelist64-2/
  17. 17. Attribute Handling • Remove unwanted attributes • Beware of large lists And don’t store geometry as an attribute!
  18. 18. Poll Where does it hurt?
  19. 19. The Eagle – David EagleWorking with Databases
  20. 20. Turbocharging FME:Working with DatabasesDavid EaglePrincipal ConsultantFME Certified Professional & Trainer1Spatial Group Ltd December 12, 2012
  21. 21. Learn to be a ‘Speed Reader’ Being able to read quickly is often beneficial When you have a database, you often have volume  Read data intelligently and achieve efficiency! A typical approach is where FME does the heavy lifting 1 - Read_and_clip.fmw
  22. 22. Reader Parameters Master FME ‘Parameters’ - become an FME Hero!  Parameters allow you to apply control  Enable flexible workspaces – Prompt and Run  Make the database do the work
  23. 23. Embrace the Spatial Index Spatial indices are used by spatial databases (databases which store information related to objects in space) to optimize spatial queries. Many conventional index types do not efficiently handle features such as how far two points differ, or whether points fall within a spatial area of interest.http://en.wikipedia.org/wiki/Spatial_database
  24. 24. Read ‘within’ All Readers since FME 2011 have allowed you to specify a reader bounds Not all formats have a true spatial index  No performance gain but less data in memory Spatial databases can have a spatial index Create index with SQL or create with FME! 2 - Read_with_spatial_index.fmw
  25. 25. Read ‘where’ and ‘what’ Use FME to issue a SQL Where Clause  Be specific about the features you need  www.w3schools.com/sql beginners resource Leverage the power of the parameter ‘Feature Types to Read’  Always see what’s new 3 - Read_WhereandWhat.fmw
  26. 26. Synchronous Reading/Writing Mid translation reading/writing may be required Beneficial when you want to be specific Allows native format query – greater efficiency SynchronousReading.fmw
  27. 27. Ask your friendly DBA Prebuild the data you want as a ‘View’  A stored query that forms a virtual table  More efficient and dynamic for repetitive queries  FME treats Views like normal Tables  Allows you simplify your Workspace Materialized Views (Oracle, DB2, MS SQL Server, PostgreSQL)  Results of a Query stored on disk
  28. 28. Database Joins  There’s lots of options for carrying out Joins (not limited to databases)  Joiner vs FeatureMerger  http://evangelism.safe.com/fmeevangelist79/  FME 2012 introduced the InlineQuerier  Cache data to disk mid-process (SQLite)  Write SQL against non-database formats  http://evangelism.safe.com/fmeevangelist97/Transformation Challenges
  29. 29. Thank You! For more information, contact:  fme@1spatial.com  +44 (0)1223 420414  @david_eagle  @1spatial  Web resource:  www.1spatial.com/fme Over to you Dale!
  30. 30. The Founder – Dale LutzMultiprocessing - Faster than Ever
  31. 31. Multiprocessing: Definition
  32. 32. Multiprocessing: Definition Multiprocessing is the use oftwo or more central processing units (CPUs) within a single computer system - Wikipedia
  33. 33. Multiprocessing: Motivation Multi-Core and HyperThreaded CPUs  Very common  Potential to increase compute-bound throughput
  34. 34. Multi-Processing:Sample Performance Boosts 3D Clipping of Point Clouds  2 X faster LineJoiner and Buffer  2 X faster Raster Cell Coercer/Point on Raster  4 X faster
  35. 35. Multiprocessing & FME Option on Many Transformers Multiprocesses Groups
  36. 36. Multiprocessing & FME
  37. 37. Multiprocessing & FME Option on All Custom Transformers
  38. 38. Multiprocessing: Levels Of Parallelism None Minimal Moderate Aggressive Extreme
  39. 39. Multiprocessing: Levels Of ParallelismParameter ProcessesNo 1 License CapParallelism Base 4Minimal Cores / 2 Professional 8Moderate Cores Other 16Aggressive Cores * 1.5Extreme Cores * 2
  40. 40. Multiprocessing: Summary New in FME 2012 Leverages Multi-Core and HyperThreading CPUs Controlled by:  Levels of Parallelism  Group-By Can give significant performance boost http://fmeevangelist.com/fmeevangelist94 http://fme.ly/parallel (FMEpedia Article)
  41. 41. Multiprocessing DEMO
  42. 42. Share Today’s Webinar Today’s webinar was recorded View our archive at:  http://fme.ly/archive
  43. 43. Special Offer! Win Free Training! View the offerings at:  http://safe.com/training APRIL
  44. 44. Thank You! For more information contact:  Safe Partners  http://www.safe.com/partners/find-a-sales-partner  Safe Sales:  sales@safe.com  www.FMEPedia.com  www.safe.com/support  Dale  David @DaleAtSafe @david_eagle  Mark  Ken @FMEEvangelist @KenatSafe

×