Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using Data Integration to Deliver Intelligence to Anyone, Anywhere


Published on

Data integration makes it possible to deliver intelligence and keep decision makers, first responders, and civilians informed. For over 20 years, FME has been trusted by federal governments to move data from nearly any source to the target destination, while saving time and budget resources.

With FME, federal governments can deliver open data, improve emergency & disaster response, enhance land management, turn public safety and defense into actionable results, and integrate & deliver location intelligence.

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

Using Data Integration to Deliver Intelligence to Anyone, Anywhere

  1. 1. Using Data Integration to Deliver Intelligence to Anyone, Anywhere Craig Hantke, Dean Hintz
  2. 2. Dean Hintz Senior Applications Analyst Meet the experts Craig Hantke Account Manager Use the GoToWebinar Control Panel to chat in questions Erin Lemky Senior Product Marketing Manager
  3. 3. “The world’s most valuable resource is no longer oil, but data.” THE ECONOMIST
  4. 4. Lightning Demo - Fire Notifications
  5. 5. ● 66% of enterprises rank Location Intelligence as either critical or very important to ongoing revenue growth strategies (Forbes) ● Gain insight into your key assets & security risks ● Integrate geospatial into your intelligence Importance of Location
  6. 6. AGENDA 01. 02. 03. 04. 05. 06. 07. Data Challenges About Safe Software Disaster Response Public Safety, Defense, Hazard Mapping Indoor Mapping Resources Q&A Send questions into chat or We also have live chat at
  7. 7. Data Silos 7
  8. 8. SOLUTION: DATA INTEGRATION “The discipline of data integration comprises the practices, architectural techniques and tools for achieving the consistent access and delivery of data across the spectrum of data subject areas and data structure types in the enterprise to meet the data consumption requirements of all applications and business processes.” - Gartner
  9. 9. Years of solving data challenges 25 Safe So ware COMPANY PROFILE 10,000 Organizations trusting us worldwide Our mission is to help you maximize the value of your data. Partners supporting our network 150 128 Countries with FME customers
  10. 10. CUSTOMERS
  11. 11. Connect Your Data Sources Transform Your Data Automate Your Workflows
  12. 12. CONNECT CAD GIS FME release by year NumberofsupportedformatsinFME 1995 10500100300 20192015201020052000 TABULAR DATABASE RASTER POINT CLOUD BIM 3D WEB XML CLOUD BIG DATA IOT BI AR / VR AI / ML INDOOR MAPPING GAMING
  13. 13. Chat Question: What is your most pressing integration or data fusion need?
  14. 14. FME and Intelligence: Conceptual Approach There are several key areas where FME can support deriving and delivering key intelligence resources to your decision makers: ● Extraction / collection ● Integration / fusion ● Filtering / cleaning ● Enrichment / analysis ● Distribution With all the above workflows, the key to scalability of both volume and complexity is automation
  15. 15. Disaster Response
  16. 16. Plan Mitigate Respond Recover Integration Disaster Management Cycle
  17. 17. CUSTOMER STORY “We love FME. We’ve been using it for about 20 years.” - Piet Nooij, Fortis BC PROJECT Assess the current wildfire threat to assets. SOLUTION Integrate active wildfire data from provincial government with their GIS. RESULTS ● Workflow automatically runs at same interval as source dataset updates. ● Notifications & reports are immediately sent to Operations Managers who can coordinate with Emergency Services. FORTIS BC >
  18. 18. Demo
  19. 19. Demo: Vancouver Coastal Flood Hazard Assessment Hazard = Criticality * Severity hazard = criticality * impact severity (* probability for risk - not included here)
  20. 20. Vancouver Flood Hazard: Input DEM Hazard = Criticality * Severity
  21. 21. Flood Hazard: FloodAreaExtractor 1. Subtract flood level from each pixel / cell 2. Classify cells by flood severity 3. Convert severity levels to vector areas
  22. 22. Vancouver Flood Hazard Assessment: Result
  23. 23. Powerlink is a leading Australian provider of high voltage electricity transmission network services, combining innovation with insight to deliver safe, cost effective and reliable solutions. About Powerlink The Emergency Readiness Project Team Mark West – Manager of Technical Information Services Aaron Ambler – Information & Services Team Leader Alexei Radun – Data & Systems Team Leader Matthew Cooper – Information Analyst John Mockler – Information Analyst Tim Mackay – Spatial Coordinator Ron Innes – Project Server Coordinator Kyle Morris – Information Support Specialist John Marshall – FME Developer A Queensland Government Owned Corporation that owns, develops, operates and maintains the transmission network that extends 1,700km from north of Cairns to the New South Wales border, and comprises 15,337 circuit kilometres of transmission lines and 140 substations.
  24. 24. Emergency Spatial Journey 2019 2018 2017 2015 2011 2013 Pre 2010 Fully Automated, Data driven approach to Emergency awareness Optimised data & information management Expanded data access to increase reliability Full automation of data downloads FME Server Introduced manual reactive mapping & data retrieval & creation No internal emergency data or Information Support Development of FME mapping automation Patent Pendin g
  25. 25. 1010 010 11 Download Data System Methodology Faults Products Event End Rationalise Data to interest Areas Stack & Store Event Start Analyse by business rules Emai lSM S Notification Mapping Reporting
  26. 26. Bushfire Workflows Fire started Analysis Risk levels Bushfire Low Risk Bushfire Medium Risk Bushfire High Risk Gas Wells QRFS Planned Burns QRFS Sentinel >=50 Power and >=80 Confidence QRFS Sentinel – all Sentinel less than business criteria Gas Wells Sentinel QRFS Planned Burns Legend PQ Easement Buffered Area
  27. 27. BOM JTWC BOM Cyclone Workflows BOM Responsible Area FIJI Responsible Area TL Early Warning Entered Australian Region Cyclone Preparedness Cyclone Monitor Stage 1 & 2 Active Impact Post Event Damaging Destructive Very Destructive Wind Areas Asset Wind Rating Analysis with Wind Areas FIJI 4 Predictive Analysis 4 4 TL
  28. 28. Flood Workflows Flood Plain Minor Moderate Major Moderate Alert Area Minor Moderate Major BOM River Gauging Stations River River Catchment
  29. 29. Mapping Products
  30. 30. Email SMS Notifications FME Server Customers On Call Staff Email SMS
  31. 31. Summary FME Server Average 2,500 jobs/day Max 3,500 jobs/day addition due to emergency event Emergency Workflows = 50% of scheduled load Service Now Emai l SM S Notifications sent 3,500 past year Mapping Products 8,000 past year Faults 900 past year Workflows 52 In total What was days of effort now typically down to hours per event
  32. 32. Poll Question: What sector are you in?
  33. 33. Public Safety / Defense
  34. 34. EU INSPIRE: Related Themes •Addresses •Administrative Areas •Networks: Pipelines, electricity •Transportation •Hydrography •Elevation •Natural Risk Zones •Environmental monitoring •Land use •Land cover •Ortho imagery •Buildings Data Integration Example
  35. 35. Integration: EU INSPIRE • Common, open data standard (complex schema) • Multi-domain, multi-agency, & trans-national • OGC open standards based data and services • OGC formats: GML, WFS, WMS, WCS, GeoTIFF, NetCDF
  36. 36. Public Safety / Defense ● Remote Sensing
  37. 37. Satellite and UAV Remote Sensing: Common Tasks Workflows and requirements for Satellite and UAV ● Sensor platforms ● Raster data types ● Selection, conversion ● Enhancement, classification ● Preparation, enrichment, distribution ● UAV specific workflows
  38. 38. Remote Sensing Platforms ● Satellite ● Airborne ● UAVs - long range (BVLOS) ● UAVs - small ● Terrestrial
  39. 39. ● Raster imagery ● RADAR / SAR ● LIDAR ● Video Data types
  40. 40. Satellite Imagery: Cloud Sources Landsat (NASA/USGS) Sentinel 2 (ESA) Planet UrtheCast (Future) Resolution 15/30/100m 10/20/60m 3-5m* 0.5 - 5 m Status Free Free Paid Paid Frequency Monthly Weekly (or better) Daily A few times a day
  41. 41. Demo
  42. 42. Example: Landsat and Sentinel Selector ● User selects location of interest, platform and cloud cover ● Returns list of available tiles that meet criteria ● User accesses results on Amazon S3
  43. 43. Example: Landsat and Sentinel Selector
  44. 44. White Rock Pier Storm Dec 20, 2018 ● One of the worst storms in BC Hydro history ● JRCC coordinated rescue from collapsed pier (CAF, RCMSAR)
  45. 45. Damage Assessment - WR Pier: Open Drone Map GeoTIFF UAV survey JPGs to mosaicked, georeferenced GeoTIFF orthophoto using FME & ODM docker
  46. 46. Damage Assessment - WR Pier: Open Drone Map LAS UAV survey JPGs to georeferenced 3D LAS point cloud using FME & ODM docker
  47. 47. OpenDroneMapCaller, ODMMosaicker Global Medic use case: problem processing 500 images offline FME OpenDroneMapCaller from FMEHub (FME calls ODM on docker) ● Compute approximate centroid of each tile from exif tags ● Copy source tiles into folder by tile name ● Call ODM on docker for each folder to generate tile GeoTIFF ● Use FME to mosaic georeferenced ODM tiles into one large mosaic Group source images by exif location
  48. 48. Global Medic: RescUAV Data Processing and Management
  49. 49. Poll Question: Which resources do you primarily rely on for integration today?
  50. 50. Public Safety / Defense ● Data filtering / AI
  51. 51. Integrating AI to Support SAR Intelligence Automation ● Data enrichment to support search and rescue (SAR) operations ● MS Cognitive Services API - ID target type with confidence metrics ● FME allows for easy integration with third party systems - in this case via REST (see FMEHub for other web connections) ● Originally presented at 2019 World Maritime Rescue Congress FME Workflow: ● automates the feed of UAV imagery to Cognitive Services API ● filters results and compiles reports showing targets of interest, with levels of confidence
  52. 52. AI: Vision Analysis - MS Cognitive Services API FME Workspace reads directory of images and calls REST API (HTTPCaller) for each
  53. 53. Test Results: Filter Configuration
  54. 54. Computer Vision Analysis: Cognative Services API
  55. 55. Computer Vision: SAR Target Search
  56. 56. Computer Vision: SAR Target Search
  57. 57. Public Safety / Defense ● Data Management
  58. 58. CUSTOMER STORY “FME Server plays an important role in scheduling processes, publishing data, and process monitoring.” - Yann Rebois, ICRC PROJECT Integrate disparate datasets from internal departments and external sources, both globally and locally. SOLUTION Implemented FME within existing GIS architecture to enable data integration, management, and automation. RESULTS ● Automate and simplify complex workflows. ● Feed data to Tableau & ArcGIS. ● Publish data to web applications via MapBox, CartoDB & PDF reports. ICRC INTERNATIONAL COMMITTEE OF THE RED CROSS >
  59. 59. “... to protect the lives and dignity of victims of war and internal violence and to provide them with assistance.”
  60. 60. “GIS officers are based worldwide, need to get information on a daily basis, and be able to use processes without having any FME technical knowledge.” – Régis Longchamp, INSER
  61. 61. ● Data management ○ Quantity ○ Quality ○ Heterogeneity ○ Applications ● Security policies ● Global scope Challenges
  62. 62. 6 Jobs for FME at ICRC 1. Integrating source data 2. Data cleaning 3. Simplifying geometry 4. Site reporting 5. Supplying data to Tableau 6. Gazetteer – Population Kiosk
  63. 63. #1 Integrating Source Data Integrating source data from 3rd-party web services etc. Examples: 1. Free & open healthcare location data. 2. ACLED – armed conflict location & event data.
  64. 64. #2 Data Cleaning Mixed data needs to be harmonized and cleaned. Types of cleaning done by FME: ● ArcSDEGridSapper ● Self-intersection ● Minimum area ● OGC tests ● ArcPy repair
  65. 65. #3 Simplifying Geometry A single SDE database is behind: ● large scale paper maps ● cross-border maps ● web applications ● Importing to Tableau
  66. 66. #4 Site Reports Essential for safety and security of ICRC teams ● Premises Management web app ● PDF creation for site reports (offline, mobile) ● Raster backgrounds from Google or Esri services, where available
  67. 67. #5 Supplying Data to Tableau Tableau used for real time analysis of operations and programs. ● Preparing spatial data for Tableau ● Joining business data to spatial data
  68. 68. #6 Population Kiosk Reference site for risks to local populations: disease, conflict Helps answers questions like: 1. What is the affected population? 2. In what villages have abuses been committed?
  69. 69. Geospatial Environment FME’s role & key results: ● Integrate diverse range of data sources, including non-GIS (facilities) ● Feed data to Tableau & ArcGIS ● Publish data to web via MapBox, CartoDB & PDF
  70. 70. Key transformers used by Régis of INSER in this work 1. HTTPCaller 2. JSONFlattener 3. PythonCaller 4. WorkspaceRunner & FMEServerJobSubmitter 5. CSMapReprojector
  71. 71. Benefits of FME to ICRC ✓ Flexibility ○ Extract information from any data source as needed ✓ Workflow maintenance ○ Easier than maintaining many python scripts ✓ Versatility ○ Feed other systems for additional analysis and publication “Thank you @inser team and Safe Software for the Grant Program!” -- Yann Rebois ICRC
  72. 72. Chat Question: What are your source and destination systems?
  73. 73. Public Safety / Defense ● Data enrichment
  74. 74. ● Britain’s mapping service for government, businesses, and citizens. ● Geospatial data serves the national interest by enabling a safe, healthy and prosperous society. ● Vision: to ensure Britain can build a world-leading digital and connected economy of the future.
  75. 75. Ordnance Survey: Objectives and Challenges Objectives: ● Provide customers with more detail about Britain’s national landscape and building types ● Automatically detect roof type using deep learning ● Improve public safety, planning (energy, 5G), building management and 3D Challenges: ● Better management of large data volumes ● 20,000 updates / day to 500 million geospatial feature database
  76. 76. Solution 1. Building footprints of all structures in Great Britain 2. Use crowdsourcing and deep learning to classify roof types 3. Add attributes + clip buildings 4. FME adds roof type attributes to polygons and clips buildings from raster satellite imagery Key was to quickly create different patch types, test lots of data for the patches and process patches to the cloud
  77. 77. INPUT Remote Sensing Surveyors and Field Surveyors classify and label building polygons in a crowdsourcing platform.
  78. 78. FME Workspace
  79. 79. FME Workspace
  80. 80. OUTPUT Slope Patch examples DSM Rather than classifying an entire image, “patches” of roofs from each image are generated and classified
  81. 81. RESULTS ● Results on three classes: (hipped, gabled, flat) ● 90% accuracy ● Trained on the geographically diverse data set
  82. 82. “We could have created a python script, but it was quicker and simpler to utilize FME, and saved us money on processing costs. ” -- Charis Doidge, Ordnance Survey “FME was quick to set up and use, which was handy when we had several tests we wanted to run on the patch types.”
  83. 83. Public Safety / Defense ● Hazard mapping
  84. 84. Hazard Assessment: con terra GeoRiskAnalyzer
  85. 85. GeoRiskAnalyzer: Natural Hazards
  86. 86. Hazard Summary Reports
  87. 87. Public Safety / Defense ● Indoor mapping
  88. 88. Indoor Mapping ● Great opportunity ○ bring the blue dot inside ● Important area for public safety ○ responders and civilians ● Massive challenge ○ volume of buildings ○ updates
  89. 89. Indoor Mapping Challenges ● Integrate multiple sources to produce an indoor map. ○ GeoJSON, Revit, IFC, CAD (Autodesk, Bentley), Civil 3D, Esri Geodatabase, databases, CityGML … ● Must clean and transform inconsistent data: schema and geometry. ● Must comply with indoor format specifications, e.g. IMDF, HERE, ArcGIS Indoors, IndoorGML. ○ Strict data models and explicit spatial relationships. ● Venues constantly change => automatic updates ● Logical vs physical areas - security zones ● Public safety - hazard, damaged areas, wall and door materials, real time
  90. 90. Production: Build Indoor Mapping Datasets Using FME ● Convert floor plans and ancillary data into indoor mapping formats. ● Validate against specifications to ensure data meets standards. ● No coding involved. FME workflows are created using a visual interface.
  91. 91. Indoor Mapping: OGC Indoor GML Pilot ● OGC Indoor Pilot sponsored by NIST, Dept of Commerce ● Goal: LIDAR scans -> Indoor Mapping and Navigation ● Responsible for the navigation modeller component ● Consume CityGML Public Safety (PS) ADE and produce IndoorGML PS extension ● Project presented by OGC at recent Dept of Commerce conference
  92. 92. OGC Indoor Pilot: Source IFC
  93. 93. OGC Indoor Pilot: CityGML to IndoorGML Workflow
  94. 94. OGC Indoor Pilot Result: IndoorGML with Navigation
  95. 95. CUSTOMER STORY Improving operations and passenger experience to become a top airport in Europe. PROJECT Unify data across departments into a central GIS database. SOLUTION Integrated asset and infrastructure data, enhanced it, validated it, and distributed it across teams. RESULTS ● Synchronized GIS database. ● Open APIs available to developers. ● Foundation for building digital twin. ● Indoor mapping in Apple Maps (IMDF) and a custom augmented reality app. AMS Amsterdam Airport Schiphol >
  96. 96. Schiphol Airport: Results
  97. 97. ● Converting data for indoor mapping can be a challenge. ● CAD standards help, but more feature info is needed. ● Big win going between standards, e.g. TRIRIGA, BIM to IMDF; CityGML to OGC IndoorGML ● Leverage existing tools, e.g. import for ArcGIS Indoor ● Build a multi-step workflow, enriching indoor data at each step. ● Extend internal data model based on indoor requirements (e.g. doors) Lessons Learned
  98. 98. CUSTOMER STORY “FME is the cherry on top the ice cream sundae that helps bring data together and customize it for your needs.” - David Runneals, Iowa DOT PROJECT Provide road conditions and plow information to the public. SOLUTION Use FME to integrate plow locations, plow cams, and road conditions. RESULTS ● Data is retrieved 1x/min from AWS to populate Oracle database. ● FME automatically delivers a KML file to Windows Azure for AGOL. ● API delivers the data to local TV stations for their on-air software. IOWA DOT >
  99. 99. ● Automation and data integration => scalability ● Rule / AI based filtering and analysis ● Public safety / disaster response - similar patterns across defense, security sectors ● Need for a dynamic, flexible platform that enables rapid innovation ● Enterprise wide service integration to span silos CHALLENGE: INTELLIGENCE BASED DECISIONS SOLUTION: DATA INTEGRATION
  100. 100. FREE RESOURCES WE’RE ALL ABOUT Getting Started Online Courses On Demand Live chat > Tutorials and webinars > Instructor led > Live and hands-on > Video courses & demos > Knowledge Base > "FME sets the standard for support and is the leader by far". - Brad Very helpful and went above and beyond to help find a solution to my request! A+" - Justin "Thank you! Answered all my questions - AGAIN. You folks always seem to have a way of doing that." - Ray
  101. 101. Thank you! Any questions? You can reach us at: ● @SafeSoftware on Twitter ● Live chat and free trial at ● Post on