SlideShare a Scribd company logo
1 of 18
Download to read offline
CONNECT. TRANSFORM. AUTOMATE.
Managing Data
Synchronization
Jerrod Stutzman
GIS Platform Engineer – Devon Energy
Project Goal
CONNECT. TRANSFORM. AUTOMATE.
!  Create a centralized spatial system for storing
and displaying SCADA (supervisory control and
data acquisition) data
!  View data on map via desktop and mobile
!  Google-like search and performance
Challenges
!  Need to see consistent data on desktop web app
and custom iPhone app
!  Location information from SCADA database is
inconsistent and unreliable – need to join with
reliable data for mapping
!  Need to show on map along with other Devon
assets (wells, pipelines, leases, etc)
!  ArcSDE is authoritative spatial database
!  Won’t be used if it isn’t fast!
Solution
!  Spatial Reasoning System (SRS)
!  OpenGeo Suite (POSTGIS data served via
GeoServer using GeoWebCache)
!  Apache SOLR search platform
!  Internally developed web and mobile apps
!  FME & FME Server for data creation and
synchronization
!  Custom code for certain integrations
Why OpenSource
!  PERFORMANCE
!  Biggest pain point historically is our well dataset.
Over 4 million features + 3 geometries (surface
location, bottom location, directional path)
!  POSTGIS + GeoServer render and cache this
dataset quite fast
!  POSTGIS can store all 3 geometries on one row!
!  Reduced database complexity
!  Low cost
!  FME is used within the SRS
to create and sync data for
wells, pipelines, leases,
seismic, facilities, etc
!  FME Server handles
scheduling and failure
reporting (email)
SRS Integrations Diagram
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
!  SCADA Radio Tower data is maintained in Excel
!  Updated quarterly. Full dataset update
!  GPS Location and Range need to be shown
!  Need to store 1 to 3 geometries for each tower
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 2
Non-spatial Oracle to POSTGIS
Selected Workbench Example 2
Non-Spatial Oracle to POSTGIS
!  Non-spatial well production data from Oracle DB
is synchronized with POSTGIS
!  Workflow:
Retrieve last
run time from
POSTGIS
Query
ORACLE for
changes
Query
POSTGIS for
matches
Compare to
determine
INSERT/UPDATE
Determine
DELETES
Update
POSTGIS
DB triggers
update
SOLR
Selected Workbench Example 2
Non-Spatial Oracle to POSTGIS
Selected Workbench Example 3
Seismic Data from Oracle DB
Selected Workbench Example 3
Seismic Data from Oracle DB
!  Seismic data can be represented spatially as lines
or polygons
!  This is stored in the seismic DB as a coordinate list
of vertices (numbers only, non-spatial DB)
!  This data needs to be ‘spatialized’ and stored in a
spatial database – synced nightly (incremental
updates)
!  Complexity Reduction – store all data (2 different
geometry types) in same table
Selected Workbench Example 3
Seismic Data from Oracle DB
Result
Future
!  Ability to do an “UPSERT” DB transaction would
greatly simplify some of the logic required to
determine DB transaction!
!  ARCSDE30 Writer actually had this capability, if it
did not find the record to update, it would insert
!  Need to revise old workbenches to use best
practices and gain efficiencies
!  Scaling SRS to handle more data and users
Thank You!
!  Questions?
!  For more information:
!  Jerrod Stutzman
!  Jerrod.Stutzman@dvn.com
!  Devon Energy Corporation
CONNECT. TRANSFORM. AUTOMATE.

More Related Content

What's hot

FME 2013 Based Spatial Export As A Service
FME 2013 Based Spatial Export As A ServiceFME 2013 Based Spatial Export As A Service
FME 2013 Based Spatial Export As A ServiceSafe Software
 
A Study on New York City Taxi Rides
A Study on New York City Taxi RidesA Study on New York City Taxi Rides
A Study on New York City Taxi RidesCaglar Subasi
 
WHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0aWHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0aEdward Kearns
 
NDGISUC2017 - Mobile Data Collection & Reporting System
NDGISUC2017 - Mobile Data Collection & Reporting SystemNDGISUC2017 - Mobile Data Collection & Reporting System
NDGISUC2017 - Mobile Data Collection & Reporting SystemNorth Dakota GIS Hub
 
AWS EMR (Elastic Map Reduce) explained
AWS EMR (Elastic Map Reduce) explainedAWS EMR (Elastic Map Reduce) explained
AWS EMR (Elastic Map Reduce) explainedHarsha KM
 
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonoughFME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonoughIMGS
 
Streaming API, Spark and Ruby
Streaming API, Spark and RubyStreaming API, Spark and Ruby
Streaming API, Spark and RubyManohar Amrutkar
 
Apache spark vs hadoop
Apache spark vs hadoopApache spark vs hadoop
Apache spark vs hadoopPrwaTech
 
06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clustering06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clusteringSubhas Kumar Ghosh
 
Low cost solutions for Lidar and GIS analysis
Low cost solutions for Lidar and GIS analysisLow cost solutions for Lidar and GIS analysis
Low cost solutions for Lidar and GIS analysisFungis Queensland
 
Mapreduce introduction
Mapreduce introductionMapreduce introduction
Mapreduce introductionYogender Singh
 
FME World Tour 2016: Your Data in Motion (Safe Software)
FME World Tour 2016: Your Data in Motion (Safe Software)FME World Tour 2016: Your Data in Motion (Safe Software)
FME World Tour 2016: Your Data in Motion (Safe Software)GIM_nv
 
Scalable Data Analytics and Visualization with Cloud Optimized Services
Scalable Data Analytics and Visualization with Cloud Optimized ServicesScalable Data Analytics and Visualization with Cloud Optimized Services
Scalable Data Analytics and Visualization with Cloud Optimized ServicesGlobus
 
developmentSEED Presentation for Earth Observation in the Cloud Demo Day
developmentSEED Presentation for Earth Observation in the Cloud Demo DaydevelopmentSEED Presentation for Earth Observation in the Cloud Demo Day
developmentSEED Presentation for Earth Observation in the Cloud Demo DayAmazon Web Services
 
ESIP 2018 - The Case for Archives of Convenience
ESIP 2018 - The Case for Archives of ConvenienceESIP 2018 - The Case for Archives of Convenience
ESIP 2018 - The Case for Archives of ConvenienceDan Pilone
 
Hw09 Making Hadoop Easy On Amazon Web Services
Hw09   Making Hadoop Easy On Amazon Web ServicesHw09   Making Hadoop Easy On Amazon Web Services
Hw09 Making Hadoop Easy On Amazon Web ServicesCloudera, Inc.
 
CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the CloudCloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the CloudAnkur Dave
 

What's hot (20)

FME 2013 Based Spatial Export As A Service
FME 2013 Based Spatial Export As A ServiceFME 2013 Based Spatial Export As A Service
FME 2013 Based Spatial Export As A Service
 
Plantpredict: Solar Performance Modeling Made Simple
Plantpredict: Solar Performance Modeling Made SimplePlantpredict: Solar Performance Modeling Made Simple
Plantpredict: Solar Performance Modeling Made Simple
 
A Study on New York City Taxi Rides
A Study on New York City Taxi RidesA Study on New York City Taxi Rides
A Study on New York City Taxi Rides
 
WHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0aWHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0a
 
NDGISUC2017 - Mobile Data Collection & Reporting System
NDGISUC2017 - Mobile Data Collection & Reporting SystemNDGISUC2017 - Mobile Data Collection & Reporting System
NDGISUC2017 - Mobile Data Collection & Reporting System
 
AWS EMR (Elastic Map Reduce) explained
AWS EMR (Elastic Map Reduce) explainedAWS EMR (Elastic Map Reduce) explained
AWS EMR (Elastic Map Reduce) explained
 
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonoughFME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
 
Streaming API, Spark and Ruby
Streaming API, Spark and RubyStreaming API, Spark and Ruby
Streaming API, Spark and Ruby
 
Apache spark vs hadoop
Apache spark vs hadoopApache spark vs hadoop
Apache spark vs hadoop
 
06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clustering06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clustering
 
Low cost solutions for Lidar and GIS analysis
Low cost solutions for Lidar and GIS analysisLow cost solutions for Lidar and GIS analysis
Low cost solutions for Lidar and GIS analysis
 
Llnl talk
Llnl talkLlnl talk
Llnl talk
 
Mapreduce introduction
Mapreduce introductionMapreduce introduction
Mapreduce introduction
 
FME World Tour 2016: Your Data in Motion (Safe Software)
FME World Tour 2016: Your Data in Motion (Safe Software)FME World Tour 2016: Your Data in Motion (Safe Software)
FME World Tour 2016: Your Data in Motion (Safe Software)
 
Scalable Data Analytics and Visualization with Cloud Optimized Services
Scalable Data Analytics and Visualization with Cloud Optimized ServicesScalable Data Analytics and Visualization with Cloud Optimized Services
Scalable Data Analytics and Visualization with Cloud Optimized Services
 
developmentSEED Presentation for Earth Observation in the Cloud Demo Day
developmentSEED Presentation for Earth Observation in the Cloud Demo DaydevelopmentSEED Presentation for Earth Observation in the Cloud Demo Day
developmentSEED Presentation for Earth Observation in the Cloud Demo Day
 
AAPG Geoscience Technology Workshop 2019
AAPG Geoscience Technology Workshop 2019AAPG Geoscience Technology Workshop 2019
AAPG Geoscience Technology Workshop 2019
 
ESIP 2018 - The Case for Archives of Convenience
ESIP 2018 - The Case for Archives of ConvenienceESIP 2018 - The Case for Archives of Convenience
ESIP 2018 - The Case for Archives of Convenience
 
Hw09 Making Hadoop Easy On Amazon Web Services
Hw09   Making Hadoop Easy On Amazon Web ServicesHw09   Making Hadoop Easy On Amazon Web Services
Hw09 Making Hadoop Easy On Amazon Web Services
 
CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the CloudCloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
 

Similar to Managing Data Synchronization Between ArcSDE and POSTGIS using FME

Open Source Databases And Gis
Open Source Databases And GisOpen Source Databases And Gis
Open Source Databases And GisKudos S.A.S
 
Hot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark frameworkHot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark frameworkSupriya .
 
Dev Summit 2011 - Talk
Dev Summit 2011 - TalkDev Summit 2011 - Talk
Dev Summit 2011 - TalkArc2Earth
 
2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)Rudolf Husar
 
Stsg17 speaker yousunjeong
Stsg17 speaker yousunjeongStsg17 speaker yousunjeong
Stsg17 speaker yousunjeongYousun Jeong
 
GeoKettle: A powerful open source spatial ETL tool
GeoKettle: A powerful open source spatial ETL toolGeoKettle: A powerful open source spatial ETL tool
GeoKettle: A powerful open source spatial ETL toolThierry Badard
 
Building a Spatial Database in PostgreSQL
Building a Spatial Database in PostgreSQLBuilding a Spatial Database in PostgreSQL
Building a Spatial Database in PostgreSQLKudos S.A.S
 
SPL_ALL_EN.pptx
SPL_ALL_EN.pptxSPL_ALL_EN.pptx
SPL_ALL_EN.pptx政宏 张
 
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...IRJET Journal
 
Azure Cosmos DB - Technical Deep Dive
Azure Cosmos DB - Technical Deep DiveAzure Cosmos DB - Technical Deep Dive
Azure Cosmos DB - Technical Deep DiveAndre Essing
 
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...Andre Essing
 
226 team project-report-manjula kollipara
226 team project-report-manjula kollipara226 team project-report-manjula kollipara
226 team project-report-manjula kolliparaManjula Kollipara
 
Leicester 2010 notes
Leicester 2010 notesLeicester 2010 notes
Leicester 2010 notesJoanne Cook
 
NoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQLNoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQLEDB
 
Data Warehouse Offload
Data Warehouse OffloadData Warehouse Offload
Data Warehouse OffloadJohn Berns
 
EEDC - Apache Pig
EEDC - Apache PigEEDC - Apache Pig
EEDC - Apache Pigjavicid
 
Real-Time Analytics with Spark and MemSQL
Real-Time Analytics with Spark and MemSQLReal-Time Analytics with Spark and MemSQL
Real-Time Analytics with Spark and MemSQLSingleStore
 

Similar to Managing Data Synchronization Between ArcSDE and POSTGIS using FME (20)

Open Source Databases And Gis
Open Source Databases And GisOpen Source Databases And Gis
Open Source Databases And Gis
 
Hot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark frameworkHot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark framework
 
Dev Summit 2011 - Talk
Dev Summit 2011 - TalkDev Summit 2011 - Talk
Dev Summit 2011 - Talk
 
2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)
 
Srds Pres011120
Srds Pres011120Srds Pres011120
Srds Pres011120
 
Stsg17 speaker yousunjeong
Stsg17 speaker yousunjeongStsg17 speaker yousunjeong
Stsg17 speaker yousunjeong
 
GeoKettle: A powerful open source spatial ETL tool
GeoKettle: A powerful open source spatial ETL toolGeoKettle: A powerful open source spatial ETL tool
GeoKettle: A powerful open source spatial ETL tool
 
Building a Spatial Database in PostgreSQL
Building a Spatial Database in PostgreSQLBuilding a Spatial Database in PostgreSQL
Building a Spatial Database in PostgreSQL
 
SPL_ALL_EN.pptx
SPL_ALL_EN.pptxSPL_ALL_EN.pptx
SPL_ALL_EN.pptx
 
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
 
Azure Cosmos DB - Technical Deep Dive
Azure Cosmos DB - Technical Deep DiveAzure Cosmos DB - Technical Deep Dive
Azure Cosmos DB - Technical Deep Dive
 
ADAPTER
ADAPTERADAPTER
ADAPTER
 
Eedc.apache.pig last
Eedc.apache.pig lastEedc.apache.pig last
Eedc.apache.pig last
 
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
 
226 team project-report-manjula kollipara
226 team project-report-manjula kollipara226 team project-report-manjula kollipara
226 team project-report-manjula kollipara
 
Leicester 2010 notes
Leicester 2010 notesLeicester 2010 notes
Leicester 2010 notes
 
NoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQLNoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQL
 
Data Warehouse Offload
Data Warehouse OffloadData Warehouse Offload
Data Warehouse Offload
 
EEDC - Apache Pig
EEDC - Apache PigEEDC - Apache Pig
EEDC - Apache Pig
 
Real-Time Analytics with Spark and MemSQL
Real-Time Analytics with Spark and MemSQLReal-Time Analytics with Spark and MemSQL
Real-Time Analytics with Spark and MemSQL
 

More from Safe Software

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemSafe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 

More from Safe Software (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 

Recently uploaded

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024The Digital Insurer
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesBoston Institute of Analytics
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 

Recently uploaded (20)

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 

Managing Data Synchronization Between ArcSDE and POSTGIS using FME

  • 1. CONNECT. TRANSFORM. AUTOMATE. Managing Data Synchronization Jerrod Stutzman GIS Platform Engineer – Devon Energy
  • 2. Project Goal CONNECT. TRANSFORM. AUTOMATE. !  Create a centralized spatial system for storing and displaying SCADA (supervisory control and data acquisition) data !  View data on map via desktop and mobile !  Google-like search and performance
  • 3. Challenges !  Need to see consistent data on desktop web app and custom iPhone app !  Location information from SCADA database is inconsistent and unreliable – need to join with reliable data for mapping !  Need to show on map along with other Devon assets (wells, pipelines, leases, etc) !  ArcSDE is authoritative spatial database !  Won’t be used if it isn’t fast!
  • 4. Solution !  Spatial Reasoning System (SRS) !  OpenGeo Suite (POSTGIS data served via GeoServer using GeoWebCache) !  Apache SOLR search platform !  Internally developed web and mobile apps !  FME & FME Server for data creation and synchronization !  Custom code for certain integrations
  • 5. Why OpenSource !  PERFORMANCE !  Biggest pain point historically is our well dataset. Over 4 million features + 3 geometries (surface location, bottom location, directional path) !  POSTGIS + GeoServer render and cache this dataset quite fast !  POSTGIS can store all 3 geometries on one row! !  Reduced database complexity !  Low cost
  • 6. !  FME is used within the SRS to create and sync data for wells, pipelines, leases, seismic, facilities, etc !  FME Server handles scheduling and failure reporting (email) SRS Integrations Diagram
  • 7. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
  • 8. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS !  SCADA Radio Tower data is maintained in Excel !  Updated quarterly. Full dataset update !  GPS Location and Range need to be shown !  Need to store 1 to 3 geometries for each tower
  • 9. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
  • 10. Selected Workbench Example 2 Non-spatial Oracle to POSTGIS
  • 11. Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS !  Non-spatial well production data from Oracle DB is synchronized with POSTGIS !  Workflow: Retrieve last run time from POSTGIS Query ORACLE for changes Query POSTGIS for matches Compare to determine INSERT/UPDATE Determine DELETES Update POSTGIS DB triggers update SOLR
  • 12. Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS
  • 13. Selected Workbench Example 3 Seismic Data from Oracle DB
  • 14. Selected Workbench Example 3 Seismic Data from Oracle DB !  Seismic data can be represented spatially as lines or polygons !  This is stored in the seismic DB as a coordinate list of vertices (numbers only, non-spatial DB) !  This data needs to be ‘spatialized’ and stored in a spatial database – synced nightly (incremental updates) !  Complexity Reduction – store all data (2 different geometry types) in same table
  • 15. Selected Workbench Example 3 Seismic Data from Oracle DB
  • 17. Future !  Ability to do an “UPSERT” DB transaction would greatly simplify some of the logic required to determine DB transaction! !  ARCSDE30 Writer actually had this capability, if it did not find the record to update, it would insert !  Need to revise old workbenches to use best practices and gain efficiencies !  Scaling SRS to handle more data and users
  • 18. Thank You! !  Questions? !  For more information: !  Jerrod Stutzman !  Jerrod.Stutzman@dvn.com !  Devon Energy Corporation CONNECT. TRANSFORM. AUTOMATE.