CONNECT. TRANSFORM. AUTOMATE.
Managing Data
Synchronization
Jerrod Stutzman
GIS Platform Engineer – Devon Energy
Project Goal
CONNECT. TRANSFORM. AUTOMATE.
!  Create a centralized spatial system for storing
and displaying SCADA (superv...
Challenges
!  Need to see consistent data on desktop web app
and custom iPhone app
!  Location information from SCADA data...
Solution
!  Spatial Reasoning System (SRS)
!  OpenGeo Suite (POSTGIS data served via
GeoServer using GeoWebCache)
!  Apach...
Why OpenSource
!  PERFORMANCE
!  Biggest pain point historically is our well dataset.
Over 4 million features + 3 geometri...
!  FME is used within the SRS
to create and sync data for
wells, pipelines, leases,
seismic, facilities, etc
!  FME Server...
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
!  SCADA Radio Tower data is maintained in Excel
!  Updated quar...
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 2
Non-spatial Oracle to POSTGIS
Selected Workbench Example 2
Non-Spatial Oracle to POSTGIS
!  Non-spatial well production data from Oracle DB
is synchroni...
Selected Workbench Example 2
Non-Spatial Oracle to POSTGIS
Selected Workbench Example 3
Seismic Data from Oracle DB
Selected Workbench Example 3
Seismic Data from Oracle DB
!  Seismic data can be represented spatially as lines
or polygons...
Selected Workbench Example 3
Seismic Data from Oracle DB
Result
Future
!  Ability to do an “UPSERT” DB transaction would
greatly simplify some of the logic required to
determine DB trans...
Thank You!
!  Questions?
!  For more information:
!  Jerrod Stutzman
!  Jerrod.Stutzman@dvn.com
!  Devon Energy Corporatio...
Upcoming SlideShare
Loading in …5
×

Managing Data Synchronization Between ArcSDE and POSTGIS using FME

2,412 views

Published on

Devon Energy recently adopted POSTGIS as a data store for data served via web maps.  All corporate data is typically edited and maintained in ArcSDE.  This presentation focuses on the many ways FME can be used to manage transactional change and keep databases in sync, especially for tricky datasets.

See more presentations from the FME User Conference 2014 at: www.safe.com/fmeuc

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,412
On SlideShare
0
From Embeds
0
Number of Embeds
809
Actions
Shares
0
Downloads
26
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Managing Data Synchronization Between ArcSDE and POSTGIS using FME

  1. 1. CONNECT. TRANSFORM. AUTOMATE. Managing Data Synchronization Jerrod Stutzman GIS Platform Engineer – Devon Energy
  2. 2. Project Goal CONNECT. TRANSFORM. AUTOMATE. !  Create a centralized spatial system for storing and displaying SCADA (supervisory control and data acquisition) data !  View data on map via desktop and mobile !  Google-like search and performance
  3. 3. Challenges !  Need to see consistent data on desktop web app and custom iPhone app !  Location information from SCADA database is inconsistent and unreliable – need to join with reliable data for mapping !  Need to show on map along with other Devon assets (wells, pipelines, leases, etc) !  ArcSDE is authoritative spatial database !  Won’t be used if it isn’t fast!
  4. 4. Solution !  Spatial Reasoning System (SRS) !  OpenGeo Suite (POSTGIS data served via GeoServer using GeoWebCache) !  Apache SOLR search platform !  Internally developed web and mobile apps !  FME & FME Server for data creation and synchronization !  Custom code for certain integrations
  5. 5. Why OpenSource !  PERFORMANCE !  Biggest pain point historically is our well dataset. Over 4 million features + 3 geometries (surface location, bottom location, directional path) !  POSTGIS + GeoServer render and cache this dataset quite fast !  POSTGIS can store all 3 geometries on one row! !  Reduced database complexity !  Low cost
  6. 6. !  FME is used within the SRS to create and sync data for wells, pipelines, leases, seismic, facilities, etc !  FME Server handles scheduling and failure reporting (email) SRS Integrations Diagram
  7. 7. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
  8. 8. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS !  SCADA Radio Tower data is maintained in Excel !  Updated quarterly. Full dataset update !  GPS Location and Range need to be shown !  Need to store 1 to 3 geometries for each tower
  9. 9. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
  10. 10. Selected Workbench Example 2 Non-spatial Oracle to POSTGIS
  11. 11. Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS !  Non-spatial well production data from Oracle DB is synchronized with POSTGIS !  Workflow: Retrieve last run time from POSTGIS Query ORACLE for changes Query POSTGIS for matches Compare to determine INSERT/UPDATE Determine DELETES Update POSTGIS DB triggers update SOLR
  12. 12. Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS
  13. 13. Selected Workbench Example 3 Seismic Data from Oracle DB
  14. 14. Selected Workbench Example 3 Seismic Data from Oracle DB !  Seismic data can be represented spatially as lines or polygons !  This is stored in the seismic DB as a coordinate list of vertices (numbers only, non-spatial DB) !  This data needs to be ‘spatialized’ and stored in a spatial database – synced nightly (incremental updates) !  Complexity Reduction – store all data (2 different geometry types) in same table
  15. 15. Selected Workbench Example 3 Seismic Data from Oracle DB
  16. 16. Result
  17. 17. Future !  Ability to do an “UPSERT” DB transaction would greatly simplify some of the logic required to determine DB transaction! !  ARCSDE30 Writer actually had this capability, if it did not find the record to update, it would insert !  Need to revise old workbenches to use best practices and gain efficiencies !  Scaling SRS to handle more data and users
  18. 18. Thank You! !  Questions? !  For more information: !  Jerrod Stutzman !  Jerrod.Stutzman@dvn.com !  Devon Energy Corporation CONNECT. TRANSFORM. AUTOMATE.

×