Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Managing a Spatial Data Warehouse with FME

3,223 views

Published on

Devon Energy recently implemented a Spatial Data Warehouse. This is the authoritative corporate database for geospatial data. FME plays many roles in the maintenance of this database - synchronizing data, quality control, data extraction, spatial analysis, web services, etc. This talk will focus on the many ways FME can streamline spatial data across a large organization.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Managing a Spatial Data Warehouse with FME

  1. 1. Managing a Spatial Data Warehouse with FME Jerrod Stutzman
  2. 2. FLIGHT PLAN ● The Spatial Data Warehouse (SDW) ● How FME drives the SDW ● Self-Serve GIS: Web Services
  3. 3. The Spatial Data Warehouse
  4. 4. The Case for an Enterprise Spatial Data Warehouse (SDW) As the company grew, many separate SDE instances popped up for specific use cases. We used FME to move data then, as we do now. Geo Land Eng Fin EHS HR shape files GDB Web Service SDE 1 SDE 2 SDE 3 SDE 4 (Simplification of integrations) Data Service
  5. 5. SIGNIFICANT ISSUES WITH OLD SETUP • Managing data (8 different SDE instances in 2014) • Duplicated data • Inconsistency in naming conventions, coordinate systems & metadata • Access: Shared local user accounts = issues with account locks • Synchronization between geodatabases
  6. 6. THE SDWWe can • Create & Implement Standards • Control access for individual users • Add standardized metadata • Eliminate redundant datasets • And MUCH MORE! Now there is ONE place to store all spatial data.
  7. 7. Spatial Data Warehouse (SDW) Single source for spatial data: client connections, integrations, ArcGIS services, FME web services, etc. Geo Land Eng Fin EHS HR shape files GDB Web Service SDW (Simplification of integrations) Data Service FME/ArcGIS Web Services User
  8. 8. HOW FME DRIVES THE SDW
  9. 9. FME’s role in the SDW 100s of FME Server jobs manage the following tasks: • Sync and Spatialize • Ensure data quality (QA/QC) • Derive new data • Self-Serve GIS via web services • Download/Update vendor data
  10. 10. Place your screenshot here EXAMPLE Incremental Updates For data that requires high availability and frequent updates
  11. 11. EXAMPLE Incremental Updates Startup Python: ● Before the workbench runs, we want to make sure it isn’t already running. If it is, this job will be canceled. ● We get this information from the FME REST services on FME Server via Python. import requests,json url = 'http://fme.dvn.com/fmerest/v2/transformations/jobs/running?accept=json…' jobName = "WELL_MDM_WC_ALL_SH_P_v4.fmw" response = requests.get(url).json() numJobs = len(response) global runJob runJob = 'Y' jobList = [] if numJobs == 0 or numJobs is False: runJob = 'Y' elif numJobs > 0: i = 0 while i < (numJobs): jobList.append(response[i]['request']['workspacePath'].split('/')[2].strip('"')) i += 1 if jobList.count(jobName) > 1: runJob = 'N'
  12. 12. EXAMPLE Incremental Updates Writing Data: ● First write to landing table, then MERGE update DVN_GIS.WELL_MDM_WC_ALL_SH_PU SET UPDATEID = (NEXT VALUE FOR DVN_GIS.MDMWELLUPDATE); BEGIN TRAN; MERGE DVN_GIS.WELL_MDM_WC_ALL_SH_P T USING DVN_GIS.WELL_MDM_WC_ALL_SH_PU U ON (T.WELLBORE_COMPLETION_DUWI = U.WELLBORE_COMPLETION_DUWI) WHEN NOT MATCHED BY TARGET THEN INSERT [OBJECTID], [WELL…. VALUES U.[UPDATEID],U.[WELL… WHEN MATCHED THEN UPDATE SET T.[WELL…,… = U.[WELL… ,...; COMMIT; Geo WELL_MDM_WC_ALL_SH_PU FME SQL After WELL_MDM_WC_ALL_SH_P TIP: OBJECTID will be a pain! Use a database sequence to “fake” it for INSERTs
  13. 13. SELF-SERVE GIS VIA WEB SERVICES
  14. 14. Why Offer Geospatial Web Services? • EASY to set up and use & SAVES TIME! • Geospatial data/geoprocessing via web request with parameters • Users can run individually or incorporate into scripts, software, etc. • Compatibility: Data returned in JSON format • No additional software needed
  15. 15. Geospatial Web Services* at Devon • Footage Call Converter (given footages referenced to landgrid, return lat/long) • Offset wells (given wellbore, find neighboring wells in 3D) • Wellbore XYZ from MD (given measured depth along wellbore, return XYZ) • Gunbarrel Well Views (cross-sectional sub-surface view for set of well laterals) • Coordinate conversions • Well Area Analyzer (given location, return county/state/division/BU, etc.) • Convergence Calculator (Grid North <--> True North) * Provided via FME Server Data Streaming Service
  16. 16. Place your screenshot here EXAMPLE Lat/Lon to XY Web Service
  17. 17. Place your screenshot here EXAMPLE Lat/Lon to XY Web Service JSONTemplater formats attributes into JSON string
  18. 18. Place your screenshot here EXAMPLE Lat/Lon to XY Web Service - result http://fme.pre.dvn.com/fmedatastreaming/web_services/LL_to_XY_Conv.fmw? lat=35.934458&lon=-98.585498&ll_epsg=4267&end_epsg=32024&token=xxxxx
  19. 19. CREATE AUTOMATE FLEX CLONE (YOURSELF)
  20. 20. THANK YOU! Jerrod Stutzman Jerrod.Stutzman@dvn.com

×