Creating Commercial Data
Products with FME
About
setld is a next generation “geospatial+” data and service provider
Goals:
– Provide cost-effective data products
– Help transform data into actionable information
– Bring data management and science capability to orgs of any size (GIS / BI as-a-Service)
Qualifications:
– Proven industry experience
– esri Startups Program / Emerging Business Partner
– AWS Activate Partner
– Microsoft BizSpark Member (w/ Azure)
– Safe Software Solution Provider (1 of 16 in W.H.)
– FME Certified Professional (1 of 30 in U.S.)
Offerings:
– Foundational and custom data products
– Consulting (from data flows to
dissemination innovation)
– GIS / BI as-a-Service
Overview
3
• Creating Commercial Data Products?
• Challenges
• Why Choose FME?
• Our Foundation
• Innovation with FME
• Sustainable & Valuable Solutions
• What’s Next?
• Questions
Creating Commercial Data Products?
“I don’t need to do that. How will this talk help me?”
What we hear about FME adoption across industries
– It’s everywhere
– FME is easy, fast, powerful and people love it
– Business folks are flowing information to maps, charts, reports and alerts like never before
IT, data governance and sustainability planning are often an afterthought
– But, but......business folks are innovating! Get out of the way!
– My process gives me and my team what we need. Who cares
what it looks like or how it’s done?
Let’s think harder…how will you share your work and the
spirit of innovation with the rest of your organization?
4
Multiple Tools Available
Manual labor
Scripting
Pick a programming language
Open source GIS
ArcGIS
Other ETLs
FME
Multiple Actions Needed
Starting
Thinking / Understanding
Deciding
Scoping
Structuring
Ignoring
Parsing
Reconciliation
Reformatting
Calculations
Enrichment
Aggregation
Units checks and validations
Merging and appending
Transferal of attribution
Spatial self-tagging and attribution
Projections and transforms
QA / QC
Finishing
Multiple Sources
U.S. federal government
State governments
Foreign government and agencies
Academics, colleges and universities
Non-governmental organizations (NGO)
Corporate websites, reports and downloads
Acquired orthophotos, LIDAR, drone imagery
Econometrics, IMF, World Bank
United Nations (UN)
John’s external HD
Conservationists
Non-spatial
USB Stick
Presentations
Sensors
RPLS
News
Multiple Formats
Shapefiles
Spreadsheets
ASCII
Access MDBs
Web Scraping and Harvests
File Geodatabase (FGDB)
Personal Geodatabase (PGDB)
SDE feature classes / exports
CAD / BIM
Geographic Data Format (GDF)
Raster…many flavors
JSON
KML
Challenges
We lever standardized internal
policy, workflows and FME to
process, enrich, QC and deliver!
Why Choose FME?
Effective data integration is more important than ever
– More sources, more data, more formats and greater expectations for fast, actionable
information
– FME’s nearly self-documenting visual workbench, support for 400+ formats and QC tools
are fantastic
– Basing our business on unsupported open source tools would be difficult
Safe Software track record
– Solving big problems in bunches for many years
– The team innovates daily and takes feedback
– Excellent technical support community
I’ve been a user since 2005 and have grown with FME
– Began with scheduled spatial data loads
– Then, enriched data to drive important maps daily
– Moved to other roles where our team completely rebuilt all
geospatial and BI capability at a major oil and gas company
– And now……stepping further back in the process to lend a hand
6
Our Foundation
7
setld GoM Data Product
Structure and convention
– Structure what you can; may be easier than you realize
– Everything in its place, named appropriately
– Revise if product evolution dictates (don’t be lazy ++)
Scheduled automation and live monitoring
– Gather and prep inputs
– Process and enrich
– QC reporting / dashboarding
Useful Dissemination
– Meaningful atts and cartographic design
– Often enhance information to suit tools like AGOL WAB
– What are we doing?
– Input prep and inputs
– Main FME jobs w/ QC outputs
– Staged FTP deliverables (from SDE)
– Maps, .lyrs for publish, guides
– Internal product support / dev
Innovation with FME
8
FME Roles
Quick prototyping
Quick QC
One-off data fixes
Data maintenance
Production jobs w/ QC dog legs
Revise as we learn more
File management / migration
Holistic systems QC jobs
Tips & Tricks
Harvest JSON, etc. from web
Use FME to make other .FMWs
Output a .BAT from FME, use inline
Step out to Python, and back
Feature writer > .CSV > Feature Reader (sort on read)
Use 64-bit for more than a few million rows
Data Enrichment Examples
Net production
Activity order and summation
Format betterment
Classify assets
Deduce statuses
Circular references
Calcs - 3 / 6 / 12 / 24 / 60 monthly prod. avgs
Render directional events at TVD
Generate and maintain metadata
Workforce Assist Examples
Task Lists
Prioritization and progress
Marketing summaries
QA / QC Made Fun
Automatic summary emails
Slack notifications
Dashboarding
Sustainable & Valuable Solutions
9
In-progress developments include:
– U.S. Well Header
– U.S. Well Production Data
– U.S. Pipeline Set
– Mexico Data Product
– West Africa Data Product
After 1 year of proving and improving this strategy –
we’ll continue processing the majority of our data with FME
We hope to see more big data and (perhaps) AI tools
built into FME in the coming years!
10
What’s Next?
11
Questions?
www.setld.com
+1-832-563-5699
support@setld.com

Creating Commercial Data Products with FME

  • 1.
  • 2.
    About setld is anext generation “geospatial+” data and service provider Goals: – Provide cost-effective data products – Help transform data into actionable information – Bring data management and science capability to orgs of any size (GIS / BI as-a-Service) Qualifications: – Proven industry experience – esri Startups Program / Emerging Business Partner – AWS Activate Partner – Microsoft BizSpark Member (w/ Azure) – Safe Software Solution Provider (1 of 16 in W.H.) – FME Certified Professional (1 of 30 in U.S.) Offerings: – Foundational and custom data products – Consulting (from data flows to dissemination innovation) – GIS / BI as-a-Service
  • 3.
    Overview 3 • Creating CommercialData Products? • Challenges • Why Choose FME? • Our Foundation • Innovation with FME • Sustainable & Valuable Solutions • What’s Next? • Questions
  • 4.
    Creating Commercial DataProducts? “I don’t need to do that. How will this talk help me?” What we hear about FME adoption across industries – It’s everywhere – FME is easy, fast, powerful and people love it – Business folks are flowing information to maps, charts, reports and alerts like never before IT, data governance and sustainability planning are often an afterthought – But, but......business folks are innovating! Get out of the way! – My process gives me and my team what we need. Who cares what it looks like or how it’s done? Let’s think harder…how will you share your work and the spirit of innovation with the rest of your organization? 4
  • 5.
    Multiple Tools Available Manuallabor Scripting Pick a programming language Open source GIS ArcGIS Other ETLs FME Multiple Actions Needed Starting Thinking / Understanding Deciding Scoping Structuring Ignoring Parsing Reconciliation Reformatting Calculations Enrichment Aggregation Units checks and validations Merging and appending Transferal of attribution Spatial self-tagging and attribution Projections and transforms QA / QC Finishing Multiple Sources U.S. federal government State governments Foreign government and agencies Academics, colleges and universities Non-governmental organizations (NGO) Corporate websites, reports and downloads Acquired orthophotos, LIDAR, drone imagery Econometrics, IMF, World Bank United Nations (UN) John’s external HD Conservationists Non-spatial USB Stick Presentations Sensors RPLS News Multiple Formats Shapefiles Spreadsheets ASCII Access MDBs Web Scraping and Harvests File Geodatabase (FGDB) Personal Geodatabase (PGDB) SDE feature classes / exports CAD / BIM Geographic Data Format (GDF) Raster…many flavors JSON KML Challenges We lever standardized internal policy, workflows and FME to process, enrich, QC and deliver!
  • 6.
    Why Choose FME? Effectivedata integration is more important than ever – More sources, more data, more formats and greater expectations for fast, actionable information – FME’s nearly self-documenting visual workbench, support for 400+ formats and QC tools are fantastic – Basing our business on unsupported open source tools would be difficult Safe Software track record – Solving big problems in bunches for many years – The team innovates daily and takes feedback – Excellent technical support community I’ve been a user since 2005 and have grown with FME – Began with scheduled spatial data loads – Then, enriched data to drive important maps daily – Moved to other roles where our team completely rebuilt all geospatial and BI capability at a major oil and gas company – And now……stepping further back in the process to lend a hand 6
  • 7.
    Our Foundation 7 setld GoMData Product Structure and convention – Structure what you can; may be easier than you realize – Everything in its place, named appropriately – Revise if product evolution dictates (don’t be lazy ++) Scheduled automation and live monitoring – Gather and prep inputs – Process and enrich – QC reporting / dashboarding Useful Dissemination – Meaningful atts and cartographic design – Often enhance information to suit tools like AGOL WAB – What are we doing? – Input prep and inputs – Main FME jobs w/ QC outputs – Staged FTP deliverables (from SDE) – Maps, .lyrs for publish, guides – Internal product support / dev
  • 8.
    Innovation with FME 8 FMERoles Quick prototyping Quick QC One-off data fixes Data maintenance Production jobs w/ QC dog legs Revise as we learn more File management / migration Holistic systems QC jobs Tips & Tricks Harvest JSON, etc. from web Use FME to make other .FMWs Output a .BAT from FME, use inline Step out to Python, and back Feature writer > .CSV > Feature Reader (sort on read) Use 64-bit for more than a few million rows Data Enrichment Examples Net production Activity order and summation Format betterment Classify assets Deduce statuses Circular references Calcs - 3 / 6 / 12 / 24 / 60 monthly prod. avgs Render directional events at TVD Generate and maintain metadata Workforce Assist Examples Task Lists Prioritization and progress Marketing summaries QA / QC Made Fun Automatic summary emails Slack notifications Dashboarding
  • 9.
  • 10.
    In-progress developments include: –U.S. Well Header – U.S. Well Production Data – U.S. Pipeline Set – Mexico Data Product – West Africa Data Product After 1 year of proving and improving this strategy – we’ll continue processing the majority of our data with FME We hope to see more big data and (perhaps) AI tools built into FME in the coming years! 10 What’s Next?
  • 11.