Spatial Networks geospatial data assets fit the gambit of Human Geography domains and Geospatial Intelligence, with a contextually relevant and intimate understanding from local professional experts collecting data on the ground. We are transforming this data into insightful analytics and rich, robust data sets of 21 unique data types to help our clients solve challenging geospatial problems. FME has helped us to streamline our workflows and automate several processes along our data pipeline. We look to scale our operations significantly in 2018 and FME will ease many of our challenges as we move forward.
4. 86,000+
3M+
63M+
users in 172 countries
field surveys recorded per month
survey records
Mobile app platform enabling businesses to build
customized workflows for field data collection.
5. • Legacy datasets in PostGIS of varying schemas and data types
• Ongoing mobile survey field collection ETL to a standard data model(s)
• Maintain data integrity and provenance
• Data validation and QA/QC
• Standardization of data storage and workflows
• Automation of workflows with minimal programming
• Metadata, temporal accuracy and update records for attribute changes -
versioning
• Basically, a decade plus worth of data assets requiring intense curation,
management and governance for a rapidly growing company.
Challenges
6. FME - Data Pipeline Backbone
• Project Management
• Data Acquisition
• Data Exploitation
• Data Conditioning
• Data Enrichment
• Data Integration
• Product Development
• Data Representation
• Metadata Management
• Product/Data Delivery
8. Baseline Data Model
Field Survey Data Types
Standard Mobile Data Collection
Field Survey Apps
E T L
Human Geography Themes
Standard Theme Domains
World Wide Human Geography
Data Model(s)
Commodities
Urban Survey Business
Production Facilities
Economy
13. Validating Features and Attributes
• SQL Statements on Read
• TestFilters & Conditional Statements
• Consistency in Updates
• QA/QC and Error Logging into
Workflows
17. • FME can probably do it! And in more than one way!
• FME Knowledge Center
○ On-line Training Courses, Videos(YouTube) and Tutorials are excellent learning tools!
○ Q&A Forum and Knowledge Base are an excellent resource.
• Best Practices
○ Use Bookmarks!
○ Use consistent naming, numbering and versioning standards
• FME Quick Translator
○ Integral to packaging AOI datasets quickly
What We’ve learned…
18. • Transform legacy datasets in PostGIS of varying schemas and data types
into a data model standard
• Automate ETL processes of near-real time mobile survey field collection to a
standard data model
• Ensure the preservation of data integrity and provenance
• Maintain consistent data validation and QA/QC standards
• Standardization of data storage and workflows
• Automation of workflows with minimal programming
• Metadata, temporal accuracy and update records for attribute changes -
versioning
• We still have much more to explore and learn about FME’s capability to help
us streamline our workflows and automate processes.
What FME has Helped Us Achieve…