HP predicts that by 2020, 40% of all data ever collected by the human kind will have been generated by sensors. But if you can't use the data, if you can search and discover it; and if you can't make it machine-readable, then the investment into intelligent sensor networks will be unused. In this presentation, I discuss different cases of data integration and discovery, and how to turn this data into usable/readable information both for humans and machines, thus allowing data professionals, executives and data vendors all do what they do best, leaving data integration and discovery to professionals.

1. Rainer Sternfeld, CE September 2014
The Critical Role of IoT Data Integration
to develop Big Data Applications
February 5, 2015
The Unmanned Systems Expo 2015, The Hague
Rainer Sternfeld, CEO, Planet OS
@planet_os Image Credit: Northrop Grumman

2. 2 February 2015
Sensor data
will outgrow
social data in 2 years
Sensor data is
too big to move
Data Management
will not scale
as we know it
The bigger picture
Swimming in sensors, drowning in data

3. 3 February 2015
Hewlett Packard:
“By 2020, 40% of all data
ever collected by human kind
will be generated by sensors.”

4. 4 February 2015
If you can’t use this data…
what’s the point?

5. 5 February 2015
Case #1: NOAA
Real-time weather and climate data + forecasts
They are working hard on a challenge presented by:
• Tens of thousands of devices deployed in the ocean, on land, and space
• Tens of terabytes coming in every day
• 700 scattered “web services” (FTPs, flat files, Threads servers, no APIs)
• Breaking connections (all.the.time.)
• 25,000 employees
• Critical data for the government, the industry, and citizens
What if all this data would be easily discoverable and machine-readable?

6. 6 February 2015
Example to case #1: Marinexplore.org
40,000+ data streams from 33 organizations of open data
Advanced Data Discovery Raster data / heat map overlays Access with third party applications
Raster data / quiver plots Graph Monitor Build custom datasets

7. 7 February 2015
Consolidate dataflows, organize and make sense of your data
Access your data with 3rd party tools and systems
One interface to search and discover your local and remote data
Securely exchange, acquire or sell datasets
Build advanced domain specific solutions without hassle
Industrial IoT Platform for Real-World Sensor Networks
designed for ocean, land, air and space data

8. 8 February 2015
Integrate, curate, control access and monitor dataflows
Analytics dashboard Integrate and configure new data channels
Roles, authorizations and logs Dataflow dashboard

9. 9 February 2015
Case #2: Agriculture + data science
Models, predictions, outliers, multi-source data
New companies are emerging:
• Predict a farmer’s profit statement based using adaptive models
(drone data + environmental data + in-situ data + soil/seed/pesticide choices)
• Open data is sparse and scattered
• Data dissemination is hard
• Farmer’s data input in hard
• “Do we really need to build a data platform to build our business?”
What if newco’s and farmers could focus on what they know best?

10. 10 February 2015
Case #3: Surveillance and tactical in-flight data
Having a 2-way data aggregation is critical
• End-users using a variety of drones need data versatility
• “How can I get a data-edge in a chase scenario?”
• Situational awareness based on the same surrounding data
for all unmanned vehicles in your fleet
• Drones need more accurate surrounding data than manned
crafts (don’t fly with your windows taped)

11. 11 February 2015
Case #4: Planet OS data exchange in a swarm scenario
Secure data exchange for tactical navigation & interoperability
• Connected cars / drones / UGVs / AUVs / fleets
• Multi-particle computation for ensuring privacy
• Collision avoidance using encrypted data
• Behavioral data collection using encrypted data

12. 12 February 2015
Main Use Cases
Visualisation
Analytics
Event Processing
Models
Curation & Logistics
Marketplace
Search
Exploration
TECHNICAL DATA
• Spatio-Temporal
• Sensors
• Observations
• Models
• Predictions
• Metadata

13. 13 February 2015
Interfaces and Integrations
DATA FORMATS
• CSV, Spreadsheet
• GeoTiff, ESRI Raster
• HDF, NetCDF (CF), GRIB
• Shapefile, KML
• OGC Geopackage
• Metadata: Exif, SRID, ISO 19139
• Images, Audio, Video
• Custom formats
DATA ACCESS / EXPORT
• CSV, Spreadsheet
• GeoTIFF, HDF, NetCDF (CF)
• OGC WFS / WCS / WMS
• OpenDAP
• ESRI ArcGIS
• ISO-19139
• Custom formats and protocols
DATA SOURCES
• HTTP, FTP
• NFS, Samba, Sharepoint
• OpenDAP, THREDDS, ERDAP
• OGC WFS / WCS / WMS
• ESRI ArcGIS, PostGIS
• Archives: ZIP, etc
• Custom protocols

14. 14
Index the real world