From FOWA Boston 2015 Structuring Data from Unstructured Things. Sean Lorenz Data coming from Internet of Things (IoT) product sensors can be hard to manage or know what to do with. In this talk Sean will discuss ways to tame IoT data sources by organizing and pruning that information effectively. He will also discuss the importance of time series when culminating sensor, metadata and other data sources together, making it vastly easier to query or perform analytics on your newly structured data.

1. Structuring Data from Unstructured Things
Sean Lorenz, Founder & CEO, Senter
@seanlorenz | @SenterIoT

2. WE CURRENTLY LIVE IN THE
INTERNET OF THING ERA

3. IS THIS AS GOOD AS
THE IOT GETS?

4. WE WANT
ADAPTIVE APPS
FROM CROSS-MANUFACTURER DATA

5. APIs ARE THE
LIFEBLOOD OF THE IOT

6. WHAT DO YOU DO
WITH ALL THAT DATA?

7. NO MORE UNTAPPED
DATA LAKES

8. NO MORE BAD
IoT DASHBOARDS

9. MORE DATA STRUCTURING
& SENSEMAKING

10. BUILDING DATA-DRIVEN IOT APPS
1 2 3 4
INGEST ORGANIZE PREDICT ACCESS/STORE
2
ORGANIZE

11. timeseries images text sparse binary sparse analog
INGEST - PROCESS DIFFERENT DATA TYPES

12. ORGANIZE - THE SOLUTION??

13. ORGANIZE - Python Data Analysis Library (pandas)
• A fast and efficient DataFrame object for data manipulation;
• Tools for reading and writing data between in-memory data structures and
different formats;
• Intelligent data alignment and integrated handling of missing data
• Columns can be inserted and deleted from data structures for size mutability;
• High performance merging and joining of data sets;
• Time series-functionality: date range generation and frequency conversion,
moving window statistics, moving window linear regressions, date shifting and
lagging. Even create domain-specific time offsets and join time series without
losing data
http://pandas.pydata.org/

14. ORGANIZE - Python Data Analysis Library (pandas) http://pandas.pydata.org/

15. ORGANIZE - Handling Time Series Data
A few things to remember:
• Schema design that minimizes memory, disk I/O
• How often do you aggregate the data?
• Read/write to a database needs to be fast, reliable,
scalable, adaptable
• Dealing with uneven time period data inputs
• How much of the raw data do you keep?
• Appending existing vs. creating new DataFrames

16. ORGANIZE - 3 Examples
Twitter, Fitbit, Temperature

17. timeseries images text sparse binary sparse analog
Deep RNN & LSTM
coding of electrical
activity to categorize
activity peaks
Deep RBM coding of
facial anomaly
detection from
security cameras
Deep RBM coding of
CRM keywords &
phrases for concept
clustering
Sparse PCA &
LASSO of ERP
system data for
delivery probability
Sparse Bayesian
coding of IoT sensor
data for smart trigger
event notifications
PREDICT - NOT ALL ALGORITHMS ARE CREATED EQUALLY

18. PREDICT - Google TensorFlow + LSTM RRN + time series data

19. ACCESS/STORE - So, so, so many options….

20. multimodal sensor fusion
w/ cognitive deep learning
IoT home and health,
phone app, & EHR data
The Hub for Adaptive Connected Home Health
+

21. LocationStates
Activity
Patterns
HzBands
O2Prediction
MedClusters
MedUsage
Prediction
Patient Health
States
PREDICTIVE HOME HEALTH IoT EXAMPLE
CARE PLAN
ACTION

22. Thanks.