Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: Common components of an IoT solution • The challenges involved with managing time-series data in IoT applications • Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. • How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts • At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

2. @
#MDBlocal
Best practices:
Working with IoT and Time-Series
Data
Robert Walters, Director of IoT, MongoDB
@RobsCranium
LONDON
LinkedIn: https://www.linkedin.com/in/robwaltersprofile/

3. #MDBLocal

4. #MDBLocal
Agenda
Define time-series data
Review typical IoT Architecture
Designing time-series schemas
Query and visualize time-series data

5. #MDBLocal
What is time series data?

6. Data in an IoT Architecture

7. #MDBLocal

8. #MDBLocal
Common IoT Architecture

9. #MDBLocal
Common IoT Architecture

10. #MDBLocal
Common IoT Architecture

11. #MDBLocal
Common IoT Architecture

12. #MDBLocal
Common IoT Architecture

13. #MDBLocal
Common IoT Architecture

14. #MDBLocal
MongoDB IoT Reference Architecture

15. #MDBLocal
Data at the edge

16. Data requirements in IoT
solution

17. #MDBLocal
Data Platform Requirements
Agility Schema Flexibility
Real-time Data Archiving
Global Presence Data Protection
IoT data is a Big Data problem - Volume, Velocity, Variety

18. #MDBLocal
Freedom to deploy wherever you need

19. Designing time-series schemas

20. #MDBLocal
Breaking the relational mindset….

21. #MDBLocal
MongoDB Schema Design
for Time Series data

22. #MDBLocal
Effects on Document count
Per second vs Per Minute

23. #MDBLocal
Effects on data storage
Per second vs Per Minute

24. #MDBLocal
Effects on memory
Per second vs Per Minute

25. #MDBLocal
Sample code to add to the size based bucketMongoDB Schema Design
for Time Series data

26. #MDBLocal
Case Study:
https://www.ampcontrolgroup.com/
Leaders in industrial electrical and electronic
engineering
Selected MongoDB for Gas-Detection Mining Application

27. #MDBLocal
Case Study:

28. #MDBLocal
Case Study:
One Document per Device per Parameter per Hour
Pre-aggregated data updated for every update
After 3 months samples array removed
Data archived for 7 years

29. #MDBLocal
Keys to successful time series
applications
Know your application requirements
Writes Reads Data Retention Security

30. #MDBLocal
What to do with old data
• Pre-aggregation
• Offline archival
• db.remove
• TTL Indexes
• Dropping the collection
• Online archival
• Queryable backups
• Sharding archival data
• Offload to data warehouse / S3

31. #MDBLocal
MongoDB Atlas Data Lake
Analyze data in any format on S3 using MongoDB Query Language
Archive IoT and Time-series data for long term storage and analysis

32. How to query and visualize
time-series data

33. #MDBLocal
Real-time Analytics
One database for operational and analytical
data
Easy to query unstructured data
MongoDB Query Language (MQL)
Aggregation Framework

34. #MDBLocal
Real-time Analytics
Easily query unstructured data with Aggregation Framework

35. #MDBLocal
Using MQL Demo

36. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

37. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

38. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

39. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

40. Cloud – Real-time Analytics
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
{ "_id" : ISODate("2018-12-10T00:00:00Z"), "MaxValue" : 40.4 }
{ "_id" : ISODate("2018-12-11T00:00:00Z"), "MaxValue" : 40.43 }
{ "_id" : ISODate("2018-12-12T00:00:00Z"), "MaxValue" : 40.09 }
{ "_id" : ISODate("2018-12-13T00:00:00Z"), "MaxValue" : 39.7 }
…

41. #MDBLocal
Cloud – Visualization
MongoDB Charts

42. #MDBLocal
Cloud – Visualization
MongoDB BI Connector

43. #MDBLocal
Integration with

44. Why MongoDB for IoT and Time-
Series Data?

45. #MDBLocal
Why MongoDB for IoT Data?

46. #MDBLocal
Best practices summary
Know your requirements
1 sensor reading per document works but not the most efficient
Data retention policy
Data access requirements
For very large data consider multiple collections
Faster to archive/drop collections than db.remove()
Recommended design fixed-based schema
https://www.mongodb.com/collateral/time-series-best-practices
Archive data to S3 / Atlas Data Lake for long term storage and analysis

47. #MDBLocal
Resources
MongoDB IoT Reference Architecture
Time Series Data and MongoDB: Best Practices Guide
Get Started with your project in MongoDB Atlas.
Use code “IOT200” for $200 in credit!
Thinking about an IoT Solution?
Robert.Walters@MongoDB.COM

48. THANK YOU