Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: • Common components of an IoT solution • The challenges involved with managing time-series data in IoT applications • Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. • How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

2. @
#MDBlocal
Best practices:
Working with IoT and Time-Series
Data
Robert Walters, Director of IoT, MongoDB
@RobsCranium
CHICAGO
LinkedIn: https://www.linkedin.com/in/robwaltersprofile/

3. #MDBLocal

4. #MDBLocal
Agenda
Define time-series data
Review typical IoT Architecture
Designing time-series schemas
Query and visualize time-series data

5. #MDBLocal
What is time series data?

6. Data in an IoT Architecture

7. #MDBLocal
https://tinyurl.com/Mongodb-Houston-2019
https://www.mongodb.com/white-papers

8. #MDBLocal
Common IoT Architecture

9. #MDBLocal
Common IoT Architecture

10. #MDBLocal
Common IoT Architecture

11. #MDBLocal
Common IoT Architecture

12. #MDBLocal
Common IoT Architecture

13. #MDBLocal
Common IoT Architecture

14. #MDBLocal
MongoDB IoT Reference Architecture

15. #MDBLocal
Data at the edge

16. Designing time-series schemas

17. #MDBLocal
Breaking the relational mindset….

18. #MDBLocal
MongoDB Schema Design
for Time Series data

19. #MDBLocal
Effects on Document count
Per second vs Per Minute

20. #MDBLocal
Effects on data storage
Per second vs Per Minute

21. #MDBLocal
Effects on memory
Per second vs Per Minute

22. #MDBLocal
Sample code to add to the size based bucketMongoDB Schema Design
for Time Series data

23. #MDBLocal
Case Study:
https://www.ampcontrolgroup.com/
Leaders in industrial electrical and electronic
engineering
Selected MongoDB for Gas-Detection Mining Application

24. #MDBLocal
Case Study:

25. #MDBLocal
Case Study:
One Document per Device per Parameter per Hour
Pre-aggregated data updated for every update
After 3 months samples array removed
Data archived for 7 years

26. #MDBLocal
Keys to successful time series applications
Know your application requirements
Writes Reads Data Retention Security

27. #MDBLocal
What to do with old data
• Pre-aggregation
• Offline archival
• db.remove
• TTL Indexes
• Dropping the collection
• Online archival
• Queryable backups
• Sharding archival data
• Offload to data warehouse / S3

28. #MDBLocal
MongoDB Atlas Data Lake
Analyze data in any format on S3 using MongoDB Query Language
Archive IoT and Time-series data for long term storage and analysis

29. How to query and visualize
time-series data

30. #MDBLocal
Real-time Analytics
One database for operational and analytical
data
Easy to query unstructured data
MongoDB Query Language (MQL)
Aggregation Framework

31. #MDBLocal
Real-time Analytics
Easily query unstructured data with Aggregation Framework

32. #MDBLocal
Using MQL Demo

33. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

34. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

35. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

36. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

37. Cloud – Real-time Analytics
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
{ "_id" : ISODate("2018-12-10T00:00:00Z"), "MaxValue" : 40.4 }
{ "_id" : ISODate("2018-12-11T00:00:00Z"), "MaxValue" : 40.43 }
{ "_id" : ISODate("2018-12-12T00:00:00Z"), "MaxValue" : 40.09 }
{ "_id" : ISODate("2018-12-13T00:00:00Z"), "MaxValue" : 39.7 }
…

38. #MDBLocal
Cloud – Visualization
MongoDB Charts

39. #MDBLocal
Cloud – Visualization
MongoDB BI Connector

40. #MDBLocal
Integration with

41. Why MongoDB for IoT and Time-
Series Data?

42. #MDBLocal
Why MongoDB for IoT Data?

43. #MDBLocal
Best practices summary
Know your requirements
1 sensor reading per document works but not the most efficient
Data retention policy
Data access requirements
For very large data consider multiple collections
Faster to archive/drop collections than db.remove()
Recommended design fixed-based schema
https://www.mongodb.com/collateral/time-series-best-practices
Archive data to S3 / Atlas Data Lake for long term storage and analysis

44. #MDBLocal
Resources
MongoDB IoT Reference Architecture
Time Series Data and MongoDB: Best Practices Guide
Get Started with your project in MongoDB Atlas.
Use code “IOT200” for $200 in credit!
Thinking about an IoT Solution?
Robert.Walters@MongoDB.COM

46. THANK YOU