Matrix for MongoDB

1. Making Sense of
Time Series Data in
MongoDB
Matan Zohar
CTO, Matrix Open Source

2. 2
Whoami?
• Helping customers with MongoDB & Data intensive architectures for the past 3 years.
Anything from MongoDB, Kafka, Spark, Nifi, Hadoop to Docker, Kubernetes / Openshift
and the mix of them together.
• Experienced in designing and running production Data-driven, Messaging & Event-
driven architectures for more than 10 years.
• Looking constantly for new software projects to make our work life smarter and
easier.
• Loves chatting about martial arts, dogs, and philosophical thoughts.

3. 3
Agenda
• What is time series data?
• Why MongoDB for time series?
• Time series challenges and patterns
• Querying time series data in MongoDB
• Tools of the trade

4. 4
What is time series data?

5. 5
Actually it’s quite simple…
• Timestamp
• value
• data tags (optional)

6. 6
Why MongoDB?
• Flexible document model
• Transactional
• Easily horizontally scalable
• Optimized indexes
• Native analytics + reporting tools
• Mobile, On-prem, SaaS
• Massive community 30M+ downloads

7. 7
Somebody did that?
https://www.mongodb.com/customers/bosch

8. 8
Connected Cow
Pedometer Sensor

9. 9
Connected Cow

10. 10
Rich query functionality

11. 11
Transactions are here

12. What is the right pattern for time series?

13. 13
Key to successful time series app
• Writes per second?
• Read patterns?
• Read intensive?
• Data retention?
• Security requirements?
Know the
application

14. 14
General structure of time series apps
1. Write fast
2. Aggregate
1. Load to memory
2. Optimize aggregation for query pattern
3. Save to file / store
3. Query

15. 15
Pattern #1 : One document per data point
Tick per second

16. 16
Pattern #2 : Time-based bucketing
Bucket per minute

17. 17
Pattern #3 : Size-based bucketing
Update last point index
last point index

18. 18
Bucketing impact example
1 675 678
4 608 156
7 542 468
10 475 596
28 399 78 102 127 836 177 550
-
2 000 000
4 000 000
6 000 000
8 000 000
10 000 000
12 000 000
7 14 21 28
NumofDocuments
Days
Document Count
Document per sec Document per min

19. 19
Bucketing impact example
96,84
266,32
435,90
605,42
29,83
83,90
135,64
190,64
20,46
56,28
92,12
127,94
8,10
23,18
37,48
52,71
-
100,00
200,00
300,00
400,00
500,00
600,00
700,00
7 14 21 28
Size(MB)
Days
Collection size (storage)
Data size (per sec) Disk storage (per sec) Data size (per min) Disk storage (per min)

20. 20
Bucketing impact example
25,37
46,66
73,95
103,03
0,40 0,91 1,37 1,86
-
20,00
40,00
60,00
80,00
100,00
120,00
7 14 21 28
Size(MB)
Days
Index Size (memory)
Document per second Document per minute

21. I’ve heard that LSM is just better
than that B+ tree…

22. 22
Is LSM better than B+ tree?
• It depends if you only write and save or actually query…
• LSM is better for fast writes but bad for reads, that means you will usually do the reads elsewhere…
• B+ tree is better for reads but trades off on write performance.
LSM tree B+ tree

23. But I want fast writes + reads,flexibility,
and rich queries!

24. 24
Take the best for every step of the way
Fast ingest Aggregate /
bucket
Store and query

25. So how will I query time series?

26. 26
Aggregation pipeline

27. 27
Not that complicated
SELECT cust_id, SUM(amount) AS total
FROM orders
GROUP BY cust_id
db.orders.aggregate([
{ $match: { status: “A"} } },
{ $group: { _id: "$cust_id", total: { $sum: "$amount" } }
])
SQL:
MongoDB:

28. See statistics on documents

29. Run explain on queries

30. Build & Debug aggregation pipeline

31. Add new steps to pipeline

32. Preview every step change in sample documents

33. Preview every step change in sample documents

34. Preview every step change in sample documents

35. Export to selected language

36. 36
MongoDB connector for BI

37. 37
MongoDB Charts (beta)

38. But what about heavy lifting
data science?

39. 39
Analytics at scale with Spark connector

40. 40
MongoDB R driver

41. Monitor,monitor,monitor!

42. 42

43. Can you sum it up?

44. 44
Key takeaways
• Think about your end goal (business need)
• Know your application requirements
• Choose the right pattern (bucket time / size)
• Plan for retention and cleanup
• Write intensive? Buffer with Kafka
• Use aggregation and all the tools (compass, charts, BI connector)
• Don’t forget to monitor and tune it as you go

45. Thank you for your
time!
open@matrix.co.il