The document provides an overview of the Observer time-series application developed by Kévin Lovato, detailing its architecture, key features, and CQL schema for data management. It discusses various components like metrics aggregation, alerting mechanisms, schema best practices, and performance optimization strategies. Additionally, it outlines future plans including transitioning to SSDs and using different data storage configurations.

1. Observer
A real life time-series application
Kévin Lovato - @alprema

2. Index
• Observer introduction
• Architecture overview
• CQL schema
• Feedback
– Schema
– Read/Write access
• Numbers

3. Observer introduction

4. Key features
• Publish metrics from anywhere
• Track & investigate business issues
• Alert users in case of unusual behavior
• Integrate with the infrastructure features

8. Architecture overview

9. C*
Aggregator
Publisher
Send raw metrics

10. C*
Aggregator
Publisher
Aggregate metrics
(sec, min, hour)

11. C*
WebDashboard
Client
Load metrics data
HTTP

12. C*
WebDashboard
Client
Receive live metrics data
through bus
Push
(WebSocket)

13. C*
DataCruncher
Load and compute all
metrics for the day
Write daily computations
(avg, percentiles, etc.)

14. C*
Alertor
Catch up on startup
Receive live metrics data
through bus
Send alerts on the bus

15. CQL schema

16. Metric_OneSec
• Schema:
((MetricId, Day), UtcDate), Value
MetricId +
Day
UtcDate UtcDate …
Value Value

17. Metric_OneSec
• TTL: 8 days
• Max column per row: 86 400
• Average size: 1.4 MB

18. Metric_OneMin
• Schema:
((MetricId, FirstDayOfWeek), UtcDate), Value
MetricId +
FirstDayOfWeek
UtcDate UtcDate …
Value Value

19. Metric_OneMin
• TTL: 60 days
• Max column per row: 10 080
• Average size: 300 KB

20. Metric_OneHour
• Schema:
(MetricId, UtcDate), Value
MetricId
UtcDate UtcDate …
Value Value

21. Metric_OneHour
• TTL: 10 years
• Average size: 45 KB

22. Daily_Aggregate
• Schema:
(MetricId, Date), Average, Count, Percentiles, …
MetricId
Date.Average Date.Count …

23. Daily_Aggregate
• No TTL
• Average size: 23 KB

24. Feedback - Schema

25. Row sizing
• Avoid having rows spanning over long
periods
• Avoid large amounts of data / row (<100
MB is good)
• Make buckets using another component
(ex: Day, FirstDayOfWeek, etc.)

26. TTLs
• Don’t use them if you don’t really need them
(extra space wasted)
• Make sure to set it right the first time (or you
will need to reinsert your data)
• Consider changing gc_grace_period for your
CF (tombstones useless for TTLed time-
series)

27. General best practices
• Consider disabling inter-DC read repair on
your CF (read_repair_chance)
• Use collection types (map<>, etc.)

28. Feedback – Read / Write

29. Obvious but…
• Avoid Thrift (can take down your cluster on
huge rows reads)
• Do not disable paging (same effect as using
Thrift)
• Use prepared statements

30. Batches
• Warning: Not intended for performance
• But…
• Can improve insert performance under
adequate conditions
• Use small (< 5 KB) "Unlogged" batches
• Benchmark with your own use case
• Don’t tell @PatrickMcFadin you did it

31. Asynchronous queries
• Mandatory if you want to be fast (from
anything over 1 query)

32. Asynchronous queries
Vs.

33. Asynchronous queries
• For massive reads, send your queries by
bunches and wait for them

34. General best practices
• Benchmark all heavy operations in terms of
cluster load (a faster implem might just be
killing the cluster for everyone else)
• Watch out for CL: ONE (we experienced
slowdowns as the coordinator asked a
different DC under heavy load)

35. Numbers time

36. • Total number of metrics: 17K
• Metrics inserted: 10K/s
• Data points daily aggregation speed: 500K/s
• DC size: 3 nodes (spinning disks)

37. Future
• Use DTCS (MaybeTWCS? CASSANDRA-
9666 / CASSANDRA-10195)
• Move to SSDs everywhere

38. Interested? We’re hiring

39. Questions?

40. Image credits – The Noun Project
• Björn Andersson
• Creative Stall
• Gregor Cresnar
• Justin Blake
• Lemon Liu
• Mark Shorter
• Shawn Schmidt
• Stéphanie Rusch