Scaling Graphite

1. MO’ METRICS, MO’ PROBLEMS
Erin Willingham
Infrastructure Engineer at Krux Digital
Twitter: GreenSilex
https://www.linkedin.com/in/erin-willingham-104082126

2. Krux
http://www.krux.com

3. GRAPHITE:THEN & NOW
What works, what doesn't and why we did what we did
http://www.lowcountryafricana.com/wp-content/uploads/2015/10/Research-Plan-Chalkboard-Slate-1000px.jpg

4. GRAPHS
http://i.stack.imgur.com/WBsLg.png

5. <metric path> <metric value> <metric timestamp>
test.bash.stats.count_ps 50 1473048113
test/bash/stats/count_ps.wsp

6. statsd & collectd
relay
aggregator
graphite whisper

7. GRAPHITE 1.0
ARCHITECTURE

8. RULES, MERGING,
EFFICIENCY & OPERATIONS
https://s-media-cache-ak0.pinimg.com/236x/21/ba/0f/21ba0fe48349a1d5382c261ac25cb6c6.jpg
Graphite v1

9. Relays are aware of aggregation
rules

10. Graphite Whisper merges
metrics!

11. Graphite Aggregators
are really efficient.

12. THREADING, SCALING,
RELAY CPU, & STORAGE
http://i.dailymail.co.uk/i/pix/2012/06/30/article-2166781-13BCE32D000005DC-492_634x948.jpg
Graphite v1

13. Python - single threaded

14. Relay is CPU intensive

15. Graphite Whisper -
requires sharding and is very I/O intensive
http://obfuscurity.com/

16. Slow UI when using distributed
remote backends

17. What are we trying to solve?
What is forcing the change?
http://oakdome.com/k5/lesson-plans/photo-editing/wanted-poster/wanted-reward-poster-background.jpg
Storage!

18. Relay & Aggregator
CPU usage high

19. Faster UI

20. KEEP COSTS LOW
http://3.bp.blogspot.com/-r9l7rltAjnM/Udq8kGlp65I/AAAAAAAAANo/VyQZN48nfMk/s1600/treasurepile.jpg

21. GRAPHITE ALTERNATIVES
http://3.bp.blogspot.com/-r9l7rltAjnM/Udq8kGlp65I/AAAAAAAAANo/VyQZN48nfMk/s1600/treasurepile.jpg
Circonus:All the insights you ever wanted
Hosted Graphite
Zabbix: OSS self hosted monitoring

22. CARBON-C-RELAY, KAFKA, SOCAT,
CARBON-RELAY-NG, KAFKACAT
https://wtfbabe.files.wordpress.com/2015/06/kung-fury-23-wtf-watch-the-film-saint-pauly.jpeg
The Tools

23. Carbon-c-relay
https://github.com/grobian/carbon-c-relay
GRAPHITE 2.0
TOOLS

24. Carbon-relay-ng
https://github.com/graphite-ng/carbon-relay-ng
GRAPHITE 2.0
TOOLS

25. Kafka Producer
tcp-stream-kafka-producer
https://github.com/krux/tcp-stream-kafka-producer
GRAPHITE 2.0
TOOLS

26. kafkacat
https://github.com/edenhill/kafkacat
GRAPHITE 2.0
TOOLS

27. GRAPHITE 2.0
TOOLS
socat
“exec:/usr/bin/kafkacat
-C
-o end
-b <kafka broker>
-t <kafka topic>”
,pty,ctty,echo=0,
tcp4-connect:localhost:<relay port>

28. BACKEND - STORAGE
http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg

29. • Whisper
• Ceres
• InfluxDB
• Cyanite
• Riak
• KairosDB
• OpenTSDB

30. Graphite - Whisper

31. InfluxDB

32. KairosDB

33. GRAPHITE 2.0
ARCHITECTURE

34. GRAPHITE ARCHITECTURE -
SCALABLE
http://www.dinopit.com/wp-content/uploads/2012/07/dinosaur-cowboy.jpg
Why?

35. LOADTESTINGTHE PARTS ANDTHE
PIPELINE
https://github.com/feangulo/graphite-stresser
All the Metrics!
Metrics / min

36. WHAT WORKED?
http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg
Pre-aggregated
Post Aggregated

37. http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg

38. MIRROR PRODUCTION DATA
https://c2.staticflickr.com/6/5278/5903002116_762783602c_b.jpg

39. UH OH!
THE GRAPHS DON’T MATCH
http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg
Old Cluster
New Cluster

40. HOW DO WE FIXTHIS?
http://www.startres.net/startresWP/wp-content/uploads/2013/06/3702A.jpg

41. TESTING CARBON-RELAY-NG
http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg

42. Carbon-relay-ng uses more
than 2 CPUs!

43. FAILURE POINT FOR
CARBON-RELAY-NG
http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg
Post Aggregated
Pre-aggregated

44. Carbon-relay-ng:
room for improvement

45. • scale out aggregators horizontally
• monitor for metrics per second and scale out as
needed
• pass metrics that don’t need to be aggregated
directly to the backend
https://github.com/edenhill/kafkacat
SOLUTION

46. http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg

47. QUESTIONS?