Download to read offline
Uploaded byErin Willingham
Mo' Metrics, Mo' Problems
This document discusses improvements to the Graphite monitoring system architecture in version 2.0. It notes issues with the original design such as single-threaded processing, high CPU usage for relays and aggregators, and inefficient storage. The proposed Graphite 2.0 architecture uses new tools like Carbon-c-relay, Carbon-relay-ng, Kafka, and kafkacat to enable horizontal scaling and distribute load. It also suggests alternative backend storage options like InfluxDB, KairosDB, and OpenTSDB to address storage problems in the original Whisper system. Load testing validated the benefits of pre-aggregated versus post-aggregated metrics processing.
![5G Explained! A High Level Overview [Introduction]](https://cdn.slidesharecdn.com/ss_thumbnails/5gexplainedahighleveloverview-260119165306-cc137a3e-thumbnail.jpg?width=640&height=640&fit=bounds)
Mo' Metrics, Mo' Problems
- 1. MO’ METRICS, MO’PROBLEMS Erin Willingham Infrastructure Engineer at Krux Digital Twitter: GreenSilex https://www.linkedin.com/in/erin-willingham-104082126
- 2.
- 3. GRAPHITE:THEN & NOW Whatworks, what doesn't and why we did what we did http://www.lowcountryafricana.com/wp-content/uploads/2015/10/Research-Plan-Chalkboard-Slate-1000px.jpg
- 4.
- 5. <metric path> <metricvalue> <metric timestamp> test.bash.stats.count_ps 50 1473048113 test/bash/stats/count_ps.wsp
- 6.
- 7.
- 8. RULES, MERGING, EFFICIENCY &OPERATIONS https://s-media-cache-ak0.pinimg.com/236x/21/ba/0f/21ba0fe48349a1d5382c261ac25cb6c6.jpg Graphite v1
- 9. Relays are awareof aggregation rules
- 10.
- 11. Graphite Aggregators are reallyefficient.
- 12. THREADING, SCALING, RELAY CPU,& STORAGE http://i.dailymail.co.uk/i/pix/2012/06/30/article-2166781-13BCE32D000005DC-492_634x948.jpg Graphite v1
- 13. Python - singlethreaded
- 14. Relay is CPUintensive
- 15. Graphite Whisper - requiressharding and is very I/O intensive http://obfuscurity.com/
- 16. Slow UI whenusing distributed remote backends
- 17. What are wetrying to solve? What is forcing the change? http://oakdome.com/k5/lesson-plans/photo-editing/wanted-poster/wanted-reward-poster-background.jpg Storage!
- 18. Relay & Aggregator CPUusage high
- 19.
- 20.
- 21. GRAPHITE ALTERNATIVES http://3.bp.blogspot.com/-r9l7rltAjnM/Udq8kGlp65I/AAAAAAAAANo/VyQZN48nfMk/s1600/treasurepile.jpg Circonus:All theinsights you ever wanted Hosted Graphite Zabbix: OSS self hosted monitoring
- 22. CARBON-C-RELAY, KAFKA, SOCAT, CARBON-RELAY-NG,KAFKACAT https://wtfbabe.files.wordpress.com/2015/06/kung-fury-23-wtf-watch-the-film-saint-pauly.jpeg The Tools
- 23.
- 24.
- 25.
- 26.
- 27. GRAPHITE 2.0 TOOLS socat “exec:/usr/bin/kafkacat -C -o end -b<kafka broker> -t <kafka topic>” ,pty,ctty,echo=0, tcp4-connect:localhost:<relay port>
- 28.
- 29. • Whisper • Ceres •InfluxDB • Cyanite • Riak • KairosDB • OpenTSDB
- 30.
- 31.
- 32.
- 33.
- 34.
- 35. LOADTESTINGTHE PARTS ANDTHE PIPELINE https://github.com/feangulo/graphite-stresser Allthe Metrics! Metrics / min
- 36.
- 37.
- 38.
- 39. UH OH! THE GRAPHSDON’T MATCH http://www.xzbackup.com/content/wp-content/uploads/2016/01/datacenter_triinti.jpg Old Cluster New Cluster
- 40. HOW DO WEFIXTHIS? http://www.startres.net/startresWP/wp-content/uploads/2013/06/3702A.jpg
- 41.
- 42.
- 43.
- 44.
- 45. • scale outaggregators horizontally • monitor for metrics per second and scale out as needed • pass metrics that don’t need to be aggregated directly to the backend https://github.com/edenhill/kafkacat SOLUTION
- 46.
- 47.