Download to read offline
Uploaded byWil Heitritter
Graphite at CityGrid - LA DevOps April 2014
The document discusses the introduction and evaluation of Graphite, a scalable real-time graphing system for numeric time-series data managed by carbon. While it highlights the simplicity of sending and retrieving metrics, dashboard creation, and backfilling past metrics, it also notes challenges such as dashboard ownership and data retention complexities. Additionally, the document outlines metric naming conventions and provides setup instructions for using Graphite, aimed at enhancing performance and utility in a real-world application.
More Related Content
Similar to Graphite at CityGrid - LA DevOps April 2014
Graphite at CityGrid - LA DevOps April 2014
- 1. Graphite at CityGrid ifyou can’t measure it, you can’t fix it Wil Heitritter Director, Tech Ops Los Angeles DevOps 2014/04/28
- 2. Magnum esse solem philosophusprobabit, quantus sit mathematicus -Seneca
- 3. Objectives - Introduce Graphiteto new users - Show what we like, what we hate - Present some interesting use-cases - Generate discussion
- 4. Before Graphite Ganglia • Predictableinterface • Text “metrics” to store versions • Slow • Couldn’t pick and choose metrics to see
- 5. Why ganglia sucked -Clusters had to be pre-configured - Multicast vs. Unicast - Data Retention - Static Web Interface (can’t pick and choose) - Static Host List
- 6. What did wethink wanted? Ease of adding metrics Ease of sending metrics Powerful metric display Retain ganglia-style cluster dashboards Long-term configurable metric retention
- 7.
- 8. What is Graphite? ahighly scalable real-time graphing system which collects numeric time-series data is managed by carbon and stored as whisper files and visualized through web interfaces or queried via the API http://graphite.wikidot.com/
- 9. Graphite: what welike Sending metrics is simple Retrieving metrics is simple Dashboard creation and sharing… is simple Many functions() 120MM+ metric values received daily Backfilling past metrics is simple Expandable - different frontends
- 10. Graphite: what sucks Dashboardownership/promotion No ganglia-like standard dashboard Data retention… is NOT as simple as we thought
- 11.
- 12. Metric Naming Business Metrics -These are metrics that are not specific to a specific server - Format: business.${hierarchical}.${path}.${here}.$metric - Example: business.ec2.testaccount.us-east-1a.OnDemand.running.m2.4xlarge
- 13. Metric Naming Server Metrics -These metrics are specific to a particular server (just like ganglia) - Format: servers.${class}.${f_q_d_n}.${metric} - Example: servers.rvw.aws1prdrvw1_subdom_cityg_com.LW_api_reviews_QPS
- 14. Sending metrics Sending directlyfrom metric scripts - /etc/graphite.conf - May need to spread out sending if in volume Collecting from gmond every minute - Metrics are spread out to prevent spiking - False data (gmond acts as a cache)
- 15. Impact of staggeredsending
- 16. Sending is simply... echo$metric $value $timestamp | nc $relay $port
- 17.
- 18. Maintenance Changing retention - whisper-auto-resize.py Fillingholes - whisper-fill $source $destination Backups - Dashboards - Metrics
- 19.
- 20.
- 21.
- 22. Key Metrics Dashboard Examplesof Key Metrics - QPS - Processing Time (Max/Mean/Distribution) - Metrics about sub-requests - Network usage - CPU/load
- 23.
- 25.
- 26. How about PieCharts?
- 28.
- 29.
- 30. Trying it outfor yourself
- 31. Quick Setup Install &Start # pip install https://github.com/graphite-project/ceres/tarball/master # pip install whisper # pip install carbon # pip install graphite-web start it up... send it a metric: echo business.test.metric1 1 `date “+%s”` | nc localhost 2003 OK, it’s almost that easy...
- 32.