LSPE Presentation: Actionable Metrics at Netflix

3,277 views

Published on

2012-06-27 presentation to the Large Scale Production Engineering meetup on Netflix's approach to metrics

Published in: Technology, Business
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,277
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
35
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

LSPE Presentation: Actionable Metrics at Netflix

  1. 1. Actionable MetricsEnabling Decision-Making in Netflix’s Decentralized Environment #lspe June 27, 2012 Roy Rapoport @royrapoport, rsr@netflix.com 1
  2. 2. Me• Been in tech for about 20 years• Systems engineering, networking, software development, QA, release management• Time at Netflix: 1094 days (3y-2d)• (Current) job at Netflix: Make things better (Security Monkey, Python Platform, Central Alert Gateway, ... ) 2
  3. 3. Metrics Humor 3I want to start with a joke. This is may be the world’s longest joke, given that I started tellingit about a year ago.I had just attended a presentation at Velocity 2011 where someone said “collect all themetrics you possibly can, because you don’t know what will prove useful.”Exposing the timeline, see that I’ve got more than a year’s information here.And that the numbers are actually constrained within a very small range.And that I’m showing the percent of our instances in our production environment that haveeven -- that is, divisible by two -- public IP addresses.
  4. 4. Metrics Humor 3I want to start with a joke. This is may be the world’s longest joke, given that I started tellingit about a year ago.I had just attended a presentation at Velocity 2011 where someone said “collect all themetrics you possibly can, because you don’t know what will prove useful.”Exposing the timeline, see that I’ve got more than a year’s information here.And that the numbers are actually constrained within a very small range.And that I’m showing the percent of our instances in our production environment that haveeven -- that is, divisible by two -- public IP addresses.
  5. 5. Metrics Humor % of instances with even public IP addresses 3I want to start with a joke. This is may be the world’s longest joke, given that I started tellingit about a year ago.I had just attended a presentation at Velocity 2011 where someone said “collect all themetrics you possibly can, because you don’t know what will prove useful.”Exposing the timeline, see that I’ve got more than a year’s information here.And that the numbers are actually constrained within a very small range.And that I’m showing the percent of our instances in our production environment that haveeven -- that is, divisible by two -- public IP addresses.
  6. 6. Technology Overview 4Going into the cloud, went from heavy multi-purpose stacks to a highly distributed SOAenvironment.Tons of different services, dynamic binding and communication (no ESB)
  7. 7. Technology Overview • SoA, REST, Mostly Java 4Going into the cloud, went from heavy multi-purpose stacks to a highly distributed SOAenvironment.Tons of different services, dynamic binding and communication (no ESB)
  8. 8. Technology Overview • SoA, REST, Mostly Java • Simple overall architecture: 4Going into the cloud, went from heavy multi-purpose stacks to a highly distributed SOAenvironment.Tons of different services, dynamic binding and communication (no ESB)
  9. 9. Technology Overview • SoA, REST, Mostly Java • Simple overall architecture: 4Going into the cloud, went from heavy multi-purpose stacks to a highly distributed SOAenvironment.Tons of different services, dynamic binding and communication (no ESB)
  10. 10. Culture Overview 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.
  11. 11. Culture Overview • Freedom and Responsibility 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.
  12. 12. Culture Overview • Freedom and Responsibility • Distributed Operations 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.
  13. 13. Culture Overview • Freedom and Responsibility • Distributed Operations • Get out of the way of Developers 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.
  14. 14. Culture Overview • Freedom and Responsibility • Distributed Operations • Get out of the way of Developers 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.
  15. 15. The Metric Lifecycle 6
  16. 16. The Metric Lifecycle• Send 6
  17. 17. The Metric Lifecycle•Send• Look 6
  18. 18. The Metric Lifecycle•Send• Look• Alert 6
  19. 19. Systems • Flexible • Scalable • Self-Service 7Developers own sending metricsDevelopers specify what metrics to sendSmart aggregation (critical given churn potential)
  20. 20. Telemetry Flexible, Scalable, Self-Service 8On the fly definition of metricsVery low barrier to entryJava, Python, Perl (and if someone wanted to, they could -- pretty easily -- create a bashinterface)
  21. 21. Telemetry Flexible, Scalable, Self-Service import netflix.metrics as NM [...] self.nm = NM.Metrics("core_cag") [...] def api(self): self.nm.counter("api") [...] app_label = “application_%s” % application self.nm.counter(app_label) [...] 8On the fly definition of metricsVery low barrier to entryJava, Python, Perl (and if someone wanted to, they could -- pretty easily -- create a bashinterface)
  22. 22. Visualization Flexible, Scalable, Self-Service 9GUI helper for creating graphs, URL-driven fetchingThree engines: highcharts, dygraphs, RRDflexible ‘today vs some other time’ capability
  23. 23. Alerting Flexible, Scalable, Self-Service 10For some patterns -- e.g. traffic -- you’ve got to have the ability to have your thresholds bedynamic. We played with Holt-Winters, but found that it was expensive, and took too long tocalibrate. We’ve found that Double Exponential Smoothing has worked really well for us.But once you start doing more interesting alerting configuration, it’s harder to know whetheror not you’ve set your thresholds correctly -- and the ability to map your alert configurationto historical metric values to see when your alert WOULD have triggered makes a hugedifference in making your first attempt at rational threasholds most likely to be successful.
  24. 24. Alerting Flexible, Scalable, Self-Service • Static vs Dynamic Thresholds 10For some patterns -- e.g. traffic -- you’ve got to have the ability to have your thresholds bedynamic. We played with Holt-Winters, but found that it was expensive, and took too long tocalibrate. We’ve found that Double Exponential Smoothing has worked really well for us.But once you start doing more interesting alerting configuration, it’s harder to know whetheror not you’ve set your thresholds correctly -- and the ability to map your alert configurationto historical metric values to see when your alert WOULD have triggered makes a hugedifference in making your first attempt at rational threasholds most likely to be successful.
  25. 25. Alerting Flexible, Scalable, Self-Service • Static vs Dynamic Thresholds • Historical Testing 10For some patterns -- e.g. traffic -- you’ve got to have the ability to have your thresholds bedynamic. We played with Holt-Winters, but found that it was expensive, and took too long tocalibrate. We’ve found that Double Exponential Smoothing has worked really well for us.But once you start doing more interesting alerting configuration, it’s harder to know whetheror not you’ve set your thresholds correctly -- and the ability to map your alert configurationto historical metric values to see when your alert WOULD have triggered makes a hugedifference in making your first attempt at rational threasholds most likely to be successful.
  26. 26. For Example ... Last 3 hours’ core_tools.core_cag_api What the ... 11core_tools.core_cag_api is the alert volume through our Central Alerting Gateway (CAG). Iwent to look for this graph for this presentation when noticed we had dropped our volumesignificantly over the last 20 minutes (the relatively flat last part of this graph). So Iexpanded the time range to the last few days ...
  27. 27. For Example ... Visualization (Continued) Last 4 days’ core_tools.core_cag_api even more questions! 12Which just raised more questions -- like what happened with the drop in alerts on Monday,at 11AM and 11PM?So let’s expand the range further back -- for the last two weeks ...
  28. 28. For Example ... Visualization (Continued) Last 10 days’ core_tools.core_cag_api What caused the spike? 13OK, that looks like basically we had a spike in alerts starting as of about 10 days ago or so,so the drops on Monday were just going back to normal volume. But what caused the spikeearlier?The good news is that since we send metrics not just for alerts but for alerts per application(see the earlier code example), we could see alert volume per application...
  29. 29. For Example ... Visualization (Continued) Show alert volume per application Someone had a rough few days... 14The purple line is alerts for one of our applications -- which clearly had had a pretty roughfew days.Now that I had the answers, let’s make sure we alert on alert volume ...
  30. 30. Don’t Like Surprises...{ "alerts": [ { "applyTo": "cluster", "condition": { "minPercent": 90.0, "noise" : .2, "maxPercent": 25.0, "type": "DoubleExponential" }, // here’s my number overrides : { ‘api_key’ : ‘93528d3baa599b727097d73cfdbd5934’ } "metricName": "core_cag_api", // so call me maybe "severity": "major" } ], "clusters": [ "core_tools" ]} 15
  31. 31. I Didn’t Mention • End-to-end testing and alerting • External availability and performance • Events • Open Connect • Jobs 16Things I didn’t talk about:* end-to-end testing and visibility into transactions through our system;* making sure our site’s available to the world;* how we’ve promoted events into first-class monitored objects in our environment;* monitoring our new CDN, the Open Connect platform;* the jobs we have open, at http://jobs.netflix.com
  32. 32. I Can Haz Question? (Photo credit: My wife) 17

×