LSPE Presentation: Actionable Metrics at Netflix

Actionable MetricsEnabling Decision-Making in Netﬂix’s Decentralized Environment #lspe June 27, 2012 Roy Rapoport @royrapoport, rsr@netﬂix.com 1

Me• Been in tech for about 20 years• Systems engineering, networking, software development, QA, release management• Time at Netﬂix: 1094 days (3y-2d)• (Current) job at Netﬂix: Make things better (Security Monkey, Python Platform, Central Alert Gateway, ... ) 2

Metrics Humor 3I want to start with a joke. This is may be the world’s longest joke, given that I started tellingit about a year ago.I had just attended a presentation at Velocity 2011 where someone said “collect all themetrics you possibly can, because you don’t know what will prove useful.”Exposing the timeline, see that I’ve got more than a year’s information here.And that the numbers are actually constrained within a very small range.And that I’m showing the percent of our instances in our production environment that haveeven -- that is, divisible by two -- public IP addresses.

Metrics Humor % of instances with even public IP addresses 3I want to start with a joke. This is may be the world’s longest joke, given that I started tellingit about a year ago.I had just attended a presentation at Velocity 2011 where someone said “collect all themetrics you possibly can, because you don’t know what will prove useful.”Exposing the timeline, see that I’ve got more than a year’s information here.And that the numbers are actually constrained within a very small range.And that I’m showing the percent of our instances in our production environment that haveeven -- that is, divisible by two -- public IP addresses.

Technology Overview 4Going into the cloud, went from heavy multi-purpose stacks to a highly distributed SOAenvironment.Tons of different services, dynamic binding and communication (no ESB)

Technology Overview • SoA, REST, Mostly Java 4Going into the cloud, went from heavy multi-purpose stacks to a highly distributed SOAenvironment.Tons of different services, dynamic binding and communication (no ESB)

Technology Overview • SoA, REST, Mostly Java • Simple overall architecture: 4Going into the cloud, went from heavy multi-purpose stacks to a highly distributed SOAenvironment.Tons of different services, dynamic binding and communication (no ESB)

Culture Overview 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.

Culture Overview • Freedom and Responsibility 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.

Culture Overview • Freedom and Responsibility • Distributed Operations 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.

Culture Overview • Freedom and Responsibility • Distributed Operations • Get out of the way of Developers 5Freedom and Responsibility means hiring smart people and assuming they’ll make smart decisions. We want to empower them to do whatever the heckthey think they need to be doing to make the business succeed, which also means giving them all the tools THEY say they need to be successful. We biasfor innovation speed rather than safety, and we try to create highly aligned, but very loosely coupled, teams.Distributed ops means No NOC. No staring at dashboards. CORE - Cross-functional, tools buildersBottom line, it’s about getting out of the way of developers. We give them the tools they need, and the infrastructure pieces to make building whatever theywant to build easier, and then stand back and let them create whatever they want -- even and especially when what they want to build is, for example ...A small moon.

The Metric Lifecycle 6

The Metric Lifecycle• Send 6

The Metric Lifecycle•Send• Look 6

The Metric Lifecycle•Send• Look• Alert 6

Systems • Flexible • Scalable • Self-Service 7Developers own sending metricsDevelopers specify what metrics to sendSmart aggregation (critical given churn potential)

Telemetry Flexible, Scalable, Self-Service 8On the ﬂy deﬁnition of metricsVery low barrier to entryJava, Python, Perl (and if someone wanted to, they could -- pretty easily -- create a bashinterface)

Telemetry Flexible, Scalable, Self-Service import netflix.metrics as NM [...] self.nm = NM.Metrics("core_cag") [...] def api(self): self.nm.counter("api") [...] app_label = “application_%s” % application self.nm.counter(app_label) [...] 8On the ﬂy deﬁnition of metricsVery low barrier to entryJava, Python, Perl (and if someone wanted to, they could -- pretty easily -- create a bashinterface)

Visualization Flexible, Scalable, Self-Service 9GUI helper for creating graphs, URL-driven fetchingThree engines: highcharts, dygraphs, RRDﬂexible ‘today vs some other time’ capability

Alerting Flexible, Scalable, Self-Service 10For some patterns -- e.g. traffic -- you’ve got to have the ability to have your thresholds bedynamic. We played with Holt-Winters, but found that it was expensive, and took too long tocalibrate. We’ve found that Double Exponential Smoothing has worked really well for us.But once you start doing more interesting alerting configuration, it’s harder to know whetheror not you’ve set your thresholds correctly -- and the ability to map your alert configurationto historical metric values to see when your alert WOULD have triggered makes a hugedifference in making your first attempt at rational threasholds most likely to be successful.

Alerting Flexible, Scalable, Self-Service • Static vs Dynamic Thresholds 10For some patterns -- e.g. traffic -- you’ve got to have the ability to have your thresholds bedynamic. We played with Holt-Winters, but found that it was expensive, and took too long tocalibrate. We’ve found that Double Exponential Smoothing has worked really well for us.But once you start doing more interesting alerting configuration, it’s harder to know whetheror not you’ve set your thresholds correctly -- and the ability to map your alert configurationto historical metric values to see when your alert WOULD have triggered makes a hugedifference in making your first attempt at rational threasholds most likely to be successful.

Alerting Flexible, Scalable, Self-Service • Static vs Dynamic Thresholds • Historical Testing 10For some patterns -- e.g. traffic -- you’ve got to have the ability to have your thresholds bedynamic. We played with Holt-Winters, but found that it was expensive, and took too long tocalibrate. We’ve found that Double Exponential Smoothing has worked really well for us.But once you start doing more interesting alerting configuration, it’s harder to know whetheror not you’ve set your thresholds correctly -- and the ability to map your alert configurationto historical metric values to see when your alert WOULD have triggered makes a hugedifference in making your first attempt at rational threasholds most likely to be successful.

For Example ... Last 3 hours’ core_tools.core_cag_api What the ... 11core_tools.core_cag_api is the alert volume through our Central Alerting Gateway (CAG). Iwent to look for this graph for this presentation when noticed we had dropped our volumesigniﬁcantly over the last 20 minutes (the relatively ﬂat last part of this graph). So Iexpanded the time range to the last few days ...

For Example ... Visualization (Continued) Last 4 days’ core_tools.core_cag_api even more questions! 12Which just raised more questions -- like what happened with the drop in alerts on Monday,at 11AM and 11PM?So let’s expand the range further back -- for the last two weeks ...

For Example ... Visualization (Continued) Last 10 days’ core_tools.core_cag_api What caused the spike? 13OK, that looks like basically we had a spike in alerts starting as of about 10 days ago or so,so the drops on Monday were just going back to normal volume. But what caused the spikeearlier?The good news is that since we send metrics not just for alerts but for alerts per application(see the earlier code example), we could see alert volume per application...

For Example ... Visualization (Continued) Show alert volume per application Someone had a rough few days... 14The purple line is alerts for one of our applications -- which clearly had had a pretty roughfew days.Now that I had the answers, let’s make sure we alert on alert volume ...

Don’t Like Surprises...{ "alerts": [ { "applyTo": "cluster", "condition": { "minPercent": 90.0, "noise" : .2, "maxPercent": 25.0, "type": "DoubleExponential" }, // here’s my number overrides : { ‘api_key’ : ‘93528d3baa599b727097d73cfdbd5934’ } "metricName": "core_cag_api", // so call me maybe "severity": "major" } ], "clusters": [ "core_tools" ]} 15

I Didn’t Mention • End-to-end testing and alerting • External availability and performance • Events • Open Connect • Jobs 16Things I didn’t talk about:* end-to-end testing and visibility into transactions through our system;* making sure our site’s available to the world;* how we’ve promoted events into ﬁrst-class monitored objects in our environment;* monitoring our new CDN, the Open Connect platform;* the jobs we have open, at http://jobs.netﬂix.com

I Can Haz Question? (Photo credit: My wife) 17

LSPE Presentation: Actionable Metrics at Netflix

by royrapoport on Jul 02, 2012

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 13

Statistics