Better service monitoring through histograms
•Download as PPTX, PDF•
2 likes•1,871 views
This document discusses using histograms and percentiles to better monitor service performance. It begins by noting the limitations of synthetic monitoring and outlines how real user data can provide a more accurate picture. Percentiles like the median and 90th percentile are explained as useful metrics for understanding performance. Histograms of request latency data over time are presented as a way to detect non-normal distributions that could indicate issues. Calculating alerting thresholds based on percentiles rather than averages is advocated to avoid missing multiple high samples. Examples are given of how percentile-based alerting can more effectively detect performance problems and avoid unnecessary alerts.
Recommended
More Related Content
Similar to Better service monitoring through histograms
Similar to Better service monitoring through histograms (20)
More from Fred Moyer
More from Fred Moyer (19)
Recently uploaded
Recently uploaded (20)
Better service monitoring through histograms
- 1. Better service monitoring through histograms Fred Moyer - @phredmoyer San Francisco Perl Mongers, 07-26-2016
- 2. Systems break while we sleep How often are you woken up for false alarms? Welcome
- 3. Synthetics Easy to setup, but not a real user
- 4. Synthetics Stephen Falken: Uh, uh, General, what you see on these screens up here is a fantasy; a computer-enhanced hallucination. Those blips are not real missiles. They're phantoms. (War Games, 1983)
- 5. Real Users These are your users, right?
- 7. 500 ms is really 2,000 ms Spike Erosion
- 8. What threshold do you choose? Threshold Alerting
- 9. “Alert me if requests take longer than 200 ms” 10,10,10,10,10,10,10,10,10,5000 Alerts on one outlier in 10 Threshold Alerting
- 10. “Alert if request average over one minute is longer than 200 ms” avg(10,10,210,210,210,210) = 143 (860/6) Does not alert on multiple high samples Threshold Alerting
- 11. ‘average’ eq ‘arithmetic mean’ A=S/N A = average N = the number of terms S = the sum of the numbers in the set Math Refresher
- 12. median = midpoint of data set The 50th percentile is 555 - q(0.5) Value 111 222 333 444 555 666 777 888 999 Sample # 1 2 3 4 5 6 7 8 9 Math Refresher
- 13. 90th percentile - 90% of samples below it The 90th percentile is 1,000 - q(0.9) Value 111 222 333 444 555 666 777 888 999 1,000 1,111 Sample # 1 2 3 4 5 6 7 8 9 10 11 Math Refresher
- 14. 100th Percentile - the maximum value The 100th percentile is 1,111 - q(1) Value 111 222 333 444 555 666 777 888 999 1,000 1,111 Sample # 1 2 3 4 5 6 7 8 9 10 11 Math Refresher
- 16. Sample value Number of samples Normal Distribution
- 17. Sample value Number of samples Normal Distribution 34% within one sigma (σ)
- 18. Sample value Number of samples Non-Normal Distribution
- 19. Sample value Number of samples Non-Normal Distribution
- 20. Non-Normal Distribution Operations data groups at different points
- 21. Non-Normal Distribution Users to the right of the red line are gone
- 22. Request latency “We keep hearing from people that the website is slow. But it is fine when we test it, and the request latency graph is constant” You are only looking at part of the picture.
- 23. Heat Map Histograms over time windows
- 24. Percentiles
- 25. Practical Percentiles Bandwidth usage is often billed at 95th percentile usage Record 5 minute data usage intervals Sort samples by value of sample Throw out the highest 5% of samples Charge usage based on the remaining top sample, i.e. 300 MB transferred over 5 minutes = 1 MB/s rate billing
- 26. Practical Percentiles If I measure 95th percentile per 5 minutes all month long, I CANNOT calculate 95th percentile over the month.
- 27. Angry users How many users are you pissing off?
- 28. Angry users
- 29. “Alert me if request latency 90th percentile over one minute is exceeded” Percentile based alerting q(0.9)[10,10,10,10,10,10,10,10,5000] == 10 Alert IS NOT triggered Do you want to be woken up for this? NO!
- 30. “Alert me if request latency 90th percentile over one minute is exceeded” Percentile based alerting q(0.9)[10,10,10,10,10,10,250,300] = ~270 Alert IS triggered Do you want to be woken up for this? YES!
- 32. Who’s using this approach? Google.com Circonus.com You?
- 33. Questions? Thanks to Circonus.com for the tools and help with the math http://www.circonus.com/free-account/
Editor's Notes
- A synthetic is basically a bot check against your system. One of the benefits (perhaps the only benefit) of the synthetic is that it’s more highly available than the application you are monitoring. The response from synthetic requests don’t tell you anything meaningful about how actual users experience your application.
- What am I looking at here? This is a time series graph of response times from synthetic login checks against a website. The results are remarkably consistent, as they should be. It gives you the viewpoint of one user - a computer somewhere dispatches a request over the same network route to your server. It records several metrics about how your application responds; time to start the ssl connection, time to the first byte served, average request time... Those metrics are not only useless (unless anyone here runs a service just for one user… in that case, kudos), they lie to you. These are LIES. The falsely represent the health of your application. It’s a binary - is the service up, or is the service down? That’s all you get.
- Your user base will likely have a distribution of ages, genders, devices, network connections.
- The synthetic check used an external user agent, but you can use collection tools like statsd or log analysis to record request times for real users. This is better than only using a synthetic check, but this technique still has a number of shortcomings. The first is that collection data is averaged over an interval (generally 10 seconds to a minute). So if Cyndi, Bobby, and Mike are all shopping at your website at the same time, you only see the average of their request times over a given interval. Mike might be having a great experience because his office network is 100 megabit, but Bobby is on gig-e, and Cyndi on 10 megabit, you’ll only see Bobby’s view of the website user experience.
- The second short shortcoming of a time series average value graph is spike erosion, also known as downsampling. Spike erosion is what happens when you zoom in on specific areas of a time series graph. As you zoom in, the data is averaged over intervals closer to the actual collection intervals. As you can see on this graph, when we zoom into a 2 hour view of the graph we just looked at, the maximum value we see now is 2,000 milliseconds instead of 500 milliseconds. That’s a 400% increase!
- I don’t like this image - find a better one. If you alert based on values you get from the graphs I’ve shown, what value do you alert on? As you’ve seen, avoiding false positives is impossible.
- Correct this since one sample will trigger alert, use average alert instead
- 200 ms is too slow, so we take an average, 66% of population is over 200ms, no alert thrown, this is the solution people use to avoid the outlier in previous slide
- 0th quantile is first element
- 0th quantile is first element
- A histogram is one of the seven basic tools of quality. The Y axis indicates the number of samples, where the X axis indicates the sample value. One use of a histogram that you may have seen is plotting human height vs number of people who are that tall.
- Human height follows what is called a normal distribution (also known as a Gaussian distribution). The majority of the population tends to group around one value, and tapers off at the high and low sample values. With a perfect normal distribution, the arithmetic mean (the average) and the median are one in the same.
- The mode is also equal to the median. You’ve heard the term standard deviation before most likely. With a normal distribution, 68% of the values lie within one standard deviation for both sides of the median. 95% within 2 standard deviations, 99.7% within 3 sigma. The smaller a standard deviation, the closer the data is to the mean. The larger one sigma is, the farther the data is away from the mean. It is important to note that these metrics only make sense for normal distribution, where there is a single mode.
- This is a non normal distribution. In this example, there are large numbers of samples grouped at the highest and lowest sample values. Because there are two distinct peaks, this is called a bimodal distribution (or multi-modal distribution). In a multimodal distribution like this, standard deviation and multi-sigma values are useless.
- This is another non-normal distribution. As you can see, it only has one mode, and is a skewed distribution. Standard deviation has little to no meaning here.
- Here is a histogram of web page request time. The higher the bar, the more users are affected. This is a highly skewed distribution - notice the grouping between the spike at ~150 milliseconds, and the long tail past there. There’s another smaller spike at ~25 ms, so this is mostly a bimodal distribution. In terms of website performance, people will generally get angry if request times take longer than 250 milliseconds. So what we see here is a bunch of users who are getting acceptable response times, and a long tail of pissed off users. People on left side are having a great experience, people on right side are leaving the site.
- Here is a histogram of web page request time. The higher the bar, the more users are affected. This is a highly skewed distribution - notice the grouping between the spike at ~150 milliseconds, and the long tail past there. There’s another smaller spike at ~25 ms, so this is mostly a bimodal distribution. In terms of website performance, people will generally get angry if request times take longer than 250 milliseconds. So what we see here is a bunch of users who are getting acceptable response times, and a long tail of pissed off users. People on left side are having a great experience, people on right side are leaving the site. Note that this is for a time slice, say 5 minutes. What does this look like if we integrate over time?
- Heat maps are visual representations of histograms over time windows. It gives you a visualization of data distributions over time.
- With heat maps, you can add percentile overlays to show the 50th, 95, and any other percentile distribution over time slices
- A percentile is a barrier where to the left the samples are 95%, to the right are the remaining 5%. There is a caveat with the barrier hitting in the middle of data points. If you measure on the right including the barrier, >= 95th percentile of whole data set, if you measure to the left of the barrier, <= 95%. If you have two samples, median is every value between those two samples. Samples on the barrier are counted twice. Divide data set into two sets. Have a slide that says - bespoke things you probably didn’t know about histograms. For the purpose of our examples, we’ll avoid these edge cases. If you see a histogram where the ⅓ quantile and ⅔ quantile are equal value, they add up to > 100%. Histogram of 1 value is one example (everything is measured twice). 1,2 - 1,2,3.
- Percentiles cannot be averaged. You have to calculate them from the raw usage data. There are several monitoring solutions out there that will let you average percentiles - this is flat out WRONG
- What’s your SLA? If you set your 95% percentile at 250 ms, and you meet your SLA, you’re pissing off 5% of your users. They’re going to your competitor. Let’s try to calculate how many users you are screwing.
- Take the number of requests outside your 95 percentile (the 5th percent inverse quantile), and integrate that over time to get a cumulative number of users that you’ve screwed. Multiply that times the dollar value of each lost request - that’s how much money you’re losing.
- Circonus.com allows you to set percentile based alerts, so that you’ll be alerted if users start getting pissed off. Here is a percentile based alert - you can expand that to alert based on number of users pissed off per hour. Or even translate that to a dollar value using CAQL (circonus analytics query language). So you can say ‘alert me if we are losing more than $500 worth of users per hour’. This is something you’ll never be able to do with threshold based alerting. Thus, you can set a limit that is essentially normalized to traffic loads, say holiday sale surges.
- Circonus.com allows you to set percentile based alerts, so that you’ll be alerted if users start getting pissed off. Here is a percentile based alert - you can expand that to alert based on number of users pissed off per hour. Or even translate that to a dollar value using CAQL (circonus analytics query language). So you can say ‘alert me if we are losing more than $500 worth of users per hour’. This is something you’ll never be able to do with threshold based alerting. Thus, you can set a limit that is essentially normalized to traffic loads, say holiday sale surges.
- Circonus.com allows you to set percentile based alerts, so that you’ll be alerted if users start getting pissed off. Here is a percentile based alert - you can expand that to alert based on number of users pissed off per hour. Or even translate that to a dollar value using CAQL (circonus analytics query language). So you can say ‘alert me if we are losing more than $500 worth of users per hour’. This is something you’ll never be able to do with threshold based alerting. Thus, you can set a limit that is essentially normalized to traffic loads, say holiday sale surges.