In a lab we don’t measure misery. When we measure performance, say, in benchmarking, a lot of people focus on what’s going well. The lowest latencies. The highest throughputs. But to truly understand how things are going to work in production we have to measure our systems operating at their worst. We also have to narrow down to what operationally works, and what is a waste of time. Is P99 even a useful metric? Or do we need to track event anomalies per 100,000? Learn from Gil Tene, CEO of Azul Systems, how to stop worrying and learn to love misery.

1. Brought to you
by
Misery Metrics and
Consequences
Gil Tene
CTO & co-Founder, Azul
Systems @giltene

2. Misery Metrics
and
Consequences
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

3. Agenda
Some intro stuff
How much of monitoring is simply broken
How it’s so bad that there is no fixing it
lights and tunnels
appreciating misery

4. Understanding Latency
and
Application Responsiveness
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

5. (Micro?)services
and
Response Time
Behavior
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

6. How NOT to
Measure Latency
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

7. The “Oh S@%#!”
talk
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

8. How I learned
to
stop worrying
and
love Misery
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

9. How I learned
to
stop worrying
and
love Misery
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

10. (Micro)services

11. Source: “Visualizing and Tracking your Microservices”, AppDynamics blog

12. Source: “Architecting for Speed: How agile innovators accelerate growth through microservices”, Jesper Nordstrom & Tomas Betzholtzblog, 3gamma.com

13. Source: “Microservices at Amazon”, Alan Ho (@karlunho), apigee developer blog, image from circa 2009

14. This is a Microservice

15. This is a Microservices
Architecture

20. ?
?
?

21. Source: “Netflix OSS, Spring Cloud, or Kubernetes? How About All of Them!”, Christian Posta, DZone

26. Remember: all I'm offering is the truth.

27. We like to look at pretty charts…
95%’lie
The “We only want to show good things” chart

29. What you find when you dig deeper into the 9s

30. What you find when you dig deeper into the 9s

34. What (TF) does the Average
of the 95%’lie mean?

35. What (TF) does the Average
of the 95%’lie mean?
Lets do the same with 100%’ile; Suppose we have a set of
100%’ile values for each minute:
[1, 0, 3, 1, 601, 4, 2, 8, 0, 3, 3, 1, 1, 0, 2]
“The average 100%’ile over the past 15 minutes was 42”
Same nonsense applies to averaging any other %’lie

36. #LatencyTipOfTheDay:
You can't average percentiles.
Period.

37. What percentile levels do YOU monitor?

38. 99%’lie: a good indicator, right?
What are the chances of a single web page view
experiencing >99%’lie latency of:
- A single search engine node?
- A single Key/Value store node?
- A single Database node?
- A single CDN request?
- A single (micro)service call?

41. #LatencyTipOfTheDay:
MOST page loads will experience worse
than the 99%'lie server response
So why are we measuring it?

42. Gauging user experience
Example: If a typical user session involves 5 page loads,
averaging 40 resources per page.
-How many of our users will NOT experience something
worse than the 95%’lie of http requests?
Answer: ~0.003%
-How many of our users will NOT experience something
worse than the 99%’lie of http requests?
Answer: ~13%

43. Gauging user experience
Example: If a typical user session involves 5 page loads,
averaging 40 resources per page.
-What http response percentile will be experienced by the
95%’ile of users?
Answer: ~99.97%
-What http response percentile will be experienced by the
99%’ile of users
Answer: ~99.995%

44. Why don’t we have response time
or latency stats with multiple 9s
in them???

45. You can’t average percentiles…
And you also can’t get an hour’s
99.999%’lie out of lots
of 10 second interval 99%’lie
reports…
Why don’t we have response time
or latency stats with multiple 9s
in them???

46. You can’t average percentiles…
And you also can’t get an hour’s 99.999%’lie
out of lots
of 10 second interval 99%’lie reports…
Check out HdrHistogram
It lets you have nice things….
Why don’t we have response time or
latency stats with multiple 9s in
them???
0% 90% 99% 99.99% 99.999% 99.9999%
140
120
100
80
60
40
20
0
Hiccup
Duration
(msec)
99.9%
Percentile
Hiccups by Percentile Distribution
Hiccups by Percentile SLA

47. The coordinated omission
problem
An accidental conspiracy.
.
The lie in the 99%’lies

48. Coordinated Omission in Monitoring Code
Long operations only get measured once
delays outside of timing window do not get measured at all

49. How bad can this get?
Avg. is 1
msec over 1st
100 sec
System
Stalled for
100 Sec
System easily
handles 100
requests/sec
Responds to
each in
1msec
How would you characterize this
system?
Elapsed Time
~75%‘ile is 50
sec
~50%‘ile is 1
msec
99.99%‘ile is
~100sec
Avg. is 50 sec.
over next 100
sec
Overall Average response time is ~25
sec.

50. Measurement in practice
System
Stalled for
100 Sec
System easily
handles 100
requests/sec
Responds to
each in
1msec
What actually gets
measured?
Elapsed Time
75%‘lie is 1
msec
99.99%‘lie is 1
msec
50%‘ile is 1
msec
10,000
measurements
@ 1 msec each
1
measurement
@ 100 sec
Overall Average is 10.9 msec
(!!!)

51. Proper measurement
System
Stalled for
100 Sec
Elapsed
Time
System easily
handles 100
requests/sec
Responds to
each in
1msec
10,000
results
Varying
linearly from
100 sec
to 10 msec
10,000
results @ 1
msec each
~50%‘ile is 1
msec
~75%‘ile is 50
sec
99.99%‘ile is
~100sec

52. Proper measurement
System
Stalled for
100 Sec
Elapsed
Time
System easily
handles 100
requests/sec
Responds to
each in
1msec
10,000
results
Varying
linearly from
100 sec
to 10 msec
10,000
results @ 1
msec each
~50%‘ile is 1
msec
~75%‘ile is 50
sec
99.99%‘ile is
~100sec
Coordinated
Omission
1 msec 1 msec
1 result
@ 100
sec

53. “Better” can look “Worse”
System easily
handles 100
requests/sec
Responds to
each in
1msec
10,000 @
1msec
10,000 @ 5
msec
System
Slowed for
100 Sec
Still easily
handles 100
requests/sec
50%‘ile is 1
msec
99.99%‘lie is
~5msec
Responds to
each in 5
msec
Elapsed Time
75%‘lie is
2.5msec

54. Service Time vs. Response
Time

55. Response Time vs. Service Time @60K/sec

56. How “real” people react

58. How does the world
keep working ?!?!?

59. Luckily, I think that there is
a rainbow at the end of the
tunnel…

60. How does the world
even keep working
?!?!?

61. Something we are
doing actually works…

62. Something we are
doing actually works…

64. 3000 BC
Graduated with honors
Witch Doctor Academy

70. Timeout
s
Retries Misses
Abandoned
Shopping Carts
Loss of
Engagemen
t
Failed
Queries
Angry
Phone Calls

71. It is nonsense to ask:
“was my 99%’lie a timeout?”
Instead Ask:
“How many timeouts
did I have?”
&
“What % of my
requests timed out?”

72. TransactionLoad
Success Rate
Looking at misery

73. Focus on measuring Misery

74. @gilten
Love
Misery

75. Love
Misery
Don’t aim to make a specific %’ile acceptable
Instead, aim for an acceptable %
of misery
final #LatencyTipOfTheDay:

76. Brought to you
by
Gil Tene
@gilten
e