This document discusses canary analysis, which is a deployment process where a new change is gradually rolled out to production with checkpoints to examine the new systems versus the old systems and make go/no-go decisions. It proposes using canary analysis to test software releases by routing a small percentage of traffic to new servers and comparing metrics like error rates and requests per second between the new and old servers before fully deploying the new release. The document provides advice on automating canary analysis, focusing on relative metrics, ignoring outliers, balancing fidelity with customer impact, and letting application owners choose when differences are acceptable.

1. Canary Analyze All the
Things
Roy Rapoport
@royrapoport
June 12, 2014
Significant contributions by Chris Sanden, @chris_sanden
1

2. Oh, the Places We’ll Go!
• Introductions
• Proposed Use Case and Definition
• Continuous Improvement / MVP Model
• Issues, Solutions
• Cloud Considerations
• The Road at Netflix
2

3. A Word About Me …
3

4. A Word About Me …
•About 20 years in technology
3

5. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
3

6. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
•Time at Netflix: 1809 days
3

7. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
•Time at Netflix: 1809 days 4y:11m:14d
3

8. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
•Time at Netflix: 1809 days
•At Netflix:
4y:11m:14d
3

9. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
•Time at Netflix: 1809 days
•At Netflix:
•Systems Engineering, Service Delivery in IT/Ops
4y:11m:14d
3

10. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
•Time at Netflix: 1809 days
•At Netflix:
•Systems Engineering, Service Delivery in IT/Ops
•Troubleshooter and Builder of Python Things[tm] in Product
Engineering
4y:11m:14d
3

11. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
•Time at Netflix: 1809 days
•At Netflix:
•Systems Engineering, Service Delivery in IT/Ops
•Troubleshooter and Builder of Python Things[tm] in Product
Engineering
•Current role: Insight Engineering in Product Engineering
4y:11m:14d
3

12. A Word About Me …
•About 20 years in technology
•Systems engineering, networking, software development, QA,
release management
•Time at Netflix: 1809 days
•At Netflix:
•Systems Engineering, Service Delivery in IT/Ops
•Troubleshooter and Builder of Python Things[tm] in Product
Engineering
•Current role: Insight Engineering in Product Engineering
•Real-Time Operational Insight
4y:11m:14d
3

13. A Word About Netflix…
4

14. A Word About Netflix…
Just the Stats
4

15. A Word About Netflix…
•16 years
Just the Stats
4

16. A Word About Netflix…
•16 years
•2000+ employees
Just the Stats
4

17. A Word About Netflix…
•16 years
•2000+ employees
•48 million users
Just the Stats
4

18. A Word About Netflix…
•16 years
•2000+ employees
•48 million users
•5x10^9 hours/quarter
Just the Stats
4

19. A Word About Netflix…
5

20. A Word About Netflix…
Freedom and Responsibility Culture
5

21. A Word About Netflix…
•Optimize speed of innovation
Constrain availability
Cost will be what cost will be
Freedom and Responsibility Culture
5

22. A Word About Netflix…
•Optimize speed of innovation
Constrain availability
Cost will be what cost will be
•Hire smart (experienced)
people
Get out of their way
Freedom and Responsibility Culture
5

23. A Word About Netflix…
•Optimize speed of innovation
Constrain availability
Cost will be what cost will be
•Hire smart (experienced)
people
Get out of their way
•Anti-process bias
Freedom and Responsibility Culture
5

24. A Word About Netflix…
6

25. A Word About Netflix…
Technology and Operations
6

26. A Word About Netflix…
•Service Oriented Architecture
Technology and Operations
6

27. A Word About Netflix…
•Service Oriented Architecture
•Decentralized Operations. You
Technology and Operations
6

28. A Word About Netflix…
•Service Oriented Architecture
•Decentralized Operations. You
•Build
Technology and Operations
6

29. A Word About Netflix…
•Service Oriented Architecture
•Decentralized Operations. You
•Build
•Test
Technology and Operations
6

30. A Word About Netflix…
•Service Oriented Architecture
•Decentralized Operations. You
•Build
•Test
•Deploy
Technology and Operations
6

31. A Word About Netflix…
•Service Oriented Architecture
•Decentralized Operations. You
•Build
•Test
•Deploy
•Set up alerting and monitoring
Technology and Operations
6

32. A Word About Netflix…
•Service Oriented Architecture
•Decentralized Operations. You
•Build
•Test
•Deploy
•Set up alerting and monitoring
•Wake up at 2AM
Technology and Operations
6

33. Oh, the Places We’ll Go!
• Introductions
• Proposed Use Case and Definition
• Continuous Improvement / MVP Model
• Issues, Solutions
• Cloud Considerations
• The Road at Netflix
7

34. Why Canary Analysis?
8

35. So You’ve Just Done a Release
9

36. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/cat
9

37. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/cat
{“response”: “meow”}
9

38. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/cat
{“response”: “meow”}
9

39. So You’ve Just Done a Release
10

40. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/dog
10

41. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/dog
{“response”: “woof”}
10

42. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/dog
{“response”: “woof”}
10

43. So You’ve Just Done a Release
11

44. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/fox
11

45. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/fox
{“response”: “wa-pa-pa-pa-pa-pa-pow”}
11

46. So You’ve Just Done a Release
> curl http://WhatDoesTheFooSay.prod.netflix.net/api/v1/fox
{“response”: “wa-pa-pa-pa-pa-pa-pow”}
The correct answer to “what does the fox say?” is left an exercise for the reader
11

47. You Need Better Testing!
12

48. You Need Better Testing!
Well, yeah
12

49. You Need Better Testing!
“I’m going to push to production, though
I’m pretty sure it’s going to kill the system”
13
- Said no one, ever*
* Hopefully

50. Rate of Change
1 10 100 1000
0
1
2
3
4
5
6
Availability(nines)
Detour
Rate of Change vs Availability
14

51. Rate of Change
1 10 100 1000
0
1
2
3
4
5
6
Availability(nines)
Detour
Rate of Change vs Availability
14

52. Rate of Change
1 10 100 1000
0
1
2
3
4
5
6
Availability(nines)
Detour
Rate of Change vs Availability
14

53. Rate of Change
1 10 100 1000
0
1
2
3
4
5
6
Availability(nines)
Detour
Rate of Change vs Availability
14

54. Rate of Change
1 10 100 1000
0
1
2
3
4
5
6
Availability(nines)
Detour
Rate of Change vs Availability
14

55. Rate of Change
1 10 100 1000
0
1
2
3
4
5
6
Availability(nines)
Detour
Rate of Change vs Availability
Operations
Engineering
14

56. You Need Better Testing!Deployments!
Canary Analysis!
!
• A deployment process where
• a new change (in behavior, code, or both)
• is rolled out into production gradually,
• with checkpoints along the way to examine the new (canary) systems
• (optionally versus the old (baseline) systems)
• and make go/no-go decisions.
15

57. Canary Analysis Is Not
16

58. Canary Analysis Is Not
•A replacement for any sort of
software testing
16

59. Canary Analysis Is Not
•A replacement for any sort of
software testing
•A/B Testing
16

60. Canary Analysis Is Not
•A replacement for any sort of
software testing
•A/B Testing
•Releasing 100% to production
and hoping for the best
16

61. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
One Possible Process
17

62. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
1 server
@ 1.0.2
Automated
Canary
Analysis
One Possible Process
17

63. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
10
servers
@ 1.0.2
One Possible Process
17

64. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
1000
servers
@ 1.0.2
One Possible Process
17

65. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
1000
servers
@ 1.0.2
One Possible Process
18

66. Version
Control
System Customers
Build &
Deployment
System
Automated
Canary
Analysis
1000
servers
@ 1.0.2
One Possible Process
18

67. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
1000
servers
@ 1.0.2
One Possible Process
19

68. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
1000
servers
@ 1.0.2
One Possible Process
19

69. Oh, the Places We’ll Go!
• Introductions
• Proposed Use Case and Definition
• Continuous Improvement / MVP Model
• Issues, Solutions
• Cloud Considerations
• The Road at Netflix
20

70. Are We There Yet?
21

71. Are We There Yet?
• We’re not
21

72. Are We There Yet?
• We’re not
• You’re probably not either
21

73. Minimally …
22

74. Minimally …
• Observability
22

75. Minimally …
• Observability
• Partial traffic routing
22

76. Minimally …
• Observability
• Partial traffic routing
• Decision-making
22

77. Better Yet …
23

78. Better Yet …
• Focus on the Goal
23

79. Better Yet …
• Focus on the Goal
• Current Baseline Matters
23

80. Better Yet …
• Focus on the Goal
• Current Baseline Matters
26% fewer errors in canary
23

81. Better Yet …
• Focus on the Goal
• Current Baseline Matters
• Observability segregation
26% fewer errors in canary
23

82. Hold On a Minute!
26% fewer errors in canary
24

83. Hold On a Minute!
26% fewer errors in canary
Mission
Accomplished
24

84. Hold On a Minute!
26% fewer errors in canary
Mission
Accomplished
30% fewer requests handled in canary
25

85. Hold On a Minute!
26

86. Hold On a Minute!
26

87. Hold On a Minute!
27

88. Hold On a Minute!
• Absolute numbers are relatively
unimportant
27

89. Hold On a Minute!
• Absolute numbers are relatively
unimportant
• Relative numbers matter
27

90. Hold On a Minute!
• Absolute numbers are relatively
unimportant
• Relative numbers matter
• Error rate
27

91. Hold On a Minute!
• Absolute numbers are relatively
unimportant
• Relative numbers matter
• Error rate
• RPS per CPU cycle
27

92. Hold On a Minute!
• Absolute numbers are relatively
unimportant
• Relative numbers matter
• Error rate
• RPS per CPU cycle
27

93. Requests Rate Comparison
So You’ve Got Your Graphs requests
28

94. Requests Rate Comparison
So You’ve Got Your Graphs requests
28

95. Requests Rate Comparison
So You’ve Got Your Graphs requests
Type RAM Cores Cost
Baseline m3.medium 3.75GB 3 $.11/hr
Canary m1.small 1.7GB 1 $.06/hr
28

96. So You’ve Got Your Graphs
29

97. Automating …
30

98. Automating …
• Decision
30

99. Automating …
• Decision
• Execution
30

100. A Quick Recap
31

101. A Quick Recap
• Observe
31

102. A Quick Recap
• Observe
• Segregate metrics
31

103. A Quick Recap
• Observe
• Segregate metrics
• Partial deploy
31

104. A Quick Recap
• Observe
• Segregate metrics
• Partial deploy
• Compare to Baseline
31

105. A Quick Recap
• Observe
• Segregate metrics
• Partial deploy
• Compare to Baseline
• Absolutes are never right
31

106. A Quick Recap
• Observe
• Segregate metrics
• Partial deploy
• Compare to Baseline
• Absolutes are never right
• Automate decision
31

107. A Quick Recap
• Observe
• Segregate metrics
• Partial deploy
• Compare to Baseline
• Absolutes are never right
• Automate decision
• Automate execution
31

108. Oh, the Places We’ll Go!
• Introductions
• Proposed Use Case and Definition
• Continuous Improvement / MVP Model
• Issues, Solutions
• Cloud Considerations
• The Road at Netflix
32

109. To Save You Some Time …
Not all
metrics are
created
equal
33

110. To Save You Some Time …
Not all
metrics are
created
equal
Focus on
System and
Application
Metrics
33

111. To Save You Some Time …
Not all
metrics are
created
equal
Focus on
System and
Application
Metrics
Weight by
category
(system,
latency, etc)
33

112. To Save You Some Time …
Outliers are
out, lying
34

113. To Save You Some Time …
Outliers are
out, lying
Use a group
of servers
34

114. To Save You Some Time …
Outliers are
out, lying
Use a group
of servers
Balance
fidelity with
customer
impact
34

115. To Save You Some Time …
Exercise
without
warmup
can result
in injury
35

116. To Save You Some Time …
Exercise
without
warmup
can result
in injury
Repeat
canary
analysis
frequently
35

117. To Save You Some Time …
Exercise
without
warmup
can result
in injury
Repeat
canary
analysis
frequently
Both traffic
and startup
time are
factors
35

118. To Save You Some Time …
vive la
différence!
36

119. To Save You Some Time …
vive la
différence!
Hot-OK,
Cold-OK
36

120. To Save You Some Time …
vive la
différence!
Hot-OK,
Cold-OK
Let
Application
Owners
Choose
36

121. To Save You Some Time …
Signal is better
than no1$#[NO
CARRIER]
37

122. To Save You Some Time …
Signal is better
than no1$#[NO
CARRIER]
Ignore weak
signals
37

123. Oh, the Places We’ll Go!
• Introductions
• Proposed Use Case and Definition
• Continuous Improvement / MVP Model
• Issues, Solutions
• Cloud Considerations
• The Road at Netflix
38

124. Good News
39

125. Good News
39

126. Good News
• Software-Defined Everything
39

127. Good News
• Software-Defined Everything
• Incremental Pricing
39

128. Bad News
40

129. Bad News
40

130. Bad News
• Capacity Management
40

131. Bad News
• Capacity Management
• Unpredictable Inconsistency
40

132. Oh, the Places We’ll Go!
• Introductions
• Proposed Use Case and Definition
• Continuous Improvement / MVP Model
• Issues, Solutions
• Cloud Considerations
• The Road at Netflix
41

133. Numbers
42

134. Numbers
• 752 services in production
42

135. Numbers
• 752 services in production
• In-house telemetry platform
42

136. Numbers
• 752 services in production
• In-house telemetry platform
• A few metrics
42

137. Numbers
• 752 services in production
• In-house telemetry platform
• A few metrics
42

138. 43

139. Been there.
Done that.
Manually. Artisanally.
43

140. Been there.
• Started in the Data Center
Done that.
Manually. Artisanally.
43

141. Been there.
• Started in the Data Center
• Manual, dashboard-driven
Done that.
Manually. Artisanally.
43

142. Been there.
Done that.
Manually.
44
CPURequestsErrors

143. Been there.
Done that.
Manually.
45

144. Been there.
Done that.
Manually.
45

145. Been there.
Done that.
Manually.
46

146. Been there.
Done that.
Manually.
46

147. Been there.
Done that.
Manually.
47

148. Been there.
Done that.
Manually.
47

149. Been there.
Done that.
Manually.
48

150. Been there.
Done that.
Manually.
• Context vs Precision
48

151. Been there.
Done that.
Manually.
• Context vs Precision
• No …
48

152. Been there.
Done that.
Manually.
• Context vs Precision
• No …
• Repeatability
48

153. Been there.
Done that.
Manually.
• Context vs Precision
• No …
• Repeatability
• Trending
48

154. Been there.
Done that.
Manually.
• Context vs Precision
• No …
• Repeatability
• Trending
• Manual effort is manual
48

155. So Now What?
49

156. So Now What?
• Automate Analysis
49

157. So Now What?
• Automate Analysis
• Took Some Effort
49

158. So Now What?
• Automate Analysis
• Took Some Effort
• Approach and analytics
49

159. So Now What?
• Automate Analysis
• Took Some Effort
• Approach and analytics
• Presentation matters
49

160. Automated Canary Analysis
50

161. Automated Canary Analysis
51

162. Automated Canary Analysis
51

163. Automated Canary Analysis
52

164. Automated Canary Analysis
53

165. Automated Canary Analysis
54

166. For Our Next Trick …
55

167. For Our Next Trick …
• Configuration GUI
55

168. For Our Next Trick …
• Configuration GUI
• Deployment System Integration
55

169. For Our Next Trick …
• Configuration GUI
• Deployment System Integration
• ACA All The Things
55

170. For Our Next Trick …
• Configuration GUI
• Deployment System Integration
• ACA All The Things
• OpenConnect firmware updates
55

171. For Our Next Trick …
• Configuration GUI
• Deployment System Integration
• ACA All The Things
• OpenConnect firmware updates
• Client software changes
55

172. For Our Next Trick …
• Configuration GUI
• Deployment System Integration
• ACA All The Things
• OpenConnect firmware updates
• Client software changes
• Configuration changes in production
55

173. Summary
56

174. Summary
• Canary Analysis makes your changes
56

175. Summary
• Canary Analysis makes your changes
• Safer
56

176. Summary
• Canary Analysis makes your changes
• Safer
• Faster
56

177. Summary
• Canary Analysis makes your changes
• Safer
• Faster
• Easier
56

178. Summary
• Canary Analysis makes your changes
• Safer
• Faster
• Easier
• Most people can start doing it
56

179. Summary
• Canary Analysis makes your changes
• Safer
• Faster
• Easier
• Most people can start doing it
• Everyone can do it better
56

180. Summary
• Canary Analysis makes your changes
• Safer
• Faster
• Easier
• Most people can start doing it
• Everyone can do it better
56

181. • https://www.flickr.com/photos/cseeman
• https://www.flickr.com/photos/ransomtech
• https://www.flickr.com/photos/dougbrown47
• https://www.flickr.com/photos/andresthor/
• https://www.flickr.com/photos/dougbrown47
• https://www.flickr.com/photos/pkdesigns
Questions, Attributions, Feedback
57

182. • https://www.flickr.com/photos/cseeman
• https://www.flickr.com/photos/ransomtech
• https://www.flickr.com/photos/dougbrown47
• https://www.flickr.com/photos/andresthor/
• https://www.flickr.com/photos/dougbrown47
• https://www.flickr.com/photos/pkdesigns
Questions, Attributions, Feedback
@royrapoport
57

183. • https://www.flickr.com/photos/cseeman
• https://www.flickr.com/photos/ransomtech
• https://www.flickr.com/photos/dougbrown47
• https://www.flickr.com/photos/andresthor/
• https://www.flickr.com/photos/dougbrown47
• https://www.flickr.com/photos/pkdesigns
Questions, Attributions, Feedback
@royrapoport
?57