This talk will cover a security focused project that evolved into a chaos injection system. The system is called “Lifespan Management” and it enforces a lifespan on a cloud hosted VM. After the lifespan expires, the host is terminated, and a replacement is brought up. It has the benefits of making it easier to apply fixes for CVE’s (CVE comes out on day X, we know hosts will age out by day Y), and reducing the value of a compromised machine (“I’ve finally captured a host! It’s being shutdown?? No!”) This seemed simple enough, but the complexity it uncovered made for a fun, year-long adventure in chaos engineering. In this talk, I’ll cover the evolution of the system, and some lessons we learned along the way like: All termination API calls are not created equal Zero failing health checks does not mean a host is healthy Answering “Was this the chaos system?” quickly is essential I’ll also include anecdotes like how it helped with Spectre/Meltdown mitigations, how it mercilessly killed all our kubernetes workers, and how it locked us out of our QA environment.

1. @paulcarletonjr
How our security requirements
turned us into
accidental chaos engineers

2. @paulcarletonjr
“We don’t make mistakes, we have
happy accidents
- Bob Ross

3. @paulcarletonjr@paulcarletonjr
Hello!
I’m Paul Carleton
@ Stripe
(we’re hiring!)
3

4. @paulcarletonjr
Topics / Spoilers
1. Old Instances are bad
2. Enter Lifespan Management
3. Some stories of how things broke
4. What we learned

5. @paulcarletonjr
1.
Instance Age
What is it and why do I care?

6. @paulcarletonjr
Terminology
▷ Instance: Cloud Hosted VM (EC2)
▷ Age: Time since launch

7. @paulcarletonjr
OldYoung
# of
hosts
Just launched
Instance Age

8. @paulcarletonjr
OldYoung
# of
hosts
Weeks later
Instance Age

9. @paulcarletonjr
OldYoung
# of
hosts
Months later
Instance Age

10. @paulcarletonjr
OldYoung
# of
hosts
Terminate & Replace
Instance Age

11. @paulcarletonjr
OldYoung
# of
hosts
Instance Age

12. @paulcarletonjr
OldYoung
# of
hosts
Instance Age
New hadoop
cluster

13. @paulcarletonjr
OldYoung
# of
hosts
Instance Age
New hadoop
cluster
Big migration

14. @paulcarletonjr
OldYoung
# of
hosts
Instance Age
New hadoop
cluster
Big migration

15. @paulcarletonjr
What’s wrong with old instances?

16. @paulcarletonjr
What’s wrong with old instances?
Replacement is
like a fire
extinguisher...

17. @paulcarletonjr
OldYoung
# of
hosts
Instance Age
Last replacement
Breaking changes

18. @paulcarletonjr
What’s wrong with old instances?
Replacement is
like a fire
extinguisher...
… that might catch
on fire

19. @paulcarletonjr
What’s wrong with old instances?
Replacement is
like a fire
extinguisher...
… that might catch
on fire

20. @paulcarletonjr
What’s wrong with old instances?

21. @paulcarletonjr
What’s wrong with old instances?
Will replacing a
with a
work?

22. @paulcarletonjr
What’s wrong with old instances?

23. @paulcarletonjr
OldYoung
# of
hosts
CVE Patch Released

24. @paulcarletonjr
OldYoung
# of
hosts
CVE Patch Released

25. @paulcarletonjr
OldYoung
# of
hosts
CVE Patch Released

26. @paulcarletonjr
OldYoung
# of
hosts
CVE Patch Released

27. @paulcarletonjr
OldYoung
# of
hosts
CVE Patch Released

28. @paulcarletonjr
OldYoung
# of
hosts
CVE Patch Released

29. @paulcarletonjr
Old Instances are bad

30. @paulcarletonjr
2.
Lifespan Management

31. @paulcarletonjr
Components
▷ ASG
▷ Terminator
▷ Lifespan Manager:

32. @paulcarletonjr
What is an auto-scaling group ?
✨ ASG ✨

33. @paulcarletonjr
Terminator
Terminate
Wait
AWS
Shave yaks
Shutdown
2
What is a terminator?
3
4
1

34. @paulcarletonjr
Lifespan Manager
(waiting)
Lifespan
Manager
ASG

35. @paulcarletonjr
Terminate First vs. Launch First
Steady
State
ASG
Size
Time

36. @paulcarletonjr
Terminate First vs. Launch First
Steady
State
ASG
Size
Time

37. @paulcarletonjr
“Now what?

38. Rollout Plan
Breaking the problem up with labels
38

39. Breaking it up with labels
Stateless
Safe to replace
Stateful Automated
Replaceable with some graceful
state hand-off.
Requires Operator
Not safe to replace automatically.
Want someone watching
39

40. @paulcarletonjr@paulcarletonjr
Automated termination
What could go wrong?

41. @paulcarletonjr
A Year Long Journey

42. @paulcarletonjr
5 Chaotic Discoveries

43. @paulcarletonjr
3.1
How NOT to health check

44. @paulcarletonjr
“The lifespan manager terminated
all the LDAP servers.
We’re locked out of QA.

45. @paulcarletonjr
Don’t we check for this?

46. @paulcarletonjr
How did this happen?
What’s your health?

47. @paulcarletonjr
How did this happen?
What’s your health?
LDAP
Maintenance

48. @paulcarletonjr
How did this happen?
What’s your health?
LDAP
Maintenance
Maintenance

49. @paulcarletonjr
How did this happen?
What’s your health?
LDAP
Maintenance
Maintenance
Everything’s green!
Let’s terminate!

50. 0 unhealthy != healthy
50

51. @paulcarletonjr
How NOT to health check
▷ Pick good defaults
▷ Use pre-shared knowledge to
verify health

52. @paulcarletonjr
Explicit Expectations
What’s your LDAP
health?
LDAP
Maintenance
Maintenance

53. @paulcarletonjr
Explicit Expectations
What’s your LDAP
health?
LDAP
Maintenance
Maintenance
No LDAP?
I better wait.

54. @paulcarletonjr
3.2
RIP Kubernetes Workers

55. @paulcarletonjr
“The Kubernetes workers are going
down… HARD!

56. @paulcarletonjr
Terminator
Terminate
Wait
AWS
Shave yaks
Shutdown
2
Terminator Recap
3
4
1

57. @paulcarletonjr
Terminator
Terminate
Wait
AWS
Shave yaks
Shutdown
2
Terminator Recap
3
4
1

58. @paulcarletonjr
Terminate
Terminate

59. @paulcarletonjr
Terminate
Terminate

60. @paulcarletonjr
Terminate
Terminate

61. @paulcarletonjr
Terminate
Terminate

62. @paulcarletonjr
RIP Kubernetes Workers
● Track feature usage
● Make the chaos easy to turn off

63. 3.3
Blackhole Scenario
63

64. @paulcarletonjr
TerminatorTerminate
Wait
AWS
Shave yaks
Shutdown
1
2
34
Terminator Recap

65. Heartbeat Options
Delay termination Proceed with
termination
… but no Cancel
65

66. @paulcarletonjr
TerminatorTerminate
Wait
AWS
Shave yaks
Shutdown
1
2
3
Blackhole Scenario

67. @paulcarletonjr
Blackhole Scenario
● Non-zero exit
● Timeouts
● Rate limits

68. @paulcarletonjr
“The terminations will continue until
morale improves!

69. 69

70. @paulcarletonjr
Terminator
Terminate
AWS
Already done!
Shutdown
1
2
3
4
Two Touch Termination
Terminator
Shave yaks
1
2
3

71. @paulcarletonjr
The Blackhole Scenario
● Align incentives
● Systems vary, so adapt to match!

72. @paulcarletonjr
3.4
Self-Service Meltdown

73. @paulcarletonjr
A Year Long Journey

74. @paulcarletonjr
A Year Long Journey

75. @paulcarletonjr
Enhance!

76. @paulcarletonjr
Enhance!

77. @paulcarletonjr
“If you would like to never think
about a kernel upgrade again,
consider Lifespan Management!

78. @paulcarletonjr
Part 1:
Turning it On

79. @paulcarletonjr
I want to enable lifespan
management!

80. @paulcarletonjr
Great! Here are some docs!

81. @paulcarletonjr

82. @paulcarletonjr
Part 2:
Who does what?

83. @paulcarletonjr
What happens during
termination?

84. @paulcarletonjr
Let me tell you!

85. @paulcarletonjr
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda

86. @paulcarletonjr
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda
yadda yadda

87. @paulcarletonjr
What part of that is relevant to
me?

88. @paulcarletonjr
… Great question!

89. @paulcarletonjr
Part 3:
False Alarms

90. @paulcarletonjr
Did lifespan management just
break my thing?

91. @paulcarletonjr
Let me check!

92. @paulcarletonjr
aws

93. @paulcarletonjr
aws
5
minutes
later...

94. @paulcarletonjr
Nope!

95. @paulcarletonjr
Okay… what did break my thing?

96. @paulcarletonjr
Okay… what did break my thing?

97. @paulcarletonjr
How we changed

98. @paulcarletonjr
Part 1: Turning it On

99. @paulcarletonjr
Part 2: Who does what?

100. @paulcarletonjr
Part 3: False Alarm
aws
go/whydead/$instance_id

101. @paulcarletonjr
Self-service Meltdown
▷ Make it easy to adopt safely
▷ Explicitly state the contract
▷ Make it easy to rule chaos out

102. @paulcarletonjr
3.5
Death by a thousand
JIRA tickets

103. @paulcarletonjr
Something’s wrong,
I can’t terminate anything
These warnings should be
tickets!

104. @paulcarletonjr
Great!

105. @paulcarletonjr

106. @paulcarletonjr

107. @paulcarletonjr
Death by a thousand JIRA tickets
● File against ourselves first, then
automate
● 1% case matters more with 10x
terminations
● Measure Quantity and Reliability
of tickets

108. @paulcarletonjr
4.
Calling it Done
The End… for now

109. @paulcarletonjr
5.
Summary and Closing

110. @paulcarletonjr
Takeaway
● What automation problems can
you solve with a little chaos?

111. @paulcarletonjr
Takeaway
Do you know how old your instances are?

112. @paulcarletonjr@paulcarletonjr
Thank you!

113. @paulcarletonjr
Credits
● Photo by rawpixel on Unsplash
● Photo by Jens Lelie on Unsplash
● Photo by JohnsonMartin https://pixabay.com/en/wormhole-space-time-light-tunnel-739872/