This document discusses concepts related to chaos engineering including: 1. Many large tech companies like Amazon, Netflix, and Google practice chaos engineering to test system resiliency through failure injection and destruction testing. 2. Netflix developed open source tools like Chaos Monkey and Simian Army to randomly terminate instances and components to ensure applications can withstand infrastructure failures. 3. Chaos engineering involves injecting failures in a controlled way to test recovery capabilities and uncover weaknesses before they impact real users during production outages.

1. /
// /

2. E D
4 35 22 5 3 IK
4 145 21 2 A
4 CA 4 . 2 2. 2
4 CA 2 G M

3. ) (
) # 2 (2
1 . 2 !

4. /- . / -/ -

5. Failures are a given and
everything will eventually
fail over time.
https://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html

6. https://www.youtube.com/watch?v=zoz0ZjfrQ9s
Amazon 2006
GameDay: Creating
Resiliency Through
Destruction
Jesse Robbins

7. Netflix 2011
Chaos Monkeys:
Test the resilience
of its Infrastructure
Simian Army – Open Source
https://github.com/Netflix/SimianArmy

8. T
T r
T a o W
D e W
A W
E n
(c) Dave Hahn, A Day in the Life of a Netflix Engineer Using 37% of the Internet, re:Invent 2015
W
0 5 5 9
• ( E n Un NS
• ( , c
• ( - 1 T
T c 31 /-12
• ( /3 /12
• ( ) 31 /12 (
- 8 - 8
• ( i

9. -
https://www.youtube.com/watch?v=OczG5FQIcXwhttps://www.youtube.com/watch?v=-mL3zT1iIKw

10. Edge
ELB
Zuul
NCCP
API
Middle Tier & Platform
Product
• Bucket testing
• Subscriber
• Recommendations
Platform
• Routing
• Configuration
• Crypto
Persistence
• Cache
• Database
(c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016

11. E
• (
• ( / F o Sv c dB
xe n B l F +/
+/ k F D y u
c d
• +/ / ( v n L Cp r
: v b Cp e t s
R ic E
• (
• ( , / / / c i c
c as
• )
• ) / ) c c d d
B e / u y m

12. (c) Ruslan Meshenberg, From Asgard to Zuul, re:Invent 2014

13. F A I
R ( c
R a
Z d )/ ,
( c
R e
o nP dS e F
(c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016

14. Chaos Monkey
https://github.com/ne
tflix/chaosmonkey
Instance Fail?
Chaos Gorilla
Zone Fail?
Chaos Kong
Region Fail?
.

16. , ) )( .
j a l o i
c
e
n tF r I
n

17. •
• U
• P
•
• ), ( )
• P CP !

18. 우리 사장님이 허락해 줄까요?

19. Chaos doesn’t cause problems.
It reveals them.

20. C
E C C
!
https://www.oreilly.com/webops-
perf/free/chaos-engineering.csp

21. http://principlesofchaos.org/
http://channy.creation.net/blog/netflix-
principles-of-chaos-engineering
•
•
•
•
•

24. . ( / ) - / / /( : .
(

25. Y S
• 4 A A9 4 A4 4 4 A 4 . : %
• .AA: A9 4 A4 4 ( 9 4 04 4 04 %
• 4 AA! ) A9 4 A4 4 4 L 4 5 A9
A A M54 N 5 9A 4: G A4 1 A 2 G4 %

27. • C aP
• 0 C
•
• ? 0 V
• 5 3
•
• 4

29. •
•
•
•

30. •
• ! !
•
•

31. Users
) ( )
99%
users
1%
users
Start with...

33. • ?
• ?
•
•
•
•
•

35. ( 5 5 5
) 5

36. Failure free operations require
experience with failure.
http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf

39. ( ) ( ()

40. No single
point of failure
ReplicatedDistributed
Automated Cloud

43. 1
34
. 2

45. App
!

46. / / I
/ / C B S T
/ F AOC S D
/ / /
/ / /

47. !
!
?
?

48. !

49. .

51. Chaos
Engineering
Team

52. Chaos
Engineering
Team
ChaosMoneky
GameDay
Failure Injection
ChAP
Gremlin

54. Chaos
Engineering
Team
GameDay
Failure Injection
ChAP
Gremlin

55. - - -

56. • A
S
• ,
N ,,
S
https://github.com/netflix/chaosmonkey

57. Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software
changes with high velocity and confidence. http://www.spinnaker.io/

58. - - -

59. AWS managed
Customer account
Auto-scaling
Auto-scaling Group
HA/Backup

60. mycluster.eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl

61. Istio
Chaos Toolkit
Kube Monkey
PowerfulSeal
Gremlin
Simian Army

62. https://github.com/asobti/kube-monkey
• ) C M ) ) K
( M

63. mycluster.eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl
x
x
Health check?
Dead node?
x

65. .
Amazon EKS

69. Chaos
Engineering
Team
ChaosMonkey
Failure Injection
ChAP
Gremlin

70. ,
A
- + +
- +

71. . , ()
Bug
Integration
Distributed
Engineering
Team
Operation/Security
Team
Customer Center
Team
PR
Team

72. • G m a
• P D e
• A y
• , - P G
• A P Aa G
• I
• P )(

73. ? ?
• P a r
• e i e
• m t ?
• u e ? ?
-
• ( ( d n ) ,
• yd c Da eG e P G o

74. G ?D
G
•
•
• :
G :

75. -
• D
• ( .
• Z B a ) i a
• e R Zc a
• S ( c Ah

76. • 2
• ) . ),.1,0 C
• a (
• a
• 3C

77. © DIUS https://dius.com.au/resources/game-day/

78. © DIUS https://dius.com.au/resources/game-day/

79. • c -
-
•
,c - - n - c
• la
•
• - n - c
• - O . , . c

80. ) ( (
• 0E B Yo N…# cd e O O
• C N… # Z Z n d N…
• wr d O# n N…
R Z O m n
w al /- 0 0 w /2, w P i
3 B
R 0 h
t
n
L 51
w
% E /B DC BG Y
wr
51 ACB Ss 51 Z p
I

81. • e
: r n
A
:
• A )( , .
: )(
W I
IA
( S

82. - ,
D
• -
• ,
•
•
• )
• : , ( ,
•
•
• (

83. • 2 1:/ 36 -7 -7 63 7:3 36 :7. - 376 7 1 /.
• 2 /-267 71 9 3. -7 7 : 03: /6136//:361 1 / .
• 2 71 6/ :/ 3- -7 1 / . / 361
https://www.infoq.com/presentations/gameday-chaos-engineering

84. https://dius.com.au/resources/game-day !
© DIUS https://dius.com.au/resources/game-day/

85. Microservices (applications)
DevOps(Culture)
C
haos
Engineering
Cloud (Scale)

86. https://bit.ly/2uKOJMQ

87. https://github.com/chaoseng/wg-chaoseng

88. • @LLH N AJ BJ@ L / 5 H=J 2 = 183A H
• @LLH ?J= A
• @LLH = = J? =L A -A
• @LLH L J== ?A ==JA ? A
• @LLH ?AL@ : L=J? = = J=
• @LLH = AP J? L= A = =J= = A A H H=J H
• @LLH = A .4=L AP7= @0 ?
• @LLH H= =J = L : L @ = ?A ==JA ? : L H
• @LLH ?AL@ : @ J = = @ = ?A ==JA ?
• @LLH A HJ= = L LA =L AP @ A J =JNA =
• @LLH J HA ? H L= L H HA ? 8 HLA =8 @= L8 @==L H
• @LLH A ? =? =JA J L = A
References

89. https://www.facebook.com/groups/chaosengkorea/
https://www.meetup.com/Korea-Chaos-Engineering-Community/