1. 0 / . 21 0 26
08 229 C A

2. F ) ( , ( ) ,
I

3. ) # 2 (2
1 . 2 !

4. /- . / -/ -

5. . .
.
.
Werner Vogels
CTO – Amazon.com
“ “

6. Netflix 2013
Active-Active for Multi-
Regional Resiliency
https://medium.com/netflix-techblog/active-active-
for-multi-regional-resiliency-c47719f6685b
Amazon 2011
GameDay: Creating
Resiliency Through
Destruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

7. Chaos Monkey
https://github.com/ne
tflix/chaosmonkey
Instance Fail?
Chaos Gorilla
Zone Fail?
Chaos Kong
Region Fail?
.

8. 100s of microservices
1000s of daily production changes
10,000s of instances
100,000s of customer interactions per minute
1,000,000s of customers
1,000,000,000s of metrics
10,000,000,000 hours of streamed
(c) Dave Hahn, A Day in the Life of a Netflix Engineer Using 37% of the Internet, re:Invent 2015
10s of operations engineers
No Datacenter
• 2008 – Datacenter failure
(must be a better way)
• 2009 – Cloud effort starts
• 2010 – The first devices are
talking to Netflix in the cloud
(US-EAST-1)
• 2011 – EU-WEST-1
• 2013 – US-WEST-2
(active/active)
• 2015 – Migration Complete

9. Edge
ELB
Zuul
NCCP
API
Middle Tier & Platform
Product
• Bucket testing
• Subscriber
• Recommendations
Platform
• Routing
• Configuration
• Crypto
Persistence
• Cache
• Database
(c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016

10. Microservices (applications)
DevOps(Culture)
C
haos
Engineering
Cloud (Scale)

11. http://principlesofchaos.org/
http://channy.creation.net/blog/netflix-
principles-of-chaos-engineering
•
•
•
•
•

12. C
E C C
!
https://www.oreilly.com/webops-
perf/free/chaos-engineering.csp

13. !
,
https://github.com/dastergon/awesome-
chaos-engineering

14. https://bit.ly/2uKOJMQ

15. 4B
Chaos
Engineering
Team
Security
Red
Team
2
B

16. Chaos
Engineering
Team
Simian Army
Failure Injection
ChAP
Gremlin

17. ( ,) : , . : ) .
S R C E
O ?
ER Q Q D
()

18. - - -

19. Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software
changes with high velocity and confidence. http://www.spinnaker.io/

20. Chaos Testing in Kubernetes
https://github.com/arun-gupta/kubernetes-chaos
Chaos Toolkit, Istio, Gremlin
KubeMonkey Chaos
https://github.com/asobti/kube-monkey
kube-monkey is an implementation of Netflix's
Chaos Monkey for Kubernetes clusters

22. - - -

23. ) ( )
App App App
X

25. • @LLH N AJ BJ@ L / 5 H=J 2 = 183A H
• @LLH ?J= A
• @LLH = = J? =L A -A
• @LLH L J== ?A ==JA ? A
• @LLH ?AL@ : L=J? = = J=
• @LLH = AP J? L= A = =J= = A A H H=J H
• @LLH = A .4=L AP7= @0 ?
• @LLH H= =J = L : L @ = ?A ==JA ? : L H
• @LLH ?AL@ : @ J = = @ = ?A ==JA ?
• @LLH A HJ= = L LA =L AP @ A J =JNA =
• @LLH J HA ? H L= L H HA ? 8 HLA =8 @= L8 @==L H
• @LLH A ? =? =JA J L = A
References

26. https://www.facebook.com/groups/chaosengkorea/
https://www.meetup.com/Korea-Chaos-Engineering-Community/