1. / . 10 / 21
/7 2 118 2

2. -
https://www.youtube.com/watch?v=OczG5FQIcXwhttps://www.youtube.com/watch?v=-mL3zT1iIKw

3. (c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016
Netflix DVD Data Center - 2000
Linux Host
Apache
Tomcat
Javaweb
STORE
LoadBalancer
BILLING
HTTP
JDBC
DB Link
HTTP/S
Monolithic code base
Monolithic database
Tightly coupled architecture

4. 100s of microservices
1000s of daily production changes
10,000s of instances
100,000s of customer interactions per minute
1,000,000s of customers
1,000,000,000s of metrics
10,000,000,000 hours of streamed
(c) Dave Hahn, A Day in the Life of a Netflix Engineer Using 37% of the Internet, re:Invent 2015
10s of operations engineers
No Datacenter
• 2008 – Datacenter failure
(must be a better way)
• 2009 – Cloud effort starts
• 2010 – The first devices are
talking to Netflix in the cloud
(US-EAST-1)
• 2011 – EU-WEST-1
• 2013 – US-WEST-2
(active/active)
• 2015 – Migration Complete

5. Edge
ELB
Zuul
NCCP
API
Middle Tier & Platform
Product
• Bucket testing
• Subscriber
• Recommendations
Platform
• Routing
• Configuration
• Crypto
Persistence
• Cache
• Database
(c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016

6. E
• (
• ( / F o Sv c dB
xe n B l F +/
+/ k F D y u
c d
• +/ / ( v n L Cp r
: v b Cp e t s
R ic E
• (
• ( , / / / c i c
c as
• )
• ) / ) c c d d
B e / u y m

7. (c) Ruslan Meshenberg, From Asgard to Zuul, re:Invent 2014

8. ( )
Chaos Monkey
https://github.com/ne
tflix/chaosmonkey
Instance Fail?
Chaos Gorilla
Zone Fail?
Chaos Kong
Region Fail?
.

9. - , , -, ,
, A
,
,
, A ,
, ,
,
, ,
, A
, ,
&
,
(c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016

10. http://channy.creation.net/blog/netflix-principles-of-chaos-engineering
s ). 13 ,,2 h ea
o g
n
E ‘ 4
n re i
e o ’ C n
( n

11. •
•
•
•
•
http://channy.creation.net/blog/netflix-principles-of-chaos-engineering

12. Presented by Adrian Cockcroft,
VP of Cloud Strategy, AWS

13. ( ) ( ()

14. No single
point of failure
ReplicatedDistributed
Automated Cloud

17. 1
34
. 2

20. App
!

21. B
C
-

23. !
!
?
?

24. .

26. Chaos
Engineering
Team

27. Chaos
Engineering
Team
Game days
Simian Army
Failure Injection
ChAP
Gremlin

28. Chaos
Engineering
Team
Security
Red
Team
Safestack
AVA
Metasploit
Nmap
AttackIQ
SafeBreach

29. 4B
Chaos
Engineering
Team
Security
Red
Team
2
B

30. :) : ( . ( :
R S l i S l h ?
gE C ? Oa
e a nD b ?
Q :)
-

31. https://www.facebook.com/groups/chaosengkorea/
https://www.meetup.com/Korea-Chaos-Engineering-Community/