카오스 엔지니어링(Chaos Engineering)이란 프로덕션 서비스의 각종 장애 조건을 견딜 수 있는 시스템의 신뢰성을 확보하기 위해 분산 시스템을 실험 하고 배우는 분야입니다. 즉, 개발자들이 현실 세계에서 발견되는 시스템 장애를 미리 탐지하여 복원성 높은 아키텍처를 구성하는 방법을 공유합니다.클라우드 컴퓨팅의 발전과 데브옵스 방법론을 기반으로 자동화를 통해 좀 더 쉽게 개발자들이 직접 분산 시스템을 통제된 환경에서 실험을 하는 동안 나오는 결과를 관찰함으로써 현실에서 실제 행동 방법을 배울 수 있습니다. 본 세션에서는 카오스 엔지니어링의 기본 개념과 함께 Kubernetes용 Chaos Tool인 KubeMonkey를 통해 무작위로 클러스터의 포드를 삭제하여 장애 복구 서비스 아키텍처를 검증하는 방법을 설명합니다.

1. 0 / . 21 0 26
08 229 C A

2. F ) ( , ( ) ,
I

3. ) # 2 (2
1 . 2 !

4. /- . / -/ -

5. . .
.
.
Werner Vogels
CTO – Amazon.com
“ “

6. Netflix 2013
Active-Active for Multi-
Regional Resiliency
https://medium.com/netflix-techblog/active-active-
for-multi-regional-resiliency-c47719f6685b
Amazon 2011
GameDay: Creating
Resiliency Through
Destruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

7. Chaos Monkey
https://github.com/ne
tflix/chaosmonkey
Instance Fail?
Chaos Gorilla
Zone Fail?
Chaos Kong
Region Fail?
.

8. 100s of microservices
1000s of daily production changes
10,000s of instances
100,000s of customer interactions per minute
1,000,000s of customers
1,000,000,000s of metrics
10,000,000,000 hours of streamed
(c) Dave Hahn, A Day in the Life of a Netflix Engineer Using 37% of the Internet, re:Invent 2015
10s of operations engineers
No Datacenter
• 2008 – Datacenter failure
(must be a better way)
• 2009 – Cloud effort starts
• 2010 – The first devices are
talking to Netflix in the cloud
(US-EAST-1)
• 2011 – EU-WEST-1
• 2013 – US-WEST-2
(active/active)
• 2015 – Migration Complete

9. Edge
ELB
Zuul
NCCP
API
Middle Tier & Platform
Product
• Bucket testing
• Subscriber
• Recommendations
Platform
• Routing
• Configuration
• Crypto
Persistence
• Cache
• Database
(c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016

10. Microservices (applications)
DevOps(Culture)
C
haos
Engineering
Cloud (Scale)

11. http://principlesofchaos.org/
http://channy.creation.net/blog/netflix-
principles-of-chaos-engineering
•
•
•
•
•

12. C
E C C
!
https://www.oreilly.com/webops-
perf/free/chaos-engineering.csp

13. !
,
https://github.com/dastergon/awesome-
chaos-engineering

14. https://bit.ly/2uKOJMQ

15. 4B
Chaos
Engineering
Team
Security
Red
Team
2
B

16. Chaos
Engineering
Team
Simian Army
Failure Injection
ChAP
Gremlin

17. ( ,) : , . : ) .
S R C E
O ?
ER Q Q D
()

18. - - -

19. Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software
changes with high velocity and confidence. http://www.spinnaker.io/

20. Chaos Testing in Kubernetes
https://github.com/arun-gupta/kubernetes-chaos
Chaos Toolkit, Istio, Gremlin
KubeMonkey Chaos
https://github.com/asobti/kube-monkey
kube-monkey is an implementation of Netflix's
Chaos Monkey for Kubernetes clusters

22. - - -

23. ) ( )
App App App
X

25. • @LLH N AJ BJ@ L / 5 H=J 2 = 183A H
• @LLH ?J= A
• @LLH = = J? =L A -A
• @LLH L J== ?A ==JA ? A
• @LLH ?AL@ : L=J? = = J=
• @LLH = AP J? L= A = =J= = A A H H=J H
• @LLH = A .4=L AP7= @0 ?
• @LLH H= =J = L : L @ = ?A ==JA ? : L H
• @LLH ?AL@ : @ J = = @ = ?A ==JA ?
• @LLH A HJ= = L LA =L AP @ A J =JNA =
• @LLH J HA ? H L= L H HA ? 8 HLA =8 @= L8 @==L H
• @LLH A ? =? =JA J L = A
References

26. https://www.facebook.com/groups/chaosengkorea/
https://www.meetup.com/Korea-Chaos-Engineering-Community/