Successfully reported this slideshow.
Your SlideShare is downloading. ×

Kubernates를 위한 Chaos Engineering in Action :: 윤석찬 (AWS 테크에반젤리스트)

More Related Content

More from Channy Yun

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Kubernates를 위한 Chaos Engineering in Action :: 윤석찬 (AWS 테크에반젤리스트)

  1. 1. 3/ . 103 / 21 3/ 2 118 2
  2. 2. 1 . 2
  3. 3. ) ( # 2 . 1
  4. 4. /- . / -/ -
  5. 5. Failures are a given and everything will eventually fail over time. https://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html
  6. 6. https://www.youtube.com/watch?v=zoz0ZjfrQ9s Amazon 2006 GameDay: Creating Resiliency Through Destruction Jesse Robbins
  7. 7. Netflix 2011 Chaos Monkeys: Test the resilience of its Infrastructure Simian Army – Open Source https://github.com/Netflix/SimianArmy
  8. 8. T T r T a o W D e W A W E n (c) Dave Hahn, A Day in the Life of a Netflix Engineer Using 37% of the Internet, re:Invent 2015 W 0 5 5 9 • ( E n Un NS • ( , c • ( - 1 T T c 31 /-12 • ( /3 /12 • ( ) 31 /12 ( - 8 - 8 • ( i
  9. 9. ELB Zuul API (c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016
  10. 10. E • ( • ( / F o Sv c dB xe n B l F +/ +/ k F D y u c d • +/ / ( v n L Cp r : v b Cp e t s R ic E • ( • ( , / / / c i c c as • ) • ) / ) c c d d B e / u y m
  11. 11. (c) Ruslan Meshenberg, From Asgard to Zuul, re:Invent 2014
  12. 12. F A I R ( c R a Z d )/ , ( c R e o nP dS e F (c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016
  13. 13. Chaos Monkey https://github.com/ne tflix/chaosmonkey Instance Fail? Chaos Gorilla Zone Fail? Chaos Kong Region Fail? .
  14. 14. , ) )( . j a l o i c e n tF r I n
  15. 15. • • U • P • • ), ( ) • P CP !
  16. 16. Chaos doesn’t cause problems. It reveals them.
  17. 17. C E C C ! https://www.oreilly.com/webops- perf/free/chaos-engineering.csp
  18. 18. http://principlesofchaos.org/ http://channy.creation.net/blog/netflix- principles-of-chaos-engineering • • • • •
  19. 19. . ( / ) - / / /( : . (
  20. 20. Y S • 4 A A9 4 A4 4 4 A 4 . : % • .AA: A9 4 A4 4 ( 9 4 04 4 04 % • 4 AA! ) A9 4 A4 4 4 L 4 5 A9 A A M54 N 5 9A 4: G A4 1 A 2 G4 %
  21. 21. • C aP • 0 C • • ? 0 V • 5 3 • • 4
  22. 22. • • • •
  23. 23. • • ! ! • •
  24. 24. Users ) ( ) 99% users 1% users Start with...
  25. 25. • ? • ? • • • • •
  26. 26. ( 5 5 5 ) 5
  27. 27. Microservices (applications) DevOps(Culture) C haos Engineering Cloud (Scale)
  28. 28. Chaos Toolkit PowerfulSeal Gremlin
  29. 29. mycluster.eks.amazonaws.com Availability Zone 1 Availability Zone 2 Availability Zone 3 Kubectl
  30. 30. mycluster.eks.amazonaws.com Availability Zone 1 Availability Zone 2 Availability Zone 3 Kubectl x x Health check? Dead node?
  31. 31. https://github.com/asobti/kube-monkey C
  32. 32. apiVersion: apps/v1 kind: Deployment . . . template: metadata: labels: app: greeting kube-monkey/enabled: enabled kube-monkey/identifier: redis-master kube-monkey/mtbf: 2 kube-monkey/kill-mode: random-max-percent kube-monkey/kill-value: 40 spec: containers: - name: greeting apiVersion: v1 kind: Confirm metadata: name: kube-monkey-config-map namespace: kube-system data: config.toml: | [kubemonkey] run_hour = 8 start_hour = 10 end_hour = 16 blacklisted_namespaces = ["kube-system"] whitelisted_namespaces = [”default”] time_zone = “UTC”
  33. 33. . Amazon EKS
  34. 34. Chaos Toolkit PowerfulSeal Gremlin
  35. 35. Service Mesh Plane Svc A Svc B Side Car Side Car
  36. 36. M S I • • • • , //. , 2 C 1
  37. 37. apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: ratings-destination-rule spec: host: ratings subsets: - name: ratings labels: ratings: hello trafficPolicy: connectionPool: tcp: maxConnections: 100 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: reviews spec: hosts: - reviews http: - route: - destination: host: reviews subset: v1 weight: 75 - destination: host: reviews subset: v2 weight: 25
  38. 38. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: ratings spec: hosts: - ratings http: - fault: delay: fixedDelay: 10s percent: 100 route: - destination: host: ratings subset: v1 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: ratings spec: hosts: - ratings http: - fault: abort: httpStatus: 500 percent: 100 route: - destination: host: ratings subset: v1
  39. 39. https://github.com/chaostoolkit/chaostoolkit-kubernetes
  40. 40. & & & https://help.gremlin.com/installation/#ho w-to-install-gremlin-with-kubernetes https://github.com/bloomberg/powerfulseal https://www.youtube.com/watch?v=00BMn0UjsG4
  41. 41. https://bit.ly/2uKOJMQ
  42. 42. https://github.com/chaoseng/wg-chaoseng
  43. 43. • @LLH N AJ BJ@ L / 5 H=J 2 = 183A H • @LLH ?J= A • @LLH = = J? =L A -A • @LLH L J== ?A ==JA ? A • @LLH ?AL@ : L=J? = = J= • @LLH = AP J? L= A = =J= = A A H H=J H • @LLH = A .4=L AP7= @0 ? • @LLH H= =J = L : L @ = ?A ==JA ? : L H • @LLH ?AL@ : @ J = = @ = ?A ==JA ? • @LLH A HJ= = L LA =L AP @ A J =JNA = • @LLH J HA ? H L= L H HA ? 8 HLA =8 @= L8 @==L H • @LLH A ? =? =JA J L = A References
  44. 44. https://www.facebook.com/groups/chaosengkorea/ https://www.meetup.com/Korea-Chaos-Engineering-Community/

×