Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernates를 위한 Chaos Engineering in Action :: 윤석찬 (AWS 테크에반젤리스트)

547 views

Published on

Kubernates를 위한 Chaos Engineering in Action

Published in: Technology
  • Be the first to comment

Kubernates를 위한 Chaos Engineering in Action :: 윤석찬 (AWS 테크에반젤리스트)

  1. 1. 3/ . 103 / 21 3/ 2 118 2
  2. 2. 1 . 2
  3. 3. ) ( # 2 . 1
  4. 4. /- . / -/ -
  5. 5. Failures are a given and everything will eventually fail over time. https://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html
  6. 6. https://www.youtube.com/watch?v=zoz0ZjfrQ9s Amazon 2006 GameDay: Creating Resiliency Through Destruction Jesse Robbins
  7. 7. Netflix 2011 Chaos Monkeys: Test the resilience of its Infrastructure Simian Army – Open Source https://github.com/Netflix/SimianArmy
  8. 8. T T r T a o W D e W A W E n (c) Dave Hahn, A Day in the Life of a Netflix Engineer Using 37% of the Internet, re:Invent 2015 W 0 5 5 9 • ( E n Un NS • ( , c • ( - 1 T T c 31 /-12 • ( /3 /12 • ( ) 31 /12 ( - 8 - 8 • ( i
  9. 9. ELB Zuul API (c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016
  10. 10. E • ( • ( / F o Sv c dB xe n B l F +/ +/ k F D y u c d • +/ / ( v n L Cp r : v b Cp e t s R ic E • ( • ( , / / / c i c c as • ) • ) / ) c c d d B e / u y m
  11. 11. (c) Ruslan Meshenberg, From Asgard to Zuul, re:Invent 2014
  12. 12. F A I R ( c R a Z d )/ , ( c R e o nP dS e F (c) Josh Evans, Mastering Chaos A Netflix Guide to Microservices, QCon SF 2016
  13. 13. Chaos Monkey https://github.com/ne tflix/chaosmonkey Instance Fail? Chaos Gorilla Zone Fail? Chaos Kong Region Fail? .
  14. 14. , ) )( . j a l o i c e n tF r I n
  15. 15. • • U • P • • ), ( ) • P CP !
  16. 16. Chaos doesn’t cause problems. It reveals them.
  17. 17. C E C C ! https://www.oreilly.com/webops- perf/free/chaos-engineering.csp
  18. 18. http://principlesofchaos.org/ http://channy.creation.net/blog/netflix- principles-of-chaos-engineering • • • • •
  19. 19. . ( / ) - / / /( : . (
  20. 20. Y S • 4 A A9 4 A4 4 4 A 4 . : % • .AA: A9 4 A4 4 ( 9 4 04 4 04 % • 4 AA! ) A9 4 A4 4 4 L 4 5 A9 A A M54 N 5 9A 4: G A4 1 A 2 G4 %
  21. 21. • C aP • 0 C • • ? 0 V • 5 3 • • 4
  22. 22. • • • •
  23. 23. • • ! ! • •
  24. 24. Users ) ( ) 99% users 1% users Start with...
  25. 25. • ? • ? • • • • •
  26. 26. ( 5 5 5 ) 5
  27. 27. Microservices (applications) DevOps(Culture) C haos Engineering Cloud (Scale)
  28. 28. Chaos Toolkit PowerfulSeal Gremlin
  29. 29. mycluster.eks.amazonaws.com Availability Zone 1 Availability Zone 2 Availability Zone 3 Kubectl
  30. 30. mycluster.eks.amazonaws.com Availability Zone 1 Availability Zone 2 Availability Zone 3 Kubectl x x Health check? Dead node?
  31. 31. https://github.com/asobti/kube-monkey C
  32. 32. apiVersion: apps/v1 kind: Deployment . . . template: metadata: labels: app: greeting kube-monkey/enabled: enabled kube-monkey/identifier: redis-master kube-monkey/mtbf: 2 kube-monkey/kill-mode: random-max-percent kube-monkey/kill-value: 40 spec: containers: - name: greeting apiVersion: v1 kind: Confirm metadata: name: kube-monkey-config-map namespace: kube-system data: config.toml: | [kubemonkey] run_hour = 8 start_hour = 10 end_hour = 16 blacklisted_namespaces = ["kube-system"] whitelisted_namespaces = [”default”] time_zone = “UTC”
  33. 33. . Amazon EKS
  34. 34. Chaos Toolkit PowerfulSeal Gremlin
  35. 35. Service Mesh Plane Svc A Svc B Side Car Side Car
  36. 36. M S I • • • • , //. , 2 C 1
  37. 37. apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: ratings-destination-rule spec: host: ratings subsets: - name: ratings labels: ratings: hello trafficPolicy: connectionPool: tcp: maxConnections: 100 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: reviews spec: hosts: - reviews http: - route: - destination: host: reviews subset: v1 weight: 75 - destination: host: reviews subset: v2 weight: 25
  38. 38. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: ratings spec: hosts: - ratings http: - fault: delay: fixedDelay: 10s percent: 100 route: - destination: host: ratings subset: v1 apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: ratings spec: hosts: - ratings http: - fault: abort: httpStatus: 500 percent: 100 route: - destination: host: ratings subset: v1
  39. 39. https://github.com/chaostoolkit/chaostoolkit-kubernetes
  40. 40. & & & https://help.gremlin.com/installation/#ho w-to-install-gremlin-with-kubernetes https://github.com/bloomberg/powerfulseal https://www.youtube.com/watch?v=00BMn0UjsG4
  41. 41. https://bit.ly/2uKOJMQ
  42. 42. https://github.com/chaoseng/wg-chaoseng
  43. 43. • @LLH N AJ BJ@ L / 5 H=J 2 = 183A H • @LLH ?J= A • @LLH = = J? =L A -A • @LLH L J== ?A ==JA ? A • @LLH ?AL@ : L=J? = = J= • @LLH = AP J? L= A = =J= = A A H H=J H • @LLH = A .4=L AP7= @0 ? • @LLH H= =J = L : L @ = ?A ==JA ? : L H • @LLH ?AL@ : @ J = = @ = ?A ==JA ? • @LLH A HJ= = L LA =L AP @ A J =JNA = • @LLH J HA ? H L= L H HA ? 8 HLA =8 @= L8 @==L H • @LLH A ? =? =JA J L = A References
  44. 44. https://www.facebook.com/groups/chaosengkorea/ https://www.meetup.com/Korea-Chaos-Engineering-Community/

×