서비스 가용성을 높이기 위해 Amazon EKS를 멀티 AZ로 사용할 경우, 노드의 위치에 따라 데이터 전송 비용이 추가로 과금됩니다. 본 세션에서는 쿠버네티스 내에서 같은 서비스를 하는 포드의 경우 동일 AZ에서 통신을 하도록 로컬리티 설정을 통해 비용 절감한 사례를 소개합니다. Amazon EKS 환경에서 오토스케일링을 위해 Karpenter를 쓰는 경우, 노드 그룹이 죽거나 DB 연결에 문제가 생기는 등 장애 상황을 미리 검증하기 어렵습니다. 본 세션에서는 카오스 엔지니어링에 사용되는 AWS Fault Injection을 활용하여 EKS 장애 검증 사례를 소개합니다.

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S E O U L | M A Y 4 , 2 0 2 3

2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ EKS
Locality LB
(Koriel)
Site Reliability Engineer

3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio Locality LB
Istio OutlierDetection
Topology Aware Hints
Agenda

4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-zone traffic
Latency
VPC Flog Logs
K8s Pod Metadata
AWS Athena
Visibility

6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
확보

7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
•Istio Ingress Gateway
•Istio Mesh
•Istiod
•Monitoring

9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio Locality LB

10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio Locality LB
ISTIO LOCALITY LB ?
•EKS Worker Node Topology

11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio Locality LB
ISTIO LOCALITY LB ?
•Istio Proxy Config

12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio Locality LB
ISTIO LOCALITY LB ?

13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio Locality LB
ISTIO LOCALITY LB ?

14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio Locality LB
ISTIO LOCALITY LB ?
•
• Istio Locality LB Traffic
• Istio OutlierDetection

15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio OutlierDetection

16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio OutlierDetection
ISTIO LOCALITY LB ISTIO OUTLIERDETECTION
•Virtual Service Destination Rule outlierDetection
•outlierDetection
zone zone pod
failover zone

17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio OutlierDetection
ISTIO LOCALITY LB ISTIO OUTLIERDETECTION

18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Istio OutlierDetection
ISTIO LOCALITY LB ISTIO OUTLIERDETECTION

19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Topology Aware Hints
ISTIO POD ?
•K8s Service Topology Aware Hints Locality LB
•Endpoint Slice Hints

20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Topology Aware Hints
ISTIO POD ?

21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
?

23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
86% 98%
Istio mesh cross-zone traffic 86%
K8s service cross-zone traffic 98%
Istio Locality LB Topolgy Aware Hints

24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AZ ?
• AZ failover
•
AZ a 2 , AZ c 4 AZ a AZ c 2
TopologySpreadContraints maxSkew 1
EKS default scheduler custom scheduler
• Pod
• EKS Active-Active

25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Fault Injection Simulator
EKS
DevOps Engineer
LG

26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LG U+ DevOps Engineer
Backend Developer(APM)
→ DevOps Engineer
AWS
K8s
SRE
linkedin.com/in/youngjin-jung
github.com/YoungJinJung

27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Fault Injection Simulator Chaos Mesh
EKS
Agenda

28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Good intentions never work, you
need good mechanisms to make
anything happen
Jeff Bezos
CEO of Amazon

29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
INFRASTRUCTURE
APPLICATION OPERTAION
…
Multi A-Z
Auto
Scaling
Replica and
Sharding
Dynamic
Routing
Event
Driven
Automation
IaC
Content
Caching
Timeout Monitoring Incidents
Retries with
Backoff
Monitor
Post
Mortem
Exception
Handling
Circuit
Breaker

32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
WHAT WHY
,
,
HOW
(OS, ,
)
,
SPOF
/

34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaos
Engineering
Improvement
Define
Steady State
Hypothesis
Run
Experiment
Verify

35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability
Visibility Alerting Less time in Incident Acceleration
Observability

36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Fault Injection Simulator
Chaos Mesh

37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Fault Injection Simulator
03.
Integrated
Security Model
01.
Simple Setup
02.
Run Real World
Scenarios
04.
Visibility throughout
an Experiment

38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS FIS

39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Flexible experiment orchestration
Cloud Native and easy-to-use system
Kubernetes
High security and fully authenticated
,
Chaos Mesh ?

40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaos Mesh
Input Fault From User
• kubectl
• API
• Dashboard

41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaos Mesh
Monitor Resource and Schedule
• Kubernetes API

42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaos Mesh
Injection of a specific fault
• Chaos Controller Manager
•

43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EKS

44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EKS
EKS
Scaling Monitoring
Deploy
Network
Mesh

45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network Chaos
Network
Delay Injection

46. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network Chaos
Hypothesis
Network Delay
502
02 03 04 05
Run Exp.
Pod Network
Delay 300ms
Verify
Network Delay
1000ms , 502
Improve
502 , Retry
Steady State
100ms
01

47. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
?

48. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
References
• Principles of Chaos Engineering
https://principlesofchaos.org/
• Fault Injection Simulator
https://docs.aws.amazon.com/fis/latest/userguide/what-is.html
• Chaos Mesh
https://chaos-mesh.org/docs/
• AWS Well-Architected Framework - Reliability
https://wa.aws.amazon.com/wat.pillar.reliability.ko.html

49. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
감사합니다
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.