SlideShare a Scribd company logo
1 of 28
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
JARVIS, 장애 대응 해줘!
Team 5. 장애 시나리오 및 대응 솔루션
발표자 : 이상원(그라운드엑스)
1
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Contents
- Introduction
- Architecture
- Flow
- Scenario
- Used Services
2
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introduction
3
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Background
장애상황 발생 시, 관리자가 시스템 접근이 어려운 경우 신속한 대응이 어려움.
4
상황에 제약받지 않고 Slack으로 장애에 대응할 수 있게 하자.
Idea description
신속한 대응을 통한 장애복구를 통한 서비스 안정성.
Expected outcome
Introduction
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
5
Application
6
Terraform으로 EKS 셋업
Helm chart로 MSA 환경 구성
>> Terraform code
Architecture Diagram
7
Architecture Diagram
8
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Flow
9
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10
Step1. 장애 상황 알림
CloudWatch alert
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11
Step1. 장애 상황 확인
>> Grafana dashboard with CloudWatch
CloudWatch + Pixie 로 구성한
Grafana Dashboard
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12
Step1. 장애 상황 확인
>> Grafana dashboard with Pixie
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13
Step2. 장애 상황 대처
AWS Chatbot
[Run] command
>> Workflows
>> Lambda code
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14
Step3. 대응 결과 확인
슬랙봇 Result alert
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15
Step3. 대응 결과 확인
BotKube로 인프라 확인
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario
16
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17
Scale hpa maxreplicas
FIS로 front-end 부하
CloudWatch alert
Dashboard & Botkube로 확인
AWS Chatbot으로 scale-out
Success alert
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18
Rollout Restart
비정상 동작발생, Botkube 확인
Secret 값 미적용으로
pod restart 필요
AWS Chatbot command
Success alert
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19
Helm Rollback
릴리즈 문제 발생하여 롤백필요
AWS Chatbot command
Success alert
Botkube로 결과 확인
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20
Change WAF ID
Application 접근 차단 필요
AWS Chatbot command
Success alert
Botkube changed WAF Id,
차단된 ErrorPage 화면 확인
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Used Services
21
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22
AWS Services
EKS
Cloudwatch
FIS
Managed
Grafana
Chatbot
Lambda S3 ACM WAF
SNS
Route53
IAM
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23
Technology Stacks
Botkube Github Action
Chaos Mesh
형상관리
24
>> github link
Expected Cost
25
Expected Cost
26
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you :)
27
Q&A
28

More Related Content

Similar to 2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.

re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdf
re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdfre:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdf
re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdfHeitor Lessa
 
AWS Lambda Powertools walkthrough.pdf
AWS Lambda Powertools walkthrough.pdfAWS Lambda Powertools walkthrough.pdf
AWS Lambda Powertools walkthrough.pdfHeitor Lessa
 
Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...
Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...
Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...Chargebee
 
Continuous Delivery on AWS with Zero Downtime
Continuous Delivery on AWS with Zero DowntimeContinuous Delivery on AWS with Zero Downtime
Continuous Delivery on AWS with Zero DowntimeCasey Lee
 
Innovate - Breaking Down The Monolith
Innovate - Breaking Down The MonolithInnovate - Breaking Down The Monolith
Innovate - Breaking Down The MonolithShouvikKnightmare
 
Desafios da transição de estado em um mundo serverless
Desafios da transição de estado em um mundo serverlessDesafios da transição de estado em um mundo serverless
Desafios da transição de estado em um mundo serverlessAlex Barbosa Coqueiro
 
AWS DevDay Vienna - Automating building blocks choices you will face with con...
AWS DevDay Vienna - Automating building blocks choices you will face with con...AWS DevDay Vienna - Automating building blocks choices you will face with con...
AWS DevDay Vienna - Automating building blocks choices you will face with con...Cobus Bernard
 
Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019
Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019 Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019
Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019 Amazon Web Services
 
DevConfZA 2020 : Automating your cloud: What are the building blocks
DevConfZA 2020 : Automating your cloud: What are the building blocksDevConfZA 2020 : Automating your cloud: What are the building blocks
DevConfZA 2020 : Automating your cloud: What are the building blocksCobus Bernard
 
kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...
kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...
kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...kreuzwerker GmbH
 
AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON) December 7, 2022 - COM311 Mi...
AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON)  December 7, 2022 - COM311 Mi...AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON)  December 7, 2022 - COM311 Mi...
AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON) December 7, 2022 - COM311 Mi...Matt Houghton
 
COM311 Migrating 600 Databases To AWS
COM311 Migrating 600 Databases To AWS COM311 Migrating 600 Databases To AWS
COM311 Migrating 600 Databases To AWS Matt Houghton
 
현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...
현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...
현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...Amazon Web Services Korea
 
AWS SSA Webinar 12 - Getting started on AWS with Containers
AWS SSA Webinar 12 - Getting started on AWS with ContainersAWS SSA Webinar 12 - Getting started on AWS with Containers
AWS SSA Webinar 12 - Getting started on AWS with ContainersCobus Bernard
 
AWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless Backends
AWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless BackendsAWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless Backends
AWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless BackendsPatrick Sard
 
Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...
Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...
Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...Amazon Web Services
 
AWS Community Day, Chennai - 2019
AWS Community Day, Chennai - 2019AWS Community Day, Chennai - 2019
AWS Community Day, Chennai - 2019Pranesh Vittal
 
Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...
Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...
Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...Amazon Web Services
 
Breaking the Monolith Using AWS Container Services
Breaking the Monolith Using AWS Container ServicesBreaking the Monolith Using AWS Container Services
Breaking the Monolith Using AWS Container ServicesAmazon Web Services
 

Similar to 2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis. (20)

re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdf
re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdfre:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdf
re:Invent OPN306 AWS Lambda Powertools Lessons 10M downloads.pdf
 
AWS Lambda Powertools walkthrough.pdf
AWS Lambda Powertools walkthrough.pdfAWS Lambda Powertools walkthrough.pdf
AWS Lambda Powertools walkthrough.pdf
 
Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...
Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...
Powering Test Environments with Amazon EKS using Serverless Tool | AWS Commun...
 
Continuous Delivery on AWS with Zero Downtime
Continuous Delivery on AWS with Zero DowntimeContinuous Delivery on AWS with Zero Downtime
Continuous Delivery on AWS with Zero Downtime
 
Innovate - Breaking Down The Monolith
Innovate - Breaking Down The MonolithInnovate - Breaking Down The Monolith
Innovate - Breaking Down The Monolith
 
Vue presentation
Vue presentationVue presentation
Vue presentation
 
Desafios da transição de estado em um mundo serverless
Desafios da transição de estado em um mundo serverlessDesafios da transição de estado em um mundo serverless
Desafios da transição de estado em um mundo serverless
 
AWS DevDay Vienna - Automating building blocks choices you will face with con...
AWS DevDay Vienna - Automating building blocks choices you will face with con...AWS DevDay Vienna - Automating building blocks choices you will face with con...
AWS DevDay Vienna - Automating building blocks choices you will face with con...
 
Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019
Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019 Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019
Innovating FIPS crypto validation in the Cloud - SEP321 - AWS re:Inforce 2019
 
DevConfZA 2020 : Automating your cloud: What are the building blocks
DevConfZA 2020 : Automating your cloud: What are the building blocksDevConfZA 2020 : Automating your cloud: What are the building blocks
DevConfZA 2020 : Automating your cloud: What are the building blocks
 
kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...
kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...
kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...
 
AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON) December 7, 2022 - COM311 Mi...
AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON)  December 7, 2022 - COM311 Mi...AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON)  December 7, 2022 - COM311 Mi...
AWS DATABASE USER GROUP - LAUNCH EVENT (LONDON) December 7, 2022 - COM311 Mi...
 
COM311 Migrating 600 Databases To AWS
COM311 Migrating 600 Databases To AWS COM311 Migrating 600 Databases To AWS
COM311 Migrating 600 Databases To AWS
 
현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...
현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...
현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...
 
AWS SSA Webinar 12 - Getting started on AWS with Containers
AWS SSA Webinar 12 - Getting started on AWS with ContainersAWS SSA Webinar 12 - Getting started on AWS with Containers
AWS SSA Webinar 12 - Getting started on AWS with Containers
 
AWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless Backends
AWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless BackendsAWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless Backends
AWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless Backends
 
Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...
Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...
Build a PCI SAQ A-EP-compliant serverless service to manage credit card payme...
 
AWS Community Day, Chennai - 2019
AWS Community Day, Chennai - 2019AWS Community Day, Chennai - 2019
AWS Community Day, Chennai - 2019
 
Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...
Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...
Introduction to Amazon Route 53 Resolver for Hybrid Cloud (NET215) - AWS re:I...
 
Breaking the Monolith Using AWS Container Services
Breaking the Monolith Using AWS Container ServicesBreaking the Monolith Using AWS Container Services
Breaking the Monolith Using AWS Container Services
 

Recently uploaded

Artificial Intelligence in due diligence
Artificial Intelligence in due diligenceArtificial Intelligence in due diligence
Artificial Intelligence in due diligencemahaffeycheryld
 
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas SachpazisSeismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas SachpazisDr.Costas Sachpazis
 
Operating System chapter 9 (Virtual Memory)
Operating System chapter 9 (Virtual Memory)Operating System chapter 9 (Virtual Memory)
Operating System chapter 9 (Virtual Memory)NareenAsad
 
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptxSLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptxCHAIRMAN M
 
5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...archanaece3
 
The Entity-Relationship Model(ER Diagram).pptx
The Entity-Relationship Model(ER Diagram).pptxThe Entity-Relationship Model(ER Diagram).pptx
The Entity-Relationship Model(ER Diagram).pptxMANASINANDKISHORDEOR
 
Raashid final report on Embedded Systems
Raashid final report on Embedded SystemsRaashid final report on Embedded Systems
Raashid final report on Embedded SystemsRaashidFaiyazSheikh
 
AI in Healthcare Innovative use cases and applications.pdf
AI in Healthcare Innovative use cases and applications.pdfAI in Healthcare Innovative use cases and applications.pdf
AI in Healthcare Innovative use cases and applications.pdfmahaffeycheryld
 
analog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptxanalog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptxKarpagam Institute of Teechnology
 
Introduction to Artificial Intelligence and History of AI
Introduction to Artificial Intelligence and History of AIIntroduction to Artificial Intelligence and History of AI
Introduction to Artificial Intelligence and History of AISheetal Jain
 
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdflitvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdfAlexander Litvinenko
 
Geometric constructions Engineering Drawing.pdf
Geometric constructions Engineering Drawing.pdfGeometric constructions Engineering Drawing.pdf
Geometric constructions Engineering Drawing.pdfJNTUA
 
What is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, FunctionsWhat is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, FunctionsVIEW
 
Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2T.D. Shashikala
 
8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...josephjonse
 
Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1T.D. Shashikala
 
Passive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.pptPassive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.pptamrabdallah9
 
UNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptxUNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptxkalpana413121
 
21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological university21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological universityMohd Saifudeen
 
CLOUD COMPUTING SERVICES - Cloud Reference Modal
CLOUD COMPUTING SERVICES - Cloud Reference ModalCLOUD COMPUTING SERVICES - Cloud Reference Modal
CLOUD COMPUTING SERVICES - Cloud Reference ModalSwarnaSLcse
 

Recently uploaded (20)

Artificial Intelligence in due diligence
Artificial Intelligence in due diligenceArtificial Intelligence in due diligence
Artificial Intelligence in due diligence
 
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas SachpazisSeismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
Seismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis
 
Operating System chapter 9 (Virtual Memory)
Operating System chapter 9 (Virtual Memory)Operating System chapter 9 (Virtual Memory)
Operating System chapter 9 (Virtual Memory)
 
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptxSLIDESHARE PPT-DECISION MAKING METHODS.pptx
SLIDESHARE PPT-DECISION MAKING METHODS.pptx
 
5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...5G and 6G refer to generations of mobile network technology, each representin...
5G and 6G refer to generations of mobile network technology, each representin...
 
The Entity-Relationship Model(ER Diagram).pptx
The Entity-Relationship Model(ER Diagram).pptxThe Entity-Relationship Model(ER Diagram).pptx
The Entity-Relationship Model(ER Diagram).pptx
 
Raashid final report on Embedded Systems
Raashid final report on Embedded SystemsRaashid final report on Embedded Systems
Raashid final report on Embedded Systems
 
AI in Healthcare Innovative use cases and applications.pdf
AI in Healthcare Innovative use cases and applications.pdfAI in Healthcare Innovative use cases and applications.pdf
AI in Healthcare Innovative use cases and applications.pdf
 
analog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptxanalog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptx
 
Introduction to Artificial Intelligence and History of AI
Introduction to Artificial Intelligence and History of AIIntroduction to Artificial Intelligence and History of AI
Introduction to Artificial Intelligence and History of AI
 
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdflitvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
litvinenko_Henry_Intrusion_Hong-Kong_2024.pdf
 
Geometric constructions Engineering Drawing.pdf
Geometric constructions Engineering Drawing.pdfGeometric constructions Engineering Drawing.pdf
Geometric constructions Engineering Drawing.pdf
 
What is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, FunctionsWhat is Coordinate Measuring Machine? CMM Types, Features, Functions
What is Coordinate Measuring Machine? CMM Types, Features, Functions
 
Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2
 
8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...8th International Conference on Soft Computing, Mathematics and Control (SMC ...
8th International Conference on Soft Computing, Mathematics and Control (SMC ...
 
Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1
 
Passive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.pptPassive Air Cooling System and Solar Water Heater.ppt
Passive Air Cooling System and Solar Water Heater.ppt
 
UNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptxUNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptx
 
21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological university21scheme vtu syllabus of visveraya technological university
21scheme vtu syllabus of visveraya technological university
 
CLOUD COMPUTING SERVICES - Cloud Reference Modal
CLOUD COMPUTING SERVICES - Cloud Reference ModalCLOUD COMPUTING SERVICES - Cloud Reference Modal
CLOUD COMPUTING SERVICES - Cloud Reference Modal
 

2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.

  • 1. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. JARVIS, 장애 대응 해줘! Team 5. 장애 시나리오 및 대응 솔루션 발표자 : 이상원(그라운드엑스) 1
  • 2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Contents - Introduction - Architecture - Flow - Scenario - Used Services 2
  • 3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Introduction 3
  • 4. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Background 장애상황 발생 시, 관리자가 시스템 접근이 어려운 경우 신속한 대응이 어려움. 4 상황에 제약받지 않고 Slack으로 장애에 대응할 수 있게 하자. Idea description 신속한 대응을 통한 장애복구를 통한 서비스 안정성. Expected outcome Introduction
  • 5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Architecture 5
  • 6. Application 6 Terraform으로 EKS 셋업 Helm chart로 MSA 환경 구성 >> Terraform code
  • 9. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Flow 9
  • 10. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10 Step1. 장애 상황 알림 CloudWatch alert
  • 11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11 Step1. 장애 상황 확인 >> Grafana dashboard with CloudWatch CloudWatch + Pixie 로 구성한 Grafana Dashboard
  • 12. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12 Step1. 장애 상황 확인 >> Grafana dashboard with Pixie
  • 13. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13 Step2. 장애 상황 대처 AWS Chatbot [Run] command >> Workflows >> Lambda code
  • 14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14 Step3. 대응 결과 확인 슬랙봇 Result alert
  • 15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15 Step3. 대응 결과 확인 BotKube로 인프라 확인
  • 16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scenario 16
  • 17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17 Scale hpa maxreplicas FIS로 front-end 부하 CloudWatch alert Dashboard & Botkube로 확인 AWS Chatbot으로 scale-out Success alert
  • 18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18 Rollout Restart 비정상 동작발생, Botkube 확인 Secret 값 미적용으로 pod restart 필요 AWS Chatbot command Success alert
  • 19. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19 Helm Rollback 릴리즈 문제 발생하여 롤백필요 AWS Chatbot command Success alert Botkube로 결과 확인
  • 20. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20 Change WAF ID Application 접근 차단 필요 AWS Chatbot command Success alert Botkube changed WAF Id, 차단된 ErrorPage 화면 확인
  • 21. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Used Services 21
  • 22. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22 AWS Services EKS Cloudwatch FIS Managed Grafana Chatbot Lambda S3 ACM WAF SNS Route53 IAM
  • 23. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23 Technology Stacks Botkube Github Action Chaos Mesh
  • 27. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you :) 27