여러 어카운트와 리전에 구축된 워크로드의 운영 가시성을 확보하는 것은 안정적인 서비스 운영에 있어 중요한 요소 중 하나입니다. Amazon Managed Prometheus를 이용해 Cross region 및 Cross account 환경에서 Metric data를 중앙 집중식으로 구현하는 효과적인 설계를 삼성전자와 SendBird의 사례를 통해 알아봅니다.

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S E O U L | M A Y 3 , 2 0 2 3

2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Global scale service
Observability
정영민
프로
삼성전자
정찬훈
Software Engineer
Sendbird
김아린
Solutions Architect,
Monitoring & Observability AoD
AWS

3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Full Stack observability” 를
Agenda
Intro: Observability란 ?
“Multi Accounts” Account
- Sendbird
-

5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
글로벌
사용자 수백만 명 이상
데이터 처리량 TB, PB, EB
서비스 지역 전 세계
요구 성능 Micro-second latency
초 당 요청 수 수백만 건 이상
사용 기기 Mobile, IoT 기기
> 106
> 4K video x 50
= 24 x 7
Micro-second
Mobile, IoT
수백만 건

6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability = Monitoring ?
CPU 35%,
Memory 72%,
Disk IO, 210 IOPS

7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability, Infra Monitoring !
?
현재 ?
95% ?
Traffic volume ?

8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability란?
‘Observability’는,
‘Monitoring’은 , ‘Observability’
,

9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability의
Logs Metrics Traces
> Log : , Issue Context
> Metric : , Graph Trend
> Trace : ,

10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability

11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS ,
CloudWatch ServiceLens
Container
Insights
Lambda
Insights
Synthetics
Contributor
Insights
CloudWatch
Logs
CloudWatch
Metrics
X-ray
CloudWatch Agent X-ray Agent
Instrumentation
AWS Distro for OpenTelemery (ADOT)
AWS-native monitoring options
Amazon Managed
Open source options
Amazon Managed Service for Grafana
Do it yourself (DIY)
AMP
Amazon
OpenSearch
Observability
Accelerator

12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Full-Stack Observability”
삼성전자

13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
If you can not measure,
Peter Ferdinand Drucker
Management Consultant (1909.11 ~ 2005.11)
you can not manage.

14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability Data
RCS – +
31mil 2500mps 50%
MAU(Monthly Active User) A2P
10 ~11
P2P
50%

15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prometheus
PROMETHEUS (2019)
P
G
Prometheus
Server
Prometheus
Alertmanager
Grafana
Target Services
3rd-party exporter
node exporter
kube-state-metrics
Prometheus
Operator

16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prometheus
PROMETHEUS (2019)
2019-03 2019-06 2019-09 2019-12 2020-03 2020-06 2020-09 2020-12
MAU A2P P2P
700%
3

17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prometheus
LocalStorage
Metrics
Prometheus read
CPU
Clustering
HA
PROMETHEUS (2019)
P
G
Prometheus
Server
Prometheus
Alertmanager
Grafana
Target Services
3rd-party exporter
node exporter
kube-state-metrics
Prometheus
Operator

18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thanos
PROMETHEUS & THANOS (2020~2022)
Scale-Out
Downsampling
P G
Prometheus
Server
Prometheus
Alertmanager
Grafana
Target Services
3rd-party exporter
node exporter
kube-state-metrics
Prometheus
Operator
A
Alerting
System
Sidecar
Store
Compact
S3
Query

19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PROMETHEUS & AMP (2023)
Amazon Managed Service for Prometheus
Thanos (2020~2022) AMP (2023)
VPC
P G
Prometheus
Server
Prometheus
Alertmanager
Grafana
Target Services
3rd-party exporter
node exporter
kube-state-
metrics Prometheus
Operator
A
Alerting System
Sidecar
Store
Compact
S3
Query
P G
Prometheus
Server
Grafana
Target Services
3rd-party exporter
node exporter
kube-state-
metrics Prometheus
Operator
A
Alerting System
VPC
AWS Lambda Amazon SNS

20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• AWS
• Alerting System
/
• Scale-Out/In
, ,
•

21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS
AWS
All Managed Observability
ADOT
MAAP Account RCS Account1 RCS Account2
AMP
Insight Account
ADOT ADOT
ADOT
AWS X-Ray

22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability
RCS OBSERVABILITY
2019 2020~2022 2023
Prometheus Prometheus
&
Thanos
Prometheus
&
AMP
ADOT
&
AMP

23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi Accounts
Account
Sendbird사례

24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
300M+
Monthly Active Users
7B+
Messages Per Month
9 Regions
Global Infrastructure
+ 1,200 others

26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current
Metrics
Logs
Profiles
Traces
Types Number of resources
AWS accounts Tens
AWS regions Tens
AWS VPC Hundreds
EKS Hundreds
EC2 Thousands
Workload
Types
VM, Container, Serverless, …

27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Issues
Heavy Cost
Vendor
Less Control
Non-standard
Lack of Support
Less Metric
Less Logs
More Context
Engineering
Fragmentation
Less
Centralization

28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Considers
Less Cost
Less work
Enough
Capacity
Better
Latency
Granular
Control
Easy to learn
Industrial
Commonality
Simple
Architecture

29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Decides
Decides Why?
Grafana / Prometheus
Many references
Skilled people in team
Collect metric per workload VPC Simple metric collection
Single aggregation point Less operating efforts
Amazon Managed Grafana
Amazon Managed Prometheus
Enough Capacity & Latency
Easy integration
AWS supports

30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
System Architecture
Amazon Managed
Grafana
Amazon Managed
Prometheus
Prometheus
Users
Cloud Watch
Region Region
Scrape
Remote
write
Query
Query

31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network Architecture
Region
Virtual private cloud (VPC)
Region
Virtual private
cloud (VPC)
Private subnet
Public subnet
NAT
gateway
Internet gateway
Peering
connection
Endpoints Amazon Managed Service
for Prometheus
Prometheus
Prometheus
Option 1
Option 2
Option 3

32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results
Less cost
Better latency
Fully
Operational
Granular Control
More metrics
Common
Observability
View

33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Next Plan
Logs Profiles
Traces
Metrics
Observability Migration
Public
VPC
Reshape Network Architecture
Private
Link
Alerts Events

34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key Takeaways & Next Steps
• One Observability Workshop
• AWS Distro for OpenTelemery
Next Steps
Key Takeaways
• 복잡한
AWS Managed Observability
• Multi region / Multi Account를
• 비즈니스
Observability

35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
감사합니다
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.