深探如何使用-Amazon-EKS-與-Prometheus-進行雲端監控

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Deep Dive in Cloud Monitoring
with Amazon EKS and Prometheus
Pahud Hsieh
Specialist SA, Serverless
Amazon Web Services
Kakashi Liu
Infra Lead
UmboCV

Amazon EKS in the Past Year
● Started in us-east-1 and us-west-2
● Released VPC CNI 1.0
● HIPPA Support
● Released AMI build scripts on Github
● Enabled GPU Support
● Support API Aggregation
● Support HPA
● Support eu-west-1
● CLI support for writing the kubeconfig
● Support for Admission Controllers

Amazon EKS in the Past Year
● Allow for additional VPC CIDR ranges
● Support for us-east-2
● Official support for ALB Ingress
● Container Marketplace
● CloudMap Integration
● Support for AWS App Mesh
● Support for eu-central1, ap-southeast-1, ap-southeast-2, ap-
northeast-1
● Support for ap-northeast-2
● Added the SLA

Immediately after that
● Achieved ISO and PCI compliance
● Support for ap-south-1, eu-west-2, eu-west-3
● Added a new qiuckstart
● Allowed private API Endpoints
● Launched an App Mesh controller at GA
● Public Preview for Windows nodes
● Deep Learning container launch
● Added 1.2 with a new cluster update API
● Released CSI Drivers for FSx and EFS
● Control plane logs
● Public Preview of A1 instances
● Released a Machine Learning Benchmark tool

CloudWatch Container Insights(preview)

Dimensions for Kubernetes
• Clusters
• Nodes
• Services
• Namespaces
• Pods

Pod Metrics
• pod_cpu_reserved_capacity
• pod_cpu_utilization
• pod_cpu_utilization_over_pod_li
mit
• pod_memory_reserved_capacity
• pod_memory_utilization
• pod_memory_utilization_over_p
od_limit
• pod_network_rx_bytes
• pod_network_tx_bytes

Other Metrics
• cluster_failed_node_count
• cluster_node_count
• namespace_number_of_runni
ng_pods
• node_cpu_limit
• node_cpu_reserved_capacity
• node_cpu_usage_total
• node_cpu_utilization
• node_filesystem_utilization
• node_memory_limit
• node_memory_reserved_capa
city
• node_memory_utilization
• node_memory_working_set
• node_network_total_bytes
• node_number_of_running_containers
• node_number_of_running_pods
• service_number_of_running_pods
Reference - https://amzn.to/2HFtHDt

Amazon EKS and Prometheus
Prometheus
Why Prometheus?
Community
Number of integrations
Ease of use
Why not Prometheus?
Manage it yourself
Complexity in large setups
Possibility: Hybrid Approach
Use Prometheus to collect metrics that
are exposed on /metrics endpoints
Send a subset of critical metrics to
Amazon CloudWatch or a third party
solution.

Hello!
I am kakashi
- Infra Lead @Umbo CV
- Co-organizer @Golang Taipei Gathering

Traditional
Solutions
Umbo
Light

Agenda
Why monitoring
Umbo CV Monitoring pipeline
Prometheus: Why and What
Prometheus with EKS
Use cases

Why monitoring

Why monitoring
Alerting Long-term trends

Umbo CV Monitoring pipeline

Monitoring types
Infrastructure
Application

Application monitoring
EC2
Metrics
Store
container
container
exporter
exporter
exporter
/metrics
EC2 /metrics
Collect
Alert
Expose
Metrics

Prometheus: Why and What
● Graduates Within CNCF.
● Can handle multi-dimensional metrics.
● Performance: can ingest millions of samples per second.
● Powerful query language: PromQL.
● Built-in alerting tool and service discovery mechanism.

Prometheus metrics
EC2 /metrics
EC2 /metrics
User request
http_requests_total{code=200, path="/api/user"} 10
metric_name labels value

PromQL example
Total requests / second
sum(rate(http_requests_total[5m]))
Total 5xx requests / second
sum(rate(http_requests_total{code=~"5.*"}[5
m]))
Current percentage of errors across all instances
sum(rate(http_requests_total{code=~"5.*"}[5m])) /
sum(rate(http_requests_total[5m]))

Alerting rule
alert: Percentage_Of_Errors_Is_High
expr: sum(rate(http_requests_total{code=~"5.*"}[5m]))
/
sum(rate(http_requests_total[5m])) > 5
for: 5m
labels:
severity: critical

Prometheus with EKS

Prometheus ❤ EKS
● Monitoring system is critical.
● Running Prometheus on Kubernetes can
easily achieve HA.
● Prometheus operator makes it ever easier
○ Automated management and upgrades of
Prometheus.
○ Native k8s configuration.

Install Prometheus on EKS by helm
1. Install Promethues Operator chart
2. Verify
$ helm install --name prom --namespace monitoring stable/prometheus-operator
$ kubectl --namespace monitoring get pods
NAME READY STATUS RESTARTS AGE
alertmanager-prom-op-alertmanager-0 2/2 Running 0 1m
prometheus-prom-op-prometheus-0 3/3 Running 1 1m
prom-op-grafana-5c59ddfb9d-zqfqt 2/2 Running 0 2m
prom-op-kube-state-metrics-76786cc9b4-8q4bj 1/1 Running 0 2m
prom-op-prometheus-node-exporter-6jclc 1/1 Running 0 2m
prom-op-prometheus-node-exporter-bxr49 1/1 Running 0 2m
prom-op-prometheus-operato-operator-6cbf5d5cfd-z6fz4 1/1 Running 0 2m

Prometheus Operator CRD
● Prometheus & AlertManager
○ Define Prometheus and AlertManager deployment.
● ServiceMonitor
○ Used to specify how metric of k8s services can be
scraped.
● PrometheusRule
○ Can be loaded by a Prometheus instance containing
Prometheus alerting and recording rules.

EKS cluster monitoring

EKS application monitoring through ServiceMonitor
apiVersion:
monitoring.coreos.com/v1
kind: Servicemonitor
metadata:
name: api-servicemonitor
spec:
selector:
matchLabels:
app: api-server
Labels:
app: api-server
Labels:
app: api-server2

Alerting by PrometheusRule
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
spec:
groups:
- name: api.rules
rules:
- alert: Percentage_Of_Errors_Is_High
expr:
sum(rate(http_requests_total{code=~"5.*"}[5m])) /
sum(rate(http_requests_total[5m])) > 5
for: 5m
labels:
severity: critical

Dashboard for EKS cluster

Monitoring camera detection pipeline
Media
Serve
r
CV
Detectio
n
API
Serve
r

Monitoring camera detection pipeline
Media
Serve
r
CV
Detectio
n
API
Serve
r
# of
frames # cv
requests
# of events

Service discovery
Media
Serve
r
CV
Detectio
n
API
Serve
r
Scraping through EC2 service
discovery

Service discovery
Media
Server
CV
Detection
API
Server
Scraping
global:
scrape_interval: 1s
evaluation_interval: 1s
scrape_configs:
- job_name: 'node'
ec2_sd_configs:
- region: eu-east-1
access_key:
<ACCESS_KEY_HERE>
secret_key:
<SECRET_KEY_HERE>
port: 9273

Application metrics
Media
Serve
r
CV
Detectio
n
API
Serve
r
ms_frames_total{env="production", service="ms", cameraId="ID-123456"}
1000
# of frames
# of cv requests cvreqest_total{env="production", service="cv", cameraId="ID-123456"} 300
# of events event_total{env="production", service="cv", cameraId="ID-123456"} 5
# of frames # of cv request # of events

Dashboard

Alerting
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
spec:
groups:
- name: camera.rules
rules:
- alert: FpsLow
annotations:
message: "{{ $labels.cameraid }} fps is lower than 2fps"
expr: sum(rate(ms_frames_total{env="production", cameraId=".+"}[10m])) < 2
for: 30mins
labels:
severity: critical

深探如何使用-Amazon-EKS-與-Prometheus-進行雲端監控

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 深探如何使用-Amazon-EKS-與-Prometheus-進行雲端監控

Similar to 深探如何使用-Amazon-EKS-與-Prometheus-進行雲端監控 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

深探如何使用-Amazon-EKS-與-Prometheus-進行雲端監控