Learn How to Use a Time Series Platform to Monitor All Aspects of Your Kubernetes Deployment

Gianluca Arbezzano / Site Reliability Engineer
Kubernetes Monitoring
with InfluxDB

© 2018 InfluxData. All rights reserved.2
Who I am
Gianluca Arbezzano
Site Reliability Engineer @InfluxData
• http://gianarb.it
• @gianarb
What I like:
• I make dirty hacks that look awesome
• I grow my vegetables 🍅🌻🍆
• Travel for fun and work

How distributed systems
monitoring is different
● Partial failure
● Fault tolerance and resiliency
○ Space dimension = replications
○ Time dimension = retries
● “normal state” is hard to define

DB 1
DB 1
Client A Client B
Load Balancer
Load Balancer
Cache A Cache B

Kubernetes architecture diagram

Telegraf as daemonset to get nodes stats
[[inputs.internal]]
[[inputs.cpu]]
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs"]
[[inputs.diskio]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.docker]]
endpoint = "unix:///var/run/docker.sock"
[[inputs.kubernetes]]
url = "http://127.0.0.1:10255"

Telegraf as daemonset to get nodes stats
volumeMounts:
- name: sys
mountPath: /rootfs/sys
readOnly: true
- name: docker
mountPath: /var/run/docker.sock
readOnly: true
- name: proc
mountPath: /rootfs/proc
readOnly: true
- name: utmp
mountPath: /var/run/utmp
readOnly: true
● hostNetwork: true
● dnsPolicy: clusterFist

Telegraf as daemonset reachable from a container
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: MONITOR_HOST
value: "http://$(HOST_IP):8086"
[[inputs.http_listener]]
## Address and port to host HTTP listener on
service_address = ":8086"

Telegraf as a Sidecar

Telegraf as a Sidecar
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: "etcd"
labels:
spec:
serviceName: "etcd"
replicas: 3
template:
metadata:
name: "etcd"
labels:
component: "etcd"
spec:
containers:
- name: "telegraf"
image: "docker.io/library/telegraf:1.4"
- name: "etcd"
image: "quay.io/coreos/etcd:v3.2.9"

https://www.influxdata.com/blog/monitoring-kubernetes-architecture/

Feedback from “real life”
● High number of Telegraf running inside the
cluster
● For Prometheus metrics there is a better way
(I will tell you how later)
● Pull vs Push

Pull and Push

/metrics
# HELP storage_cache_age_seconds Age in seconds of the current cache (time since last snapshot or initialisation).
# TYPE storage_cache_age_seconds gauge
storage_cache_age_seconds{engine_id="0",node_id="0"} 112.999976922

Kubernetes discovery with the Prometheus Plugin
[[inputs.prometheus]]
monitor_kubernetes_pods = true
Enabling this option will allow the plugin to scrape for prometheus annotation on Kubernetes pods.
• prometheus.io/scrape Enable scraping for this pod.
• prometheus.io/scheme If the metrics endpoint is secured then you will need to set this to https &
most likely set the tls config. (default 'http')
• prometheus.io/path Override the path for the metrics endpoint on the service. (default '/metrics')
• prometheus.io/port Used to override the port. (default 9102)

Monitor your ingestion pipeline
• internal_memstats
• internal_agent
– metrics_dropped
– metrics_gathered
• internal_gather
• internal_write

start = 6h
interval = 3m
from(bucket: "kube-infra/monthly")
|> range(start: start)
|> filter(fn: (r) =>
r._measurement == "internal_agent"
and r.env == "acc"
and r.host =~ /^telegraf-prom-discovery/)
r._field == "metrics_dropped"
or r._field == "metrics_gathered"
or r._field == "metrics_written")
|> window(every: interval)
|> mean() // defaults to "_value"
|> group(columns: ["_field"])
|> derivative(nonNegative: true, timeColumn: "_stop")

Monitor your ingestion pipeline
• You can use
inputs.http_response to
check if telegraf is healthy.
• You can configure k8s
Liveness and Readiness
Probe to manage Telegraf
availability

ReadinessProbe and LivenessProbe
LivenessProbe: applications eventually transition to broken states,
and cannot recover except by being restarted. Kubernetes provides
liveness probes to detect and remedy such situations.
ReadinessProbe: applications eventually get busy or temporary
unavailable. A pod with a containers reporting that they are not
ready does not receive traffic through Kubernetes Services.

Telegraf as Sidecar gives you control
[[inputs.internal]]
urls = ["http://127.0.0.1:9999/metrics"]
[[processors.converter]]
[processors.converter.tags]
string = ["user_agent"]
[[outputs.influxdb]]
urls = ["$MONITOR_HOST"]
database = "$MONITOR_DATABASE"
timeout = "5s"
[[outputs.influxdb_v2]]
urls=["http://us-west-2-1.aws.cloud2.influxdata.com"]
token = "$TOKEN"
organization = "$ORG"
bucket = "$BUCKET"
timeout = "5s"
namepass = ["internal"]

Telegraf Guard Rails
[[inputs.internal]]
urls = ["http://127.0.0.1:9999/metrics"]
[[processors.tag_limit]]
limit = 3
## List of tags to preferentially preserve
keep = ["handler", "method", "status"]
[[outputs.influxdb]]
urls = ["$MONITOR_HOST"]
database = "$MONITOR_DATABASE"
timeout = "5s"
[[outputs.influxdb_v2]]
urls=["http://us-west-2-1.aws.cloud2.influxdata.com"]
token = "$TOKEN"
organization = "$ORG"
bucket = "$BUCKET"
timeout = "5s"
namepass = ["internal"]

Lessons
Scaling is NOT More Manual Processes
Scaling is NOT saying “You’re Doing it Wrong”
Scaling IS Empowering Developers
Scaling IS Predictability of Failure Modes

Lesson
Architecture is a never ending story…
Telegraf as sidecar for your developers writes to the daemonset ->
daemonset for your ops with safeguard writes to influxdb.
Maybe complex but possible!

Monitor is up
when you are down
InfluxDB makes everything simpler but your monitor
notifies you when your infrastructure is down. It is not
simple.
● Different infrastructure
● Reliability team
● Redundancy
● Or you can use a SaaS (InfluxCloud is 100%
compatible with OSS for write/read)

Number of pod restart
from(bucket:"kube-infra/monthly")
|> range(start: dashboardTime, stop: upperDashboardTime)
r._measurement == "kube_pod_container_status_restarts_total"
and r._field == "counter"
and r.container == "xxxx"
and r.namespace == "xxxx")
|> difference(nonNegative: true)
|> group()
|> aggregateWindow(every: autoInterval, fn: sum, createEmpty: false)

Persistent Volume % usage
start = -20m
from(bucket: "kube-infra/monthly")
|> range(start: start)
r._measurement == "kubernetes_pod_volume"
and (r._field == "used_bytes" or r._field == "capacity_bytes"))
|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
|> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
|> map(fn: (r) => ({_time: r._time, _value: 100.0 * r.used_bytes / r.capacity_bytes})

Learn How to Use a Time Series Platform to Monitor All Aspects of Your Kubernetes Deployment

Recommended

Recommended

More Related Content

Similar to Learn How to Use a Time Series Platform to Monitor All Aspects of Your Kubernetes Deployment

Similar to Learn How to Use a Time Series Platform to Monitor All Aspects of Your Kubernetes Deployment (20)

More from DevOps.com

More from DevOps.com (20)

Recently uploaded

Recently uploaded (20)

Learn How to Use a Time Series Platform to Monitor All Aspects of Your Kubernetes Deployment