Kubernetes debug like a pro

~ @gianarb - https://gianarb.it ~
Debug like a pro on
Kubernetes
Golab - Florence - 2018

Gianluca Arbezzano
Site Reliability Engineer @InfluxData
● http://gianarb.it
● @gianarb
What I like:
● I make dirty hacks that look awesome
● I grow my vegetables 🍅🌻🍆
● Travel for fun and work

1. Yo n !
Your team knows
and use Docker for
local development
and testing
2. Kub te !
Everyone speaks
about kubernetes.
3. Hir !
You don’t know why
but you hired a
DevOps that kind of
know k8s.
3. Ex i m !
You are moving
everything and
everyone to
kubernetes

You need a good book
1. Short
2. Driven by experiences
3. Practical
4. Easy

We need to make our
hands dirty

Spin up a cluster that you
can break
Bring developers in the loop

Deploy CI on Kubernetes

Run your code in prod

Don’t be scared and write your
own tools!

K8s as code: From YAML to code (golang)
1. You have the ability to use Golang autocomplete as documentation, reference for every
kubernetes resources
2. You feel less a YAML engineer (great feeling btw)
3. Code is better than YAML! You can reuse it, compile it, embed it in other projects.

Tiny cli
to make
the
migration
to golang
Some
manual
refactoring

Tiny cli
to make
the
migration
to golang
Some
manual
refactoring
● Continue to improve our CI to validate that YAML and Go file are the same,
and the resources in Kubernetes are like the Go file.
● Maybe we will be able to remove the YAML at some point.

Examples
● Everything has an API because you should USE it TO make something good!
(cURL is good but you can make something better)
● Some of our tools:
○ Backup and Restore Operator for Persistent Volumes
○ We have a service to create runtime isolated environment to allow devs to test or product
people to have a safe environment to demo, try. We also use it for the integration and smoke
tests.
○ We have a tool to replicate environment locally on minikube to install and configure all the
dependencies

Instrumentation and
Observability

We need to have processes and tools that give us
the ability to take a real time picture of our system

Observability
It is all about how we collect and aggregate the data

You need knowledgeable devs to drive the team

Observability
Events, metrics, logs and traces

Golang and Kubernetes: OpenCensus
● Open Source project sponsored by Google
● It is a SPEC plus a set of libraries in different languages to instrument your
application
● To collect metrics, traces and events.

Golang and Kubernetes: OpenCensus
Common
Interface to
get stats and
traces from
your app
Different
exporters to
persist your
data

gianarb.it ~ @gianarb
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"} 3 1395066363000
# Escaping in label values:
msdos_file_access_time_seconds{path="C:DIRFILE.TXT",error="Cannot find file:n"FILE.TXT""}
1.458255915e9
# Minimalistic line:
metric_without_timestamp_and_labels 12.47
# A weird metric from before the epoch:
something_weird{problem="division by zero"} +Inf -3982045
# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

OpenMetrics
v2 Prometheus exposition format

func FetchMetricFamilies(url string, ch chan<- *dto.MetricFamily, certificate string, key string, skipServerCertCheck bool) error {
defer close(ch)
var transport *http.Transport
if certificate != "" && key != "" {
cert, err := tls.LoadX509KeyPair(certificate, key)
if err != nil {
return err
}
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert},
InsecureSkipVerify: skipServerCertCheck,
}
tlsConfig.BuildNameToCertificate()
transport = &http.Transport{TLSClientConfig: tlsConfig}
} else {
transport = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: skipServerCertCheck},
}
}
client := &http.Client{Transport: transport}
return decodeContent(client, url, ch)
}
https://github.com/prometheus/prom2json/blob/master/prom2json.go#L123

More to read
● OpenMetrics: https://github.com/OpenObservability/OpenMetrics
● OpenMetrics mailing list: https://groups.google.com/forum/#protectkern-.
1667emrelaxforum/openmetrics
● WIP branch for Python library
https://github.com/prometheus/client_python/tree/openmetrics
● Thanks RICHARD for your work! I got some slides from here:
https://promcon.io/2018-munich/slides/openmetrics-transforming-the-prometh
eus-exposition-format-into-a-global-standard.pdf

Tracing
How do you “tell stories” about concurrent
systems?

How do you “tell stories” about
concurrent systems?

OpenTracing

I was just waiting for a new
standard!
cit. Troll

© 2017 InfluxData. All rights reserved.34
Typical problems with logs
¨ Which library do I need to use?
¨ Every library has a different format
¨ Every languages exposes a different format

Tracing is not something new
¨ There are vendors
¨ Every vendor has their own format

log log log
log log
log
Parent Span Span Context / Baggage
Child
Child
Child Span
¨ Spans - Basic unit of timing and causality. Can be tagged with
key/value pairs.
¨ Logs - Structured data recorded on a span.
¨ Span Context - serializable format for linking spans across network
boundaries. Carries baggage, such as a request and client IDs.
¨ Tracers - Anything that plugs into the OpenTracing API to record
information.
¨ ZipKin, Jaeger, LightStep, others
¨ Also metrics (Prometheus) and logging

OpenTracing
API
application logic
µ-service frameworks
Lambda functions
RPC & control-flow frameworks
existing instrumentation
tracing infrastructure
main()
I N S T A N A
J a e g e r
microservice process

import "github.com/opentracing/opentracing-go"
import ".../some_tracing_impl"
func main() {
opentracing.SetGlobalTracer(
// tracing impl specific:
some_tracing_impl.New(...),
)
...
}
https://github.com/opentracing/opentracing-go
Opentracing: Configure the GlobalTracer

func xyz(ctx context.Context, ...) {
...
span, ctx := opentracing.StartSpanFromContext(ctx, "operation_name")
defer span.Finish()
span.LogFields(
log.String("event", "soft error"),
log.String("type", "cache timeout"),
log.Int("waited.millis", 1500))
...
}
Opentracing: Create a Span from the Context

func xyz(parentSpan opentracing.Span, ...) {
...
sp := opentracing.StartSpan(
"operation_name",
opentracing.ChildOf(parentSpan.Context()))
defer sp.Finish()
...
}
Opentracing: Create a Child Span

Golang and Kubernetes: pprof
● It is the Golang native profiler
● You can use it via the `go pprof` command
● `import "runtime/pprof"` writes runtime profiling data
● `import "net/http/pprof"` serves via HTTP server runtime profiling data

package main
import (
"log"
"net/http"
_ "net/http/pprof"
)
func main() {
log.Println(http.ListenAndServe("localhost:6060", nil))
}

$ go tool pprof http://localhost:6060/debug/pprof/heap
Fetching profile over HTTP from http://localhost:6060/debug/pprof/heap
Saved profile in
/home/gianarb/pprof/pprof.main.alloc_objects.alloc_space.inuse_objects.inuse_space.009.pb.gz
File: main
Type: inuse_space
Time: Oct 15, 2018 at 4:22pm (CEST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) png
Generating report in profile001.png

https://github.com/influxdata/influxdb/blob/4cbdc197b8117fee648d62e2e5be75c6575352f0/services/httpd/pprof.go
// handleProfiles determines which profile to return to the requester.
func (h *Handler) handleProfiles(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/debug/pprof/cmdline":
httppprof.Cmdline(w, r)
case "/debug/pprof/profile":
httppprof.Profile(w, r)
case "/debug/pprof/symbol":
httppprof.Symbol(w, r)
case "/debug/pprof/all":
h.archiveProfilesAndQueries(w, r)
default:
httppprof.Index(w, r)
}
}

// archiveProfilesAndQueries collects the following profiles:
// - goroutine profile
// - heap profile
// - blocking profile
// - mutex profile
// - (optionally) CPU profile
//
// It also collects the following query results:
//
// - SHOW SHARDS
// - SHOW STATS
// - SHOW DIAGNOSTICS
//
// All information is added to a tar archive and then compressed, before being
// returned to the requester as an archive file.

var allProfs = []*prof{
{Name: "goroutine", Debug: 1},
{Name: "block", Debug: 1},
{Name: "mutex", Debug: 1},
{Name: "heap", Debug: 1},
}

gz := gzip.NewWriter(&resp)
tw := tar.NewWriter(gz)
// Collect and write out profiles.
for _, profile := range allProfs {
if profile.Name == "cpu" {
if err := pprof.StartCPUProfile(&buf); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
sleep(w, time.Duration(profile.Debug)*time.Second)
pprof.StopCPUProfile()
} else {
prof := pprof.Lookup(profile.Name)
if prof == nil {
http.Error(w, "unable to find profile "+profile.Name, http.StatusInternalServerError)
return
}
if err := prof.WriteTo(&buf, int(profile.Debug)); err != nil {
return
}
}

// Collect and write out the queries.
var allQueries = []struct {
name string
fn func() ([]*models.Row, error)
}{
{"shards", h.showShards},
{"stats", h.showStats},
{"diagnostics", h.showDiagnostics},
}
tabW := tabwriter.NewWriter(&buf, 8, 8, 1, 't', 0)
for _, query := range allQueries {
rows, err := query.fn()
if err != nil {
}

func (h *Handler) showDiagnostics() ([]*models.Row, error) {
diags, err := h.Monitor.Diagnostics()
if err != nil {
return nil, err
}
// Get a sorted list of diagnostics keys.
sortedKeys := make([]string, 0, len(diags))
for k := range diags {
sortedKeys = append(sortedKeys, k)
}
sort.Strings(sortedKeys)
rows := make([]*models.Row, 0, len(diags))
for _, k := range sortedKeys {
row := &models.Row{Name: k}
row.Columns = diags[k].Columns
row.Values = diags[k].Rows
rows = append(rows, row)
}
return rows, nil
}

Remote Debugging
in Kubernetes
Delve, GDB

Remote Debugging
In Kubernetes
DOESN’T APPLY!

That’s not totally true. You can expose the debugger
endpoint. It works. It doesn’t apply for production
workload.

Thanks
@gianarb

Kubernetes debug like a pro

More Related Content

What's hot

Similar to Kubernetes debug like a pro

More from Gianluca Arbezzano

Recently uploaded

Kubernetes debug like a pro