DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned from InfluxDB v1 to v2

Gianluca Arbezzano
Site Reliability Engineer @InﬂuxData
● https://gianarb.it
● @gianarb
What I like:
● I make dirty hacks that look awesome
● I grow my vegetables 🍅🌻🍆
● Travel for fun and work

@gianarb - gianluca@influxdb.com

DevOps likes automation.
Automation likes code.
YAML is not code.

1. You Know!
Your team knows and
use Docker for local
development and
testing
2. Kubernetes!
Everyone speaks
about kubernetes.
3. Hire!
You don’t know why
but you hired a
DevOps that kind of
know k8s.
3. Excitement!
You are moving
everything and
everyone to
kubernetes

We need to make our
hands dirty

Spin up a cluster that you
can break
Bring developers in the loop

Deploy CI on Kubernetes

Run your code in prod

K8s as code: From YAML to code (golang)
1. You have the ability to use Golang autocomplete as documentation, reference for every
kubernetes resources
2. You feel less a YAML engineer (great feeling btw)
3. Code is better than YAML! You can reuse it, compile it, embed it in other projects.

Tiny cli
to make
the
migration
to golang
Some
manual
refactoring

Tiny cli
to make
the
migration
to golang
Some
manual
refactoring
● Continue to improve our CI to validate that YAML and Go file are the same,
and the resources in Kubernetes are like the Go file.
● Maybe we will be able to remove the YAML at some point.

GitOps
Your Git repository is the entrypoint for all your code changes.
Infrastructure is ‘as code’, so the place where you make it happen should be Git.
Read More on weave.com
https://www.weave.works/technologies/gitops/

Don’t be scared and write your
own tools!

Why Kubernetes
is so powerful, complex
and widely adopted?

What do you do
to justify these costs?

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "drone.fullname" . }}-agent
labels:
app: {{ template "drone.name" . }}
chart: "{{ .Chart.Name }}-{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
component: agent
spec:
replicas: {{ .Values.agent.replicas }}
template:
metadata:
annotations:
checksum/secrets: {{ include (print $.Template.BasePath "/secrets.yaml") . | sha256sum }}
{{- if .Values.agent.annotations }
{{ toYaml .Values.agent.annotations | indent 8 }
{{- end }}
labels:
app: {{ template "drone.name" . }}
release: "{{ .Release.Name }}"
component: agent

API are
the keys for
your success!
Image credit: Pixabay

containerd.io

We use docker as
replacement for systemd
for process management

DIND - Docker in Docker
$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker sh
$ docker info
Containers: 48
Running: 1
Paused: 0
Stopped: 47
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
runc version: 871ba2e58e24314d1fab4517a80410191ba5ad01
init version: fec3683
Kernel Version: 4.20.13-arch1-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.42GiB
Name: gianarb

UX for OPS.
Because everyone needs to feel
like at home...

Instrumentation, observability
and monitoring

~ @gianarb - https://gianarb.it ~
The secret is all about how do you
combine things together

Metric
s

Logs

Traces

Often our
aggregations looks
a bit twisted...

@gianarb - gianluca@influxdb.com
Distributed Tracing
Tracing is a way to correlate
logs using a set of IDs

Instrumentation code is a first citizen in your
codebase: OpenCensus
● Open Source project sponsored by Google
● It is a SPEC plus a set of libraries in different languages to instrument your
application
● To collect metrics, traces and events.

OpenCensus
Common
Interface to
collect stats
and traces
from your app
Different
exporters to
persist your
data

gianarb.it ~ @gianarb
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"} 3 1395066363000
# Escaping in label values:
msdos_file_access_time_seconds{path="C:DIRFILE.TXT",error="Cannot find file:n"FILE.TXT""}
1.458255915e9
# Minimalistic line:
metric_without_timestamp_and_labels 12.47
# A weird metric from before the epoch:
something_weird{problem="division by zero"} +Inf -3982045
# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

OpenMetrics
v2 Prometheus exposition format

gianarb.it ~ @gianarb
func FetchMetricFamilies(url string, ch chan<- *dto.MetricFamily, certificate string, key string,
skipServerCertCheck bool) error {
defer close(ch)
var transport *http.Transport
if certificate != "" && key != "" {
cert, err := tls.LoadX509KeyPair(certificate, key)
if err != nil {
return err
}
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert},
InsecureSkipVerify: skipServerCertCheck,
}
tlsConfig.BuildNameToCertificate()
transport = &http.Transport{TLSClientConfig: tlsConfig}
} else {
transport = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: skipServerCertCheck},
}
}
https://github.com/prometheus/prom2json/blob/master/prom2json.go#L123

Summary:
★ Do not be scared to write your code!
★ Use the API
★ Instrumentation code is a ﬁrst class
citizen
★ Keep calm and observe all together!

Credits and Links
¨ https://www.weave.works/technologies/gitops/
¨ http://gianarb.it
¨ https://thenewstack.io/why-you-cant-aﬀord-to-ignore-distributed-tracing-for-observability/
¨ https://www.honeycomb.io/blog/
¨ https://gianarb.it/blog/infra-as-code-short-long-ttl-resource
¨ https://gianarb.it/blog/kubernetes-shared-informer
¨ https://github.com/OpenObservability/OpenMetrics
¨ https://promcon.io/2018-munich/slides/openmetrics-transforming-the-prometheus-exposition-format-into-a
-global-standard.pdf
¨ https://medium.com/opentracing/merging-opentracing-and-opencensus-f0fe9c7ca6f0

Thanks
@gianarb

DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned from InfluxDB v1 to v2

More Related Content

What's hot

Similar to DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned from InfluxDB v1 to v2

More from DevOps_Fest

Recently uploaded

DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned from InfluxDB v1 to v2