Devoxx France 2018 : Mes Applications en Production sur Kubernetes

#DevoxxFR
Devoxx France 2018
Mes Applications en Production
sur Kubernetes
Michael Morello
@barkbay

#DevoxxFR
About me
MICHAEL MORELLO
deploy, manage, maintain { , }
Kubernetes
@
},
GO
,
} developer

#DevoxxFR
Kubernetes ?
•C’est un « cluster manager » :
K8S gère une flotte de machines (physiques ou virtuelles)
•C’est un ensemble d’ «objets » :
K8S permet de déclarer l’état attendu d’une application
•Pilotable par API :
Référence : https://kubernetes.io/docs/concepts/

#DevoxxFR
Observability
Security
Resilience

#DevoxxFR
POD ?
Interface réseau commune aux conteneurs
Partage de système de fichiers
Colocalisés sur un
même serveur

#DevoxxFR
Un POD ?
metadata:
labels:
app: lab-java
spec:
containers:
- name: lab
image: barkbay/k8s-app-lab:java-v0
ports:
- containerPort: 8080
Une liste de
conteneurs
Quelques
métadonnées

#DevoxxFR
A security context defines privilege
and access control settings for a
Pod or Container :
• User ID
• Linux Capabilities
• SELinux labels
• AllowPrivilegeEscalation
Security context

#DevoxxFR
SecurityContext
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1234
fsGroup: 2000
containers:
- name: lab
securityContext:
allowPrivilegeEscalation: false
ports:
SecurityContext
PodSecurityContext

#DevoxxFR
« SCCs are objects that define a set of conditions that a pod must run
with in order to be accepted into the system. »
TL;DR : Les SCCs permettent d’appliquer un contexte de sécurité par
défaut sur les PODs.
PSP : Pod Security Policy is a cluster-level resource that controls
security sensitive aspects of the pod specification.
OU
Un SecurityContext automatique ?

#DevoxxFR
Comprendre les SecurityContext, travailler avec vos OPS sur la mise en œuvre des PSP (ou
utilisez Openshift)
SELinux : "Every time you run setenforce 0, you make Dan Walsh weep. Dan is a nice guy and
he certainly doesn't deserve that. »
Utiliser des namespaces dédiés
Utiliser des ServiceAccount : des comptes techniques qui vous permettront de jouer avec les
RBAC
Quelle sécurité pour les flux applicatifs ? TLS de bout en bout ?
Security takeaway

#DevoxxFR
Gestion des ressources partagées

#DevoxxFR
Multi-tenant : Share Cpu and memory
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
containers:
- name: lab
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "192Mi"
cpu: "2"
securityContext:
allowPrivilegeEscalation: false
ports:
Limits control the maximum amount of
resources that the container may use
The scheduler uses resources
requests to find a node with an
appropriate fit for all containers in a
POD.

#DevoxxFR
containers:
- name: lab
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "192Mi"
cpu: "2"
« Converted to its millicore value and
multiplied by 100. The resulting value is the
total amount of CPU time that a container
can use every 100ms. A container cannot
use more than its share of CPU time during
this interval. »
On appelle ça faire du Throttling

#DevoxxFR
containers:
- name: lab
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "192Mi"
cpu: "2"
"GC task thread#0 (ParallelGC)" […] runnable
Tuning automatique
de la JVM
Runtime.getRuntime()
.availableProcessors() = 4

#DevoxxFR
containers:
- name: lab
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "192Mi"
cpu: "2"
$ cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
200000
$ cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
$ expr 200000 / 100000
2 <= ~= 2 CPUs disponibles
-XX:ParallelGCThreads=2
-XX:ConcGCThreads=2
-Djava.util.concurrent.ForkJoinPool.common.parallelism=2
-XX:CICompilerCount=2
Java 8
-XX:ActiveProcessorCount=2
https://docs.oracle.com/javase/10/tools/java.htm
Java 10

#DevoxxFR
Monitoring CPU cgroup
$ cat /sys/fs/cgroup/cpu/cpu.stat
user 1637
system 88
nr_periods 520
nr_throttled 364 : number of times tasks in a cgroup have been
throttled
throttled_time 72988838516 : the total time duration (in
nanoseconds) for which tasks in a cgroup have been throttled.
1

#DevoxxFR
Memory cgroup
PAGE
CACHE
FREE
RECLAIMABLE MEMORY
CGROUP MANAGED MEMORY
Java Virtual Machine
HEAP
Native
Memory
USED

#DevoxxFR
Memory cgroup
F
R
E
E
RECLAIMABLE
MEMORY ?
CGROUP MANAGED MEMORY
Java Virtual Machine
HEAP
Native
Memory
USED

#DevoxxFR
OOM-KILLER In Action
java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[…]
memory: usage 196608kB, limit 196608kB, failcnt 1953
[…]
[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[25616] 1000 25616 254 1 4 0 -998 pause
[25687] 1000 25687 678075 48764 165 0 872 java
Memory cgroup out of memory: Kill process 25908 (java) score 1864 or
sacrifice child
Killed process 25687 (java) total-vm:2712300kB, anon-rss:191448kB, file-
rss:3520kB, shmem-rss:0kB
The failcnt field gives the number of times that the
cgroup limit was exceeded.
limits:
memory: "192Mi"

#DevoxxFR
Avoid OOM-Killer with Java 8
$ # Dans le conteneur
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
402653184 #384Mo max
$ # A vous de calculer le Xmx qui va bien
ou
-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap

#DevoxxFR
Interlude « Collectons les Métriques »

#DevoxxFR
Métriques ?
container_cpu_cfs_throttled_seconds_total{container_name="foo"} 1027 1395066363000
Metric name
Label
Value Timestamp
GET /metrics HTTP/1.0
PROMETHEUS

#DevoxxFR
Prometheus
PROMETHEUS
ALERTING

#DevoxxFR
Fin de l’Interlude « Collectons les Métriques »

#DevoxxFR
Monitoring containers limits
• container_cpu_cfs_throttled_periods_total
• container_cpu_cfs_throttled_seconds_total
• container_memory_failcnt

#DevoxxFR
Monitoring your own metrics
kind: Service
apiVersion: v1
metadata:
name: lab-java-service
annotations:
prometheus.io/scrape: "true"
spec:
selector:
app: lab-java
ports:
- protocol: TCP
port: 80
targetPort: 8080
+
endpoint_hello_total{status="get",} 1606.0
Implement call to /metrics :

#DevoxxFR
HPA : Horizontal Pod Autoscaler
COREAPICustomMetricAPI
API
POD
de
Mediation
scale up !H.P.A.
PROMETHEUS
/metrics
POD POD POD
GET /apis/custom.metrics.k8s.io/[…]/lab-java-service/endpoint_hello
42

#DevoxxFR
Is it alive ?
spec:
containers:
- name: lab
livenessProbe:
httpGet:
path: /hello
port: 8080
readinessProbe:
httpGet:
path: /hello
port: 8080
initialDelaySeconds: 5
periodSeconds: 2
ports:
Ouvrir les flux ?
Redémarrer
le conteneur ?

#DevoxxFR
Is it alive ?
spec:
containers:
- name: lab
livenessProbe:
tcpSocket:
port: 8080
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 2
ports:
Ouvrir les flux ?
Redémarrer
le conteneur ?

#DevoxxFR
Pod Disruption Budget (a.k.a. PDB)
En cas de "disruption" "volontaire" permet de maintenir un nombre minimum
d’instances.
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: lab-java-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: lab-java

#DevoxxFR
Takeaway
•Security first
•Exposer des métriques
•Collecter des métriques
•Surveiller :
•cgroups : memory and cpu
•application restarts
•events dans les namespaces
•Implementer des tests Liveness and Readiness simples

#DevoxxFR
Merci / Thank you
Code source de l’application :
https://github.com/barkbay/k8s-app-lab/

#DevoxxFR
We love picture
We try to keep the Devox France logo and the Tweet
hashtag on all slides
3

Devoxx France 2018 : Mes Applications en Production sur Kubernetes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Devoxx France 2018 : Mes Applications en Production sur Kubernetes

Similar to Devoxx France 2018 : Mes Applications en Production sur Kubernetes (20)

Recently uploaded

Recently uploaded (20)

Devoxx France 2018 : Mes Applications en Production sur Kubernetes

Editor's Notes