Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro

Kubernetes: Shifting the mindset from servers to containers
DB Systel GmbH | Schlomo Schapiro | Chief Architect Cloud, Chief Technology Office | @schlomoschapiro | 23.03.2018
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License
DB13243 © Deutsche Bahn AG / Volker Emersleben

Did you ever use this in a Docker image?
ssh
cron
supervisord
daemontools
upstart
systemd
runit
runas
su
run.sh
DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20182

A “typical“ server ...
3 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018
SSH CRON
logrotate
Backup
Postfix
Rsyslogatd
dbus-daemon
Apache
PHP App
PHP App
MySQL
DB
DB
man-db
dpkg

A “typical“ server is 50% cruft ... which should be centralized
SSH CRON
logrotate
Backup
Postfix
Rsyslogatd
dbus-daemon
Apache
PHP App
PHP App
MySQL
DB
DB
man-db
dpkg
cruft
This is the
„real“ server

SSH CRON
logrotate
Backup
Postfix
Rsyslogatd
dbus-daemon
Apache
PHP App
PHP App
MySQL
DB
DB
man-db
dpkg

SSH CRON
logrotate
Backup
Postfix
Rsyslogdbus-daemon
atd
Apache
PHP App
PHP App
MySQL
DB
DB
man-db
dpkg
Cluster-wide orchestration,
scaling, monitoring and
deployment of processes
Great declarative
description for all
IaaS needs
is an abstraction layer

SSH CRON
logrotate
Backup
Postfix
Rsyslogatd
Apache
PHP App
PHP App
MySQL
DB
DB
= containers

Platform features
SSH CRON
logrotate
Backup
Postfix
Rsyslogatd
Apache
PHP App
PHP App
MySQL
DB
DB
containers
isolation – packaging – deployment – immutable systems
Application containers

Application Life Cycle
Build Deploy & Configure Initialize
&
Run
Maintain
as Linux Processes

Build
on Kubernetes
Docker Build

Deploy & Configure
on Kubernetes
Config Maps
Secrets
Pod Spec

on a traditional server
/etc/init.d/app
/usr/sbin/app
/etc/init.d/cron
/usr/sbin/cron
/etc/cron.daily/app
Initialize
&
Run
Maintain
Exclusive access
Prepare data files
Restore data
Apply schema upgrade
Backup data
Cleanup stale data
Run application

on
Initialize
&
Run
Maintain
Backup data
Cleanup stale data
Init Container
Main
Container
Maintenance
Container
Run application
Exclusive access
Prepare data files
Restore data

on
Initialize
&
Run
Maintain
Init Container
Main
Container
Maintenance
ContainerPod

Running a Pod with multiple Containers
init main
maintenance
t
DATA
S3 BACKUP
Backup, clean up stale data ...
Restore
if needed

Running a Pod with multiple Containers
init main
maintenance
t
DATA
S3 BACKUP
Backup, clean up stale data ...
Restore
if needed
Exclusive access
Prepare data files
Restore data
Run application
Backup data
Cleanup stale data
Run applicationCoordination!

What happens with the „cruft“?
SSH CRON
logrotate
Postfix
Rsyslogatd
dbus-daemon
man-db
dpkg

CRON
• Sends out emails
• Forks multiple processes , breaking the one task per container paradigm
• Not optimized for running single task
• Doesn‘t correctly handle INT/KILL signals
• Doesn‘t log to STDOUT / STDERR
• Cannot configure schedule and cron jobs via environment variables
... made for servers and not containers
CRON

CRON for a single job
#!/bin/bash
RUNAT=${RUNAT:-1 minute}
function wait_for_maintenance_time {
sleep_time=$(( $(date -d "$RUNAT" +%s ; echo - ; date +%s) ))
if (( sleep_time < 0 )) ; then
sleep_time=$(( 24*60*60 + sleep_time )) # wait till next day same time
fi
if (( sleep_time > 0 )) ; then
echo "Waiting $sleep_time seconds till $RUNAT before starting maintenance"
sleep $sleep_time
else
echo "Not waiting $sleep_time seconds"
fi
}
while true ; do
wait_for_maintenance_time
# do some maintenance, e.g. backup data or purge old stuff
done
https://goo.gl/EqSBJU

Email
Old server interfaces
• /usr/lib/sendmail
• /usr/bin/mail
• SMTP to 127.0.0.1:25
• trust based on „same host“
• implicit configuration by convention
Kubernetes alternatives
• cluster service for SMTP:
smtp.mynamespace.svc.cluster.local.
• trust based on „same cluster“ or
dedicated authentication
• configure via environment variables,
e.g. MAILHOST

Secure Shell
One tool – many purposes
• SSH for admin access
• SSH for automation between servers
• SSH for pull backup
SSH on Servers
• Admins are (local) users
• Technical users for automation
• Authentication with passwords or
static SSH keys
• ~/.ssh/authorized_keys as
command filter for some SSH keys
• Kubernetes provides Admin access:
kubectl exec
• authentication with Kubernetes
temporary credentials
Anti-patterns
• SSH between pods:
Application cluster probably not
aware of pods coming and going
• User authentication in pods:
Pointless as pods run non-privileged
and SSH deamon cannot switch user

Logs
Typical logging interfaces
• Syslog: /dev/log
• Syslog: UDP 127.0.0.1:514
• Write to log file:
• /var/log/messages
• /var/log/auth.log
• /data/myapp/some.log
• ...
• /dev/stdout is primary logging
interface for applications and
containers
• Kubernetes handles logging
Anti-patterns
• Custom log file:
You‘ll need an extra sidecar container
to read this log
• Syslog server:
Set up sidecar container to listen on
UDP:514 and write to STDOUT

Live
Demo
https://commons.wikimedia.org/wiki/File:MacBook_Pro,_Late-2008.jpg

Demo: WebDAV server with user-provided data and backup to GitHub
WebDAV
Server
„main“
Container
/mediaRead / Write
Data
Restore Backup
Configure git Repo
„init“
Container
Create Backup
Upload to GitHub
„backup“
Container
Demo only, git is no backup tool: http://blog.codekills.net/2009/12/08/using-git-for-backup-is-asking-for-pain/

Containers
kind: Deployment
spec:
template:
spec:
initContainers:
- name: init
... more container spec
containers:
- name: main
- name: backup
volumes:
- name: media
emptyDir: {}

initContainers:
- name: init
image: schlomo/ssh-url-with-ssh-key
volumeMounts:
- mountPath: /media
name: media
command:
- /bin/bash
- -exc
- |
test -d /media/.git && exit 0
ssh-keyscan github.com >/etc/ssh/ssh_known_hosts 2>/dev/null
git clone --depth 1 git~LS0t...tLQo=@github.com:schlomo/demo-data.git /media
chmod 700 /media/.git
chown -R 33:33 /media
chown -R 0:0 /media/.git
cd /media
git config user.email "demo$RANDOM$RANDOM@nowhere$RANDOM$RANDOM.com"
git config user.name "Demo $RANDOM"
init
Hack: https://goo.gl/gqjfuy

containers:
- name: main
image: sashgorokhov/webdav
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- mountPath: /media
name: media
main

containers:
- name: backup
image: schlomo/ssh-url-with-ssh-key
volumeMounts:
- mountPath: /media
name: media
command:
- /bin/bash
- -exc
- |
cd /media
ssh-keyscan github.com >/etc/ssh/ssh_known_hosts 2>/dev/null
while true ; do
sleep 15
git add -A && git commit -a -m "$(date)" && git push
done
backup

https://commons.wikimedia.org/wiki/File:MacBook_Pro,_Late-2008.jpg
Live
Demo

Pod with
multiple
containers
Recap: Multiple Containers in One Pod
App Service
SSH Service
Cron Service
/usr/sbin/appd
/etc/init.d/app
/usr/sbin/crond
/etc/init.d/crond
/usr/sbin/sshd
/etc/init.d/ssh
Prepare data files
Restore data
Backup data
Clean up stale data
Computer Linux Processes
init
main
maintenance

@schlomoschapiro
Throw away the old ideas,
use the Kubernetes way!
Blog, Slides & Code: goo.gl/EqSBJU
Feedback: go.schapiro.org/feedback

Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro

Recommended

Recommended

More Related Content

What's hot

What's hot (12)

Similar to Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro

Similar to Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro (20)

More from Schlomo Schapiro

More from Schlomo Schapiro (20)

Recently uploaded

Recently uploaded (20)

Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro