Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro

2,022 views

Published on

Kubernetes: Shifting the mindset from servers to containers
With Kubernetes pods and containers several fundamental assumptions of server operations don't apply any more. Some Linux services like SSH even disappear and are provided by Kubernetes instead.

This talk explores the mindset shift that developers and admins of Linux servers have to do in order to fully take advantage of the power of a Kubernetes cluster:

* Servers turn into pods
* Linux application services turn into containers
* Standard services like cron and SSH disappear completely
* How to separate between initialization, run and maintenance phases
* Building pods with multiple containers that work together

Following practical examples from real migration projects participants gain a new understanding of the role of services, init scripts, cron jobs and other standard Linux components. Key takeaways are a better understanding of how to model a complex system on top of Kubernetes and practical tips for migrating servers into Kubernetes containers.

Successfully adopting Kubernetes requires a big change in how developers and admins think about servers - bigger than any change before. Bigger than the change brought by VMs. This talk shows why it pays to change traditional concepts and to embrace the new world of Linux services modularization that Kubernetes stands for.

See Using Kubernetes with Multiple Containers for Initialization and Maintenance (http://blog.schlomo.schapiro.org/2017/06/using-kubernetes-with-multiple.html) for more information and a demo.

Published in: Technology
  • Be the first to comment

Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro

  1. 1. Kubernetes: Shifting the mindset from servers to containers DB Systel GmbH | Schlomo Schapiro | Chief Architect Cloud, Chief Technology Office | @schlomoschapiro | 23.03.2018 This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License DB13243 © Deutsche Bahn AG / Volker Emersleben
  2. 2. Did you ever use this in a Docker image? ssh cron supervisord daemontools upstart systemd runit runas su run.sh DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20182
  3. 3. A “typical“ server ... 3 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Backup Postfix Rsyslogatd dbus-daemon Apache PHP App PHP App MySQL DB DB man-db dpkg
  4. 4. A “typical“ server is 50% cruft ... which should be centralized 4 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Backup Postfix Rsyslogatd dbus-daemon Apache PHP App PHP App MySQL DB DB man-db dpkg cruft This is the „real“ server
  5. 5. 5 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Backup Postfix Rsyslogatd dbus-daemon Apache PHP App PHP App MySQL DB DB man-db dpkg
  6. 6. DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20186 SSH CRON logrotate Backup Postfix Rsyslogdbus-daemon atd Apache PHP App PHP App MySQL DB DB man-db dpkg Cluster-wide orchestration, scaling, monitoring and deployment of processes Great declarative description for all IaaS needs is an abstraction layer
  7. 7. DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20187 SSH CRON logrotate Backup Postfix Rsyslogatd Apache PHP App PHP App MySQL DB DB = containers
  8. 8. Platform features DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20188 SSH CRON logrotate Backup Postfix Rsyslogatd Apache PHP App PHP App MySQL DB DB containers isolation – packaging – deployment – immutable systems Application containers
  9. 9. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20189 Build Deploy & Configure Initialize & Run Maintain as Linux Processes
  10. 10. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201810 Build on Kubernetes Docker Build
  11. 11. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201811 Deploy & Configure on Kubernetes Config Maps Secrets Pod Spec
  12. 12. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201812 on a traditional server /etc/init.d/app /usr/sbin/app /etc/init.d/cron /usr/sbin/cron /etc/cron.daily/app Initialize & Run Maintain Exclusive access Prepare data files Restore data Apply schema upgrade Backup data Cleanup stale data Run application
  13. 13. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201813 on Initialize & Run Maintain Backup data Cleanup stale data Init Container Main Container Maintenance Container Run application Exclusive access Prepare data files Restore data Apply schema upgrade
  14. 14. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201814 on Initialize & Run Maintain Init Container Main Container Maintenance ContainerPod
  15. 15. Running a Pod with multiple Containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201815 init main maintenance t DATA S3 BACKUP Backup, clean up stale data ... Restore if needed
  16. 16. Running a Pod with multiple Containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201816 init main maintenance t DATA S3 BACKUP Backup, clean up stale data ... Restore if needed Exclusive access Prepare data files Restore data Apply schema upgrade Run application Backup data Cleanup stale data Run applicationCoordination!
  17. 17. What happens with the „cruft“? 17 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Postfix Rsyslogatd dbus-daemon man-db dpkg
  18. 18. CRON • Sends out emails • Forks multiple processes , breaking the one task per container paradigm • Not optimized for running single task • Doesn‘t correctly handle INT/KILL signals • Doesn‘t log to STDOUT / STDERR • Cannot configure schedule and cron jobs via environment variables ... made for servers and not containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201818 CRON
  19. 19. CRON for a single job DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201819 #!/bin/bash RUNAT=${RUNAT:-1 minute} function wait_for_maintenance_time { sleep_time=$(( $(date -d "$RUNAT" +%s ; echo - ; date +%s) )) if (( sleep_time < 0 )) ; then sleep_time=$(( 24*60*60 + sleep_time )) # wait till next day same time fi if (( sleep_time > 0 )) ; then echo "Waiting $sleep_time seconds till $RUNAT before starting maintenance" sleep $sleep_time else echo "Not waiting $sleep_time seconds" fi } while true ; do wait_for_maintenance_time # do some maintenance, e.g. backup data or purge old stuff done https://goo.gl/EqSBJU
  20. 20. Email Old server interfaces • /usr/lib/sendmail • /usr/bin/mail • SMTP to 127.0.0.1:25 • trust based on „same host“ • implicit configuration by convention DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201820 Kubernetes alternatives • cluster service for SMTP: smtp.mynamespace.svc.cluster.local. • trust based on „same cluster“ or dedicated authentication • configure via environment variables, e.g. MAILHOST
  21. 21. Secure Shell One tool – many purposes • SSH for admin access • SSH for automation between servers • SSH for pull backup SSH on Servers • Admins are (local) users • Technical users for automation • Authentication with passwords or static SSH keys • ~/.ssh/authorized_keys as command filter for some SSH keys DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201821 Kubernetes alternatives • Kubernetes provides Admin access: kubectl exec • authentication with Kubernetes temporary credentials Anti-patterns • SSH between pods: Application cluster probably not aware of pods coming and going • User authentication in pods: Pointless as pods run non-privileged and SSH deamon cannot switch user
  22. 22. Logs Typical logging interfaces • Syslog: /dev/log • Syslog: UDP 127.0.0.1:514 • Write to log file: • /var/log/messages • /var/log/auth.log • /data/myapp/some.log • ... DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201822 Kubernetes alternatives • /dev/stdout is primary logging interface for applications and containers • Kubernetes handles logging Anti-patterns • Custom log file: You‘ll need an extra sidecar container to read this log • Syslog server: Set up sidecar container to listen on UDP:514 and write to STDOUT
  23. 23. Live Demo https://commons.wikimedia.org/wiki/File:MacBook_Pro,_Late-2008.jpg
  24. 24. Demo: WebDAV server with user-provided data and backup to GitHub DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201824 WebDAV Server „main“ Container /mediaRead / Write Data Restore Backup Configure git Repo „init“ Container Create Backup Upload to GitHub „backup“ Container Demo only, git is no backup tool: http://blog.codekills.net/2009/12/08/using-git-for-backup-is-asking-for-pain/
  25. 25. Containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201825 kind: Deployment spec: template: spec: initContainers: - name: init ... more container spec containers: - name: main ... more container spec - name: backup ... more container spec volumes: - name: media emptyDir: {}
  26. 26. initContainers: - name: init image: schlomo/ssh-url-with-ssh-key volumeMounts: - mountPath: /media name: media command: - /bin/bash - -exc - | test -d /media/.git && exit 0 ssh-keyscan github.com >/etc/ssh/ssh_known_hosts 2>/dev/null git clone --depth 1 git~LS0t...tLQo=@github.com:schlomo/demo-data.git /media chmod 700 /media/.git chown -R 33:33 /media chown -R 0:0 /media/.git cd /media git config user.email "demo$RANDOM$RANDOM@nowhere$RANDOM$RANDOM.com" git config user.name "Demo $RANDOM" init DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201826 Hack: https://goo.gl/gqjfuy
  27. 27. containers: - name: main image: sashgorokhov/webdav ports: - containerPort: 80 protocol: TCP volumeMounts: - mountPath: /media name: media main DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201827
  28. 28. containers: - name: backup image: schlomo/ssh-url-with-ssh-key volumeMounts: - mountPath: /media name: media command: - /bin/bash - -exc - | cd /media ssh-keyscan github.com >/etc/ssh/ssh_known_hosts 2>/dev/null while true ; do sleep 15 git add -A && git commit -a -m "$(date)" && git push done backup DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201828
  29. 29. https://commons.wikimedia.org/wiki/File:MacBook_Pro,_Late-2008.jpg Live Demo
  30. 30. Pod with multiple containers Recap: Multiple Containers in One Pod App Service SSH Service Cron Service /usr/sbin/appd /etc/init.d/app /usr/sbin/crond /etc/init.d/crond /usr/sbin/sshd /etc/init.d/ssh Prepare data files Restore data Apply schema upgrade Backup data Clean up stale data Computer Linux Processes init main maintenance
  31. 31. @schlomoschapiro DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 22.03.201831 Throw away the old ideas, use the Kubernetes way! Blog, Slides & Code: goo.gl/EqSBJU Feedback: go.schapiro.org/feedback
  32. 32. DB13243 © Deutsche Bahn AG / Volker Emersleben Thank you for your attention

×