This talk will focus on a brief overview of Kubernetes, with a brief demo, and then more of an in-depth focus on issues we've faced moving PHP projects into Docker and Kubernetes like signal propagation, init systems, and logging. Talk from Cape Town PHP meetup on Feb. 7, 2016: https://www.meetup.com/Cape-Town-PHP-Group/events/237226310/ Code: https://github.com/zoidbergwill/kubernetes-php-examples Slides as markdown: http://www.zoidbergwill.com/presentations/2017/kubernetes-php/index.md

1. Adventures with Laravel and Kubernetes
at superbalist.com
William Stewart
 zatech


 zoidbergwill basically everywhere
1 / 28

2. Intro
Who am I?
What do I do?
2 / 28

3. Overview
Docker
What is Docker?
Why do we use it?
Kubernetes
What is Kubernetes?
Why do we use it?
Some extra tools
A quick demo
War stories
Logging
Signal propagation
Long running processes
Schema changes
3 / 28

4. Docker
4 / 28

5. What is Docker?
5 / 28

6. How does it help?
Language agnostic build artifact
FROM composer:alpine
COPY composer.json composer.lock /app
RUN composer install --ansi --no-interaction --no-scripts --no-dev --no-autoloader
COPY . /app
RUN composer install
or
FROM node:alpine
COPY package.json /app
RUN npm install
COPY . /app
6 / 28

7. How does it help?
No more "works on my machine", because they're the same images locally and on production
Smaller shippable artifacts than full machine images, and easier to independently scale components
A lot of useful services already in the Docker registry:
MySQL
Redis
Memcached
Elastic Search
RabbitMQ
7 / 28

8. Running Docker in production
8 / 28

9. How does it help?
- Container hosting
- Con g changes
- Supervision
- Monitoring
- Rolling deployments
- Networking / Service discovery
- and more...
9 / 28

10. Automate the boring stu f
“Automation is a force multiplier, not a panacea”
Value of automation
Consistency
Extensibility
MTTR (Mean Time To Repair)
Faster non-repair actions
Time savings
Dan Luu's Notes on Google's Site Reliability Engineering book
10 / 28

11. What is Kubernetes?
an open source container
cluster manager
11 / 28

12. How does it help?
Faster/smarter builds than a .tar.gz
Reliable rolling updates (safer than a symlink swap)
Similar build/deploy scripts for all languages and services
Autoscaling (based on CPU usage)
Fine-grained healthchecking
Accurate resource monitoring and better usage
12 / 28

13. Helps us to easily move towards SOA,
Easily start new projects in any language, that can rely on the same service discovery (using DNS)
Discovery via DNS (curl http://elasticsearch/) or automatically set Kubernetes environment
variables (env('ELASTICSEARCH_SERVICE_HOST')), that balances load over a collection of Elastic Search
clients, all automatically updated by Kubernetes.
A huge collection of base Docker les on hub.docker.com and quay.io
13 / 28

14. Avoid "works on my machine" issues
Accurately replicate most of our production environment locally (other than a mocked Google Loadbalancer)
Using:
minikube, which behaves like a single-node Kubernetes cluster
docker, moving towards rocker, to better utilise global Composer/NPM cache
NFS, to allow faster feedback when developing
some bash glue
14 / 28

15. Easily implement monitoring, logging, and LetsEncrypt
Simply implement metrics monitoring and LetsEncrypt using Kubernetes annotations:
annotations:
prometheus.io/port: "9150"
prometheus.io/scrape: "true"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: gce
kubernetes.io/tls-acme: "true"
name: general-ingress
spec:
rules:
- host: tasks.zoidbergwill.com
http:
paths:
- backend:
serviceName: tasks-app
servicePort: 80
path: /
tls:
- hosts:
- tasks.zoidbergwill.com
secretName: tasks-tls
15 / 28

16. Demo tools
Minikube
Rocker
Kube-lego
16 / 28

17. DEMO
DEMO
17 / 28

18. War stories
18 / 28

19. Logging
Logging to a le worked great on compute engine with uentd
Gcloud Stackdriver Logging is pretty amazing, but with container engine it wants stdout
± ag log tasks/config/app.php
112: | Logging Configuration
115: | Here you may configure the log settings for your application. Out of
116: | the box, Laravel uses the Monolog PHP logging library. This gives
117: | you a variety of powerful log handlers / formatters to utilize.
119: | Available Settings: "single", "daily", "syslog", "errorlog"
123: 'log' => env('APP_LOG', 'single'),
125: 'log_level' => env('APP_LOG_LEVEL', 'debug'),
212: 'Log' => IlluminateSupportFacadesLog::class,
19 / 28

20. Logging Monolog/Laravel options
single/daily le:
doesn't go to stdout and get consumed by fluentd-cloud-logging
syslog:
another dependency, and we'd need to install syslog and tail it, or talk to an external syslog daemon
errorlog:
php-fpm logs have a pre x you can't get rid of
Bubbling them to the nginx container also has a gross pre x and means you can't separate the logs for each container
20 / 28

21. Upstream php images also do some gross logging stu f
FROM php:7-fpm
[global]
error_log = /proc/self/fd/2
[www]
access.log = /proc/self/fd/2
Access logs are junk for Laravel:
:: - 07/Feb/2017:13:57:57 +0000 "GET /index.php" 200 (x 1337)
error_log can be pretty crap too:
[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: "NOTICE: PHP message: Array"
[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: "("
[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: " [6] => 363"
[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: ")"
[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: ""
21 / 28

22. Going full circle
Let's try single again:
> cat entrypoint.sh
# Tail a log file, and background that process
tail -f /var/www/html/storage/logs/laravel.log &
# Now run the more important piece
php-fpm7
22 / 28

23. Signal propagation
Linux relies on PID 1 doing important stuff, that an init system does
Most Docker entrypoints use exec to become the main process, but this kills anything running in the script.
Shells (like bash) don't propagate signals, like SIGTERM or SIGQUIT, which messes with graceful shutting down.
When a pod dies:
PID 1 in the container gets a SIGTERM and given a grace period (default: 30s) to shutdown by default
Once the grace period has expired then a SIGKILL signal is sent
dumb-init and lifecycle hooks to the rescue!
dumb-init wraps your ENTRYPOINT shell script and makes sure signals propagate to child processes
lifecycle:
preStop:
exec:
# SIGTERM triggers a quick exit; gracefully terminate instead
command: ["/usr/sbin/nginx","-s","quit"] # or php artisan queue:restart
23 / 28

24. Cron
If you run plain old cron it runs processes in a separate env, so we dump our Kubernetes environment for phpdotenv to
read:
# Cron is a bastard that has sub processes with different env vars.
env > .env
sed -i -e "s/=(.*)$/="1"/g" .env
24 / 28

25. Long running processes
The max "grace period" for a pod is 5 minutes.
Easy steps in the right direction:
Break tasks up as small as possible
Put most tasks on queues
Use Kubernetes "Jobs" when forced to
25 / 28

26. Database migrations
Three step DB Migrations
Migrating before starting a rolling update
26 / 28

27. Superbalist.com and Takealot.com are hiring!!!
Source
https://github.com/zoidbergwill/kubernetes-php-examples
Slides: zoidergwill.github.io/presentations/2017/kubernetes-php/
Links on the next slide!
27 / 28

28. Links
O'Reilly's Site Reliability Engineering: How Google Runs Production Systems Amazon
Dan Luu's notes are good too
Borg, Omega, and Kubernetes Lessons learned from three container-management systems over a decade. Essay
kubernetes-dashboard
Cloud Native Computing Foundation
Kubernetes-Anywhere: An of cial Kubernetes repo with some documentation on running Kubernetes with Docker
for Mac beta
minikube: The of cial Go binary for running a simpler local cluster.
awesome-kubernetes list on GitHub, cuz it has some neat things.
Docker and the PID 1 reaping problem
Containers really are the future
28 / 28