SlideShare a Scribd company logo
1 of 100
Download to read offline
Containerising legacy applications
with dynamic file-based
configurations and secrets
DevOps Days Toronto, May 2019


Andrew Kirkpatrick

@magickatt
Question 1:
How many of you have what you
consider to be “legacy”
applications at your place of work?
Question 2:
For those who do have legacy
applications, how many of you have no
previous maintainers (committers) of that
code still employed at your place of work?
Question 3:
For those who have legacy applications you
cannot (easily) maintain, how many of you
could not (easily) redeploy those applications
on brand new infrastructure in a disaster
recovery scenario?
This presentation is not about
‣ How wonderful Kubernetes/Nomad/DCOS is (or isn’t…)
‣ What “DevOps” culture should look or feel like
‣ How you can proxy traffic halfway round the moon and
produce pretty confusing distributed tracing graphs
‣ How you can observe even more metrics that you’ll
probably never look at
This is more about the harsh reality that there is
always legacy code…
http://www.commitstrip.com/en/2017/12/12/the-code-is-always-better-on-the-other-side/
http://www.commitstrip.com/en/2016/02/15/our-companys-greatest-project/
So why would anyone want to
migrate their application(s) into
containers without rewriting
code?
‣ Too busy working on newer (micro)services
‣ Avoid touching the nightmare codebase that the
long-gone Developer(s) that worked on it left behind
‣ Business wants to fast-track the cost-saving benefits
of containerisation
https://insights.stackoverflow.com/survey
Would you be able to even if you could?
‣Availability of hireable
Developers?
‣Prohibitive cost of
remaining “experts”
‣Un-decipherable yet
crucial business logic
‣Maintenance/
documentation of older
libraries/packages/
frameworks/tooling
0
17.5
35
52.5
70
2019 2018 2017 2016 2015 2014 2013
Javascript Java PHP Ruby
Node C# Python C
Assumptions
‣ You can’t change the code in your application(s)
‣ Familiarity
‣ Time
‣ Permissions/licensing
‣ You don’t want to change the code in your application(s)
‣ Bugs
‣ Liability
‣ You want to take the path of least resistance
Steps
‣ Lets find a legacy application
‣ Run your application in a container
‣ (re)deploy your application using templated
static configuration
‣ Update your application on-the-fly using
dynamic configuration
‣ Avoid things that might catch you out
Lets find a legacy
application…
Old(er) application candidate
‣ Must have been designed pre-containerisation
‣ Must rely on filesystem-based configuration, which
it may update itself
‣ Likely makes assumptions about running on a
single server (thus a single filesystem, etc.)
‣ Has little/no examples of a quickstart using a
container orchestrator
Lets pick an
old(er)
application
What about
WordPress?
Primarily relies on
wp-config.php
https://codex.wordpress.org/Editing_wp-config.php
Lets pick an
old(er)
application
Alright then, perhaps
something less obvious but
just as ubiquitous? phpBB?
Primarily relies on
config.php

https://www.phpbb.com/support/docs/en/3.2/kb/article/rebuilding-your-configphp-file/
Lets pick an
old(er)
application
Need to go a bit more
obscure… how about
another forum, MyBB?
Primarily relies on
config.php and
settings.php
Found the right era of application
‣ “Once you’ve uploaded your
files you will need to set the
permissions on certain files and
directories.
‣ Before granting certain
files and directories
chmod 777, you may want to
try chmod 755 or chmod 775.”
https://docs.mybb.com/1.8/install/#file-permissions
http://www.commitstrip.com/en/2016/06/29/chmod-what/
Older applications live on
Codebases like these are still
being actively used and
maintained despite their longevity
‣ MyBB was first released on 9th
December 2005 (v1.0.0)
‣ Last released on 27th February
2019 (v1.8.20)
Still plenty of issues being
reported and resolved
7 days ago,
13 days ago,
15 days
ago…
MyBB
‣ Majority of the file-based configuration for MyBB
resides in 2 files
‣ inc/config.php
‣ inc/settings.php
‣ Note that these are not JSON/YAML or even
INI/XML files, but language-specific configuration
files
config.php
Environment configuration
such as database
connectivity, logging, IP
blacklisting, etc.
Essential to the
bootstrapping of the
application
settings.php
Application-specific settings
pre-database configuration
load, such as formatting,
routing, basic settings and
some string replacements.
Example of configuration
that is traditionally updated
by the application itself

/********************************* 

DO NOT EDIT THIS FILE, PLEASE USE 

THE SETTINGS EDITOR 

*********************************/
Demo 0
Run MyBB using the PHP built-in web server
Demo 0 key points
‣ Only works on my laptop with a filesystem-based database
‣ Local development server

php -S localhost:9000 -t application
‣ SQLite database for simplicity
https://www.php.net/manual/en/features.commandline.webserver.php
Run your application in
a container
Run your application in a container
First, how do you run your application in a container
that resembles your existing environment?
‣ Bake a base container image re-using existing
provisioning
‣ Modifying your build pipeline to create container
images
‣ Additional environment configuration considerations
(external libraries, vendor integrations, etc.)
What is your “base”?
How bespoke is your application runtime environment?
‣ How are you currently provisioning yourVM/bare
metal fleet?

(hopefully using infrastructure automation? 🤞)
‣ Do you compile your own packages/libraries/
drivers and/or maintain your own Debian/RPM/etc.
repository?
‣ How many other configuration changes are you
making to hosts?
Container tutorial assumptions
Problem with most containerisation tutorials is that they
assume that your application will run with near-vanilla
configuration in a vanilla environment

https://docs.docker.com/samples/library/php/#example
Container tutorial assumptions
… and to keep the images small, they strip a lot of “extras”

https://docs.docker.com/samples/library/php/

https://www.php.net/manual/en/book.mysqli.php

https://www.php.net/manual/en/book.memcached.php
MySQL Improved Extension Memcached Extension
Container tutorial assumptions
‣ You may end up with a rather messy Dockerfile
‣ Also AUFS has a 42 layer limit (or used to)
‣ Easier to use multi-stage builds, or “bakes”
https://docs.docker.com/samples/library/php/

https://www.php.net/manual/en/book.mysqli.php

https://www.php.net/manual/en/book.memcached.php
Why not just use Dockerfiles?
Developers will often ask…

“why not just use a Dockerfile to install the bare minimum?”
or “can’t we just use a community image?”
‣ “the bare minimum” could be hundreds of packages or
binaries, custom configured or compiled
‣ If using a different distribution than the rest of your
fleet, do you really know if it’s secure?
Why not just use Dockerfiles?
CVE-2019-5021 “Alpine Linux Docker Images
Shipped for 3 Years with Root Accounts Unlocked
Alpine Linux Docker images available via the Docker Hub contained a
critical flaw allowing attackers to authenticate on systems using the root
user and no password.”
“This CVE does not impact Alpine distros that are not delivered as
Docker images.”
https://threatpost.com/alpine-linux-docker-images-unlocked/144542/

https://blog.aquasec.com/cve-2019-5021-alpine-docker-image-vulnerability
Why not just use Dockerfiles?
‣ If using a community image, are you actually
vetting the contents for vulnerabilities or
malware?

“Backdoored images downloaded 5 million times finally
removed from Docker Hub”
‣ Also your community image may mysteriously
vanish for reasons beyond your control (legal,
disagreements or even abandonment)

"My image with 10M+ pulls has just gone (completely
removed) from Docker Hub"
https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub
https://www.reddit.com/r/docker/comments/ail87k/my_image_with_10m_pulls_has_just_gone_completely
Agentless provisioning
‣ All major configuration management tools now have an
option to run agentless
‣ This allows you to createVM-esque images (e.g. AMIs)
for containers
‣ Same principle as baking images for ASG/MIG/VMSS
instance groups
Agentless provisioning
You will need to run your familiar provisioning
tool (ideally) agentless and masterless
‣ Puppet Bolt
‣ Chef Solo (Chef Zero)
‣ Salt Masterless
‣ Ansible 😏
https://puppet.com/products/puppet-bolt
https://docs.chef.io/chef_solo.html
https://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html
Differences between base VM and container images
‣ *nix distributions may not necessarily package the same
binaries in their baseVM image and base container image
‣ YourVM provisioning may stumble on OS detection assumptions
https://hub.docker.com/_/centos?tab=tags
https://app.vagrantup.com/centos/boxes/7/versions/1811.01
CentOS 7.6 via Docker Image

148 packages installed
CentOS 7.6 viaVagrant Box

318 packages installed
Baking container images
‣ HashiCorp Packer with 

Puppet/Chef/Ansible/SaltStack/etc.
‣ OpenShift Source-To-Image (s2i)
‣ Ansible Bender (formerly Ansible Container)
and Buildah
‣ Dockerfile ADD/COPY and RUN
‣ with —squash
‣ https://github.com/jwilder/docker-squash

https://www.packer.io/docs/builders/docker.html
https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/builds_and_image_streams.html#source-build
https://github.com/ansible-community/ansible-bender

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#add-or-copy
Modifying your build pipeline
What does the build phase of your deployment pipeline
involve?
‣ Version Control pull into active directory
‣ Upgrade via OS package manager (deb, rpm, etc.)
‣ Symlink swap of pre-built artifact on long-running hosts
‣ AutoScalingGroup/ManagedInstanceGroup rolling
updates or blue/green
Build-phase examples
Doesn’t matter if you are…
‣ Fetching build artifacts from
a repository
‣ Building build artifacts
there-and-then
‣ Installing dependencies from
a registry
… as long as it ends up in
the image
Additional environment configuration
There are usually other packages/libraries/binaries that will
have their own configuration also. How do you configure
these?
‣ Bake into image (different images per environment)
‣ Pull from remote source at container start 🤢
‣ Inject as files using container orchestrator
‣ Set environment variables
Example: boto
“Boto looks for credentials … through a list of possible
locations and stop as soon as it finds credentials”
‣ Passing credentials as parameters in
the boto.client() method
‣ Passing credentials as parameters
when creating a Session object
‣ Environment variables
‣ Shared credential file (~/.aws/
credentials)
‣ AWS config file (~/.aws/config)
‣ Assume Role provider
‣ Boto config file (/etc/boto.cfg and
~/.boto)
‣ Instance metadata service on an
Amazon EC2 instance that has an
IAM role configured.
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
Demo 1
Build and run MyBB container image
Demo 1 key points
‣ Single container with port forwarding
‣ Still uses SQLite database
‣ Configuration and database baked into the image
‣Any changes made to settings.php by MyBB itself
will be lost when the container stops
https://www.php.net/manual/en/features.commandline.webserver.php
Docker Compose
‣ “Compose is a tool for defining and running
multi-container Docker applications.”
‣ “Traditionally been focused on development
and testing workflows”
‣ Limited to a single host (without Swarm)
Demo 2
Run MyBB using Docker Compose
Demo 2 key points
‣Running MySQL as separate container
‣Volume mounting configuration directory
into MyBB container
‣Symlinking configuration files into
application include directory
‣Limited to single instance of MyBB
https://docs.docker.com/compose/compose-file/#volume-configuration-reference
https://docs.docker.com/compose/compose-file/#network-configuration-reference
Container Orchestrators
‣ Docker Swarm

Very basic
‣ CNCF Kubernetes

De facto standard
‣ Mesosphere Marathon

Flexible but complicated,
separate from Mesos itself
‣ HashiCorp Nomad

Basic but simpler and can
schedule non-container
work (plus Consul/Vault
integration)
‣ Uber Peloton

Have fun installing it
Docker Swarm
Basically Docker Compose for multiple machines
‣ “A swarm is a group of machines that are running Docker and joined
into a cluster. ”
‣ “You continue to run the Docker commands you’re used to, but now
they are executed on a cluster by a swarm manager”
https://docs.docker.com/engine/swarm/
https://docs.docker.com/get-started/part4/
Demo 3
Run MyBB using Docker Swarm
Demo 3 key points
‣Inject configuration files via the Swarm manager
‣MyBB now load balanced across multiple containers
‣ Specify CPU and memory limits
‣However, once again any changes made to
settings.php by MyBB itself will be lost when the
container(s) stops
https://docs.docker.com/engine/swarm/swarm-mode/
Demo 2 key points
Demo 2, Docker Compose Demo 3, Docker Compose… …Demo 3, Docker Compose
(re)deploy your application using
templated static configuration
Kubernetes
Just in case someone has been frozen like Captain America…
“Kubernetes (K8s) is an open-source system for automating
deployment, scaling, and management of containerized applications.
It groups containers that make up an application into logical units for
easy management and discovery.”
https://kubernetes.io
Container quickstarts
‣ Almost every container
quickstart guide will either
demonstrate how to run
containers with either no
configuration or using
environment variables
‣ So how would you inject
file-based configuration into
a container?
https://docs.docker.com/get-started/
https://docs.docker.com/compose/django/
https://cloud.google.com/kubernetes-engine/docs/quickstart
Stateful configuration
‣ Bake specific configuration files into the image
‣ Different images required per environment/tenant
‣ Security vulnerability if image is compromised
‣ Potentially dangerous?
Stateless configuration
‣ Orchestrator defines what configuration should be injected
‣ Developer and/or deployer only has access to configuration
and secrets they need
‣ Assumes that applications are configured once on startup
and only reconfigured if/when necessary
‣ Facilitates blue/green and canary deployments
ConfigMap config.php
Demo 4
Run MyBB using Kubernetes
Demo 4 key points
‣ Same as Docker Swarm… for now
‣ Replicated containers for MyBB on single node Kubernetes
‣ Replaced Docker Configs with Kubernetes ConfigMaps
https://docs.docker.com/engine/swarm/swarm-mode/
Demo 4 key points
Demo 4, KubernetesDemo 3, Docker Compose… …Demo 3, Docker Compose
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#generate-configmaps-from-files
Templating
‣ Docker Compose files, Kubernetes Manifests, Nomad Job
Specifications and Marathon application definitions are just
YAML/JSON
‣ Any templating language (Python/Go/etc.) should work
‣ Endless choices these days…
Templating
‣ Kubernetes
‣ Helm
‣ ksonnet
‣ Kapitan
‣ Draft
‣ Metaparticle
‣ Nomad
‣ Levant
‣ Generic
‣ jsonnet
‣ Render
‣ etc.
https://helm.sh
https://ksonnet.io
https://github.com/deepmind/kapitan
https://draft.sh
https://metaparticle.io
https://github.com/jrasell/levant
https://jsonnet.org/
https://github.com/VirtusLab/render
Helm
‣ Go-based templating for Kubernetes manifests
‣ Usually works on a server-client model
‣ Tiller (server) manages releases in the cluster itself
‣ Best suited for off-the-shelf software
Helm
Can be used just for rendering templates

(e.g. Spinnaker)
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#generate-configmaps-from-files
Kustomize
‣ “Template-free”
‣ Default manifest(s) withYAML
“overlays” for environment-specific
“variant” values
https://github.com/kubernetes-sigs/kustomize
https://kubernetes.io/blog/2018/05/29/introducing-kustomize-template-free-configuration-customization-for-kubernetes/
Jsonnet/Ksonnet
‣ JSON-based templating DSL
‣ “work on ksonnet will end and the GitHub repositories will
be archived”
https://jsonnet.org
https://ksonnet.io
Persisting configuration changes
How do we share runtime changes made to configuration files
between containers?
‣ Easiest way is a shared volume
‣ Relies on
‣ Application generating it’s own configuration
‣ Pre-populating the volume with configuration
https://medium.com/@saliloquy/storage-is-the-achilles-heel-of-containers-97d0341e8d87
Kubernetes Volumes
‣ Storage that is mounted in per container in a Pod
‣ Requires a PersistentVolume Claim
‣ Many different storage types supported
‣ If you already use NFS/iSCSI, keep doing so
https://kubernetes.io/docs/concepts/storage/volumes/
Portworx, StorageOS
‣ Useful for orchestrators with no native persistent storage
support (such as Nomad)
https://storageos.com
https://docs.portworx.com/install-with-other/nomad/
Rook
‣ Storage orchestration (orchestrate all the things)
‣ Abstract away storage implementation details
‣ Multiple storage providers* but plays well with Ceph
‣ Ceph offers object, block and file distributed storage
*These abstractions have caveats, see Avoid things that might catch you out section at the end
https://rook.io
https://ceph.com/ceph-storage/
Demo 5
Deploy MyBB using Helm and Kubernetes Shared
Volume
Demo 5 key points
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#generate-configmaps-from-files
Demo 4, Kubernetes Demo 5 Helm
Demo 5 key points
Demo 4, Kubernetes
https://helm.sh/docs/chart_template_guide/#values-files
Demo 5 HelmDemo 5 Helm
Persistent Volume for settings.php
Update your application on-the-
fly using dynamic configuration
ConfigMap updates
tl;dr They don’t update consistently
‣ “When a ConfigMap already being consumed in a volume is updated,
projected keys are eventually updated as well. Kubelet is checking
whether the mounted ConfigMap is fresh on every periodic sync.
‣ However, it is using its local ttl-based cache for getting the current value
of the ConfigMap.
‣ As a result, the total delay from the moment when the ConfigMap is
updated to the moment when new keys are projected to the pod can be
as long as kubelet sync period + ttl of ConfigMaps cache in kubelet.”
https://kubernetes.io/docs/tutorials/configuration/configure-redis-using-configmap/
ConfigMap updates
… or you Indiana Jones it 😅

(that feature request has been open since March 2016)
https://github.com/kubernetes/kubernetes/issues/22368
Dynamic configuration
Templating static configuration makes for easier deployments, but means
every time you want to update configuration you will need to redeploy.
But what if you want to
‣ Update configuration independent of deployments?
‣ Frequently change configuration values/feature toggles and need quick
rollbacks?
‣ Use automated short-lived credential rotations? (for databases, etc.)
Consul Template
‣ Consul Template is an agent that will (re)generate templated
files based on changes detected in Consul and/orVault
‣ Primary use-cases are Key-Value Store updates or Service
Discovery registration changes
‣ Built into Nomad Template Stanza, but easy to run otherwise
‣ Integration withVault for secrets injection
https://www.consul.io/docs/guides/consul-template.html
https://www.hashicorp.com/blog/introducing-consul-template
confd
‣ Wider compatibility with data sources beyond just Consul
‣ Primarily etcd
‣ DynamoDB
‣ Redis
‣ Zookeeper
‣ AWS SSM Parameter Store
‣ etc.
https://www.consul.io/docs/guides/consul-template.html
https://www.hashicorp.com/blog/introducing-consul-template
Demo 6
Deploy with Consul Template exec mode
entrypoint
Demo 6 key points
Demo 6, Consul Template
https://helm.sh/docs/chart_template_guide/#values-files
Demo 5 Helm
Demo 6 key points
Vault “Key/Value” KV Secrets Engine
https://helm.sh/docs/chart_template_guide/#values-files
Consul “Key/Value” KV Store
Secrets management
‣ Kubernetes Secrets
‣ Torus

Beta (free for now) cloud service
‣ Confidant by Lyft
‣ Secrethub

Cloud paid service
‣ Credstash

AWS-specific, integrates with KMS and
DynamoDB
‣ AWS SSM Parameter Store
‣ Keywhiz by Square
‣ Vault by HashiCorp
https://kubernetes.io/docs/concepts/configuration/secret/
https://www.torus.sh
https://lyft.github.io/confidant
https://secrethub.io/docs/reference/inject-command/
https://github.com/fugue/credstash
https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications-
using-parameter-store-and-iam-roles-for-tasks/
https://square.github.io/keywhiz/
https://www.vaultproject.io
Kubernetes Secrets
‣ Are not encrypted, but Base64
encoded in etcd
‣ Is your etcd instance secured?
‣ “With etcd and Kubernetes the setup is all or
nothing, there’s no authorisation used, so
be very careful” (pre 2.1)
‣ If it’s known to be less-than-secure,
what alternatives are there?
https://kubernetes.io/docs/concepts/configuration/secret/
https://www.shodan.io/search?query=etcd
https://www.jeffgeerling.com/blog/2019/decoding-kubernetes-ingress-auth-secrets
https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/
HashiCorp Vault
“Vault secures, stores, and tightly controls access to tokens,
passwords, certificates,API keys, and other secrets in modern
computing.
Vault handles leasing, key revocation, key rolling, auditing, and
provides secrets as a service through a unified API.”

https://www.vaultproject.io
Database Secrets Engine
‣ “The database secrets engine generates database credentials
dynamically based on configured roles.
‣ Since every service is accessing the database with unique credentials,
it makes auditing much easier when questionable data access is
discovered.
‣ Vault makes use of its own internal revocation system to ensure that
users become invalid within a reasonable time of the lease expiring.”
https://www.vaultproject.io/docs/secrets/databases/mysql-maria.html
Demo 7
Vault database credential rotation and Consul
External Services Monitor
Demo 7 key points
Password Rotation with short-term credential leases
https://helm.sh/docs/chart_template_guide/#values-files
Vault Database Secrets Engine
Demo 7 key points
https://www.hashicorp.com/blog/consul-and-external-services

https://github.com/hashicorp/consul-esm
Consul Nodes and Services (including those registered with External Service Monitor)
Demo 7 key points
Demo 6, Consul KV and
Vault KV Secrets Engine
https://helm.sh/docs/chart_template_guide/#values-files
Demo 7, Consul Service Discovery and
Vault Database Secrets Engine
Avoid things that might
catch you out
Container environment detection
Is your application making assumptions about the environment
it runs in?
https://medium.com/@yortuc/jvm-memory-allocation-in-docker-container-a26bbce3a3f2
https://developers.redhat.com/blog/2017/03/14/java-inside-docker/
Forking
‣ Does your application create child processes assuming a normal PID
namespace?
‣ ENTRYPOINT will be PID 1 by default
‣ If it forks a child that dies before any grandchildren exit, zombie
processes can accumulate
‣ “A process running as PID 1 inside a container is treated specially by Linux: it ignores any
signal with the default action. So, the process will not terminate on SIGINT or
SIGTERM unless it is coded to do so.”
https://docs.docker.com/engine/reference/run/#foreground
https://hackernoon.com/the-curious-case-of-pid-namespaces-1ce86b6bc900
https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
Restart splay
‣ If your process/container is restarting to reload
configuration (or otherwise)
‣ What splay time do you have (0s, 5s, 30s, 5m…)
‣ Given restart delay combined with splay, what is your 0%
readiness risk?
‣ Can you buffer incoming requests and replay if 0% ready?
(via gateway, traffic manager, etc.)
Storage inconsistencies
‣ Are file locks being
released upon container
termination correctly?
‣ Just because something says
it behaves like x, doesn’t
mean it actually will e.g.
limitations of Ceph RGW NFS
‣ Links, including symlinks, are
not supported
‣ NFS ACLs are not supported
‣ Directories may not be moved/
renamed
‣ Only full, sequential write i/o is
supported
‣ many typical i/o operations
such as editing files in place
will necessarily fail as they
perform non-sequential stores
http://docs.ceph.com/docs/mimic/radosgw/nfs/
Pick the simplest orchestrator possible
‣ Don’t use Kubernetes just because everyone else is
‣ Understand the security implications of default
networking
‣ If you can roll it out with just Swarm, Fargate or
even ECS…
Recap
‣ Lets find a legacy application
‣ Run your application in a container
‣ (re)deploy your application using templated static
configuration
‣ Update your application on-the-fly using dynamic
configuration
‣ Avoid things that might catch you out
Acknowledgements
‣ Thanks to my former team of Matthew Wright, Nikolai Orenstrakh and Stefan
Kolesnikowiczfor listening to my crazy ideas which led to this talk
‣ Thanks to the ExploreTech Toronto meetup for helping me with my first public speaking
opportunity a couple years ago
‣ Thanks to the London (UK) and Toronto (CA) tech communities for both being so
welcoming and supportive
‣ Finally, special thanks to the DevOps Days Toronto team for the opportunity to speak
today and the support they provided
That’s a wrap!
• Slides will be posted online
• https://github.com/magickatt/
ContainerisingLegacyApplicationsTalk
• https://twitter.com/magickatt
• https://www.linkedin.com/in/
andrewkirkpatrick/

More Related Content

What's hot

Build software like a bag of marbles, not a castle of LEGO®
Build software like a bag of marbles, not a castle of LEGO®Build software like a bag of marbles, not a castle of LEGO®
Build software like a bag of marbles, not a castle of LEGO®
Hannes Lowette
 
Laird Best Practices Ajax World West2008
Laird Best Practices Ajax World West2008Laird Best Practices Ajax World West2008
Laird Best Practices Ajax World West2008
rajivmordani
 

What's hot (20)

CLA Summit 2013: Connecting LabVIEW to Everything Else
CLA Summit 2013: Connecting LabVIEW to Everything ElseCLA Summit 2013: Connecting LabVIEW to Everything Else
CLA Summit 2013: Connecting LabVIEW to Everything Else
 
Docker introduction
Docker introductionDocker introduction
Docker introduction
 
Docker and Jenkins Pipeline
Docker and Jenkins PipelineDocker and Jenkins Pipeline
Docker and Jenkins Pipeline
 
Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO...
Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO...Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO...
Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO...
 
Cakephp vs. laravel
Cakephp vs. laravelCakephp vs. laravel
Cakephp vs. laravel
 
Visual Regression Testing: In search of an Ember solution
Visual Regression Testing: In search of an Ember solutionVisual Regression Testing: In search of an Ember solution
Visual Regression Testing: In search of an Ember solution
 
ColdBox Hierarchical MVC for ColdFusion/CFML
ColdBox Hierarchical MVC for ColdFusion/CFMLColdBox Hierarchical MVC for ColdFusion/CFML
ColdBox Hierarchical MVC for ColdFusion/CFML
 
Rspec and Capybara Intro Tutorial at RailsConf 2013
Rspec and Capybara Intro Tutorial at RailsConf 2013Rspec and Capybara Intro Tutorial at RailsConf 2013
Rspec and Capybara Intro Tutorial at RailsConf 2013
 
DevQA: make your testers happier with Groovy, Spock and Geb (Greach 2014)
DevQA: make your testers happier with Groovy, Spock and Geb (Greach 2014)DevQA: make your testers happier with Groovy, Spock and Geb (Greach 2014)
DevQA: make your testers happier with Groovy, Spock and Geb (Greach 2014)
 
Trunk based development
Trunk based developmentTrunk based development
Trunk based development
 
Zend con 2016 bdd with behat for beginners
Zend con 2016   bdd with behat for beginnersZend con 2016   bdd with behat for beginners
Zend con 2016 bdd with behat for beginners
 
A Personal Journey
A Personal JourneyA Personal Journey
A Personal Journey
 
Build software like a bag of marbles, not a castle of LEGO®
Build software like a bag of marbles, not a castle of LEGO®Build software like a bag of marbles, not a castle of LEGO®
Build software like a bag of marbles, not a castle of LEGO®
 
Cucumber jvm best practices v3
Cucumber jvm best practices v3Cucumber jvm best practices v3
Cucumber jvm best practices v3
 
vodQA Pune (2019) - Jenkins pipeline As code
vodQA Pune (2019) - Jenkins pipeline As codevodQA Pune (2019) - Jenkins pipeline As code
vodQA Pune (2019) - Jenkins pipeline As code
 
Laird Best Practices Ajax World West2008
Laird Best Practices Ajax World West2008Laird Best Practices Ajax World West2008
Laird Best Practices Ajax World West2008
 
Monitoring 改造計畫:流程觀點
Monitoring 改造計畫:流程觀點Monitoring 改造計畫:流程觀點
Monitoring 改造計畫:流程觀點
 
Migration tales from java ee 5 to 7
Migration tales from java ee 5 to 7Migration tales from java ee 5 to 7
Migration tales from java ee 5 to 7
 
Testing PHP with Codeception
Testing PHP with CodeceptionTesting PHP with Codeception
Testing PHP with Codeception
 
So You Just Inherited a $Legacy Application… NomadPHP July 2016
So You Just Inherited a $Legacy Application… NomadPHP July 2016So You Just Inherited a $Legacy Application… NomadPHP July 2016
So You Just Inherited a $Legacy Application… NomadPHP July 2016
 

Similar to Containerizing legacy applications

Docker 101 - all about Docker containers
Docker 101 - all about Docker containers Docker 101 - all about Docker containers
Docker 101 - all about Docker containers
Ian Lumb
 

Similar to Containerizing legacy applications (20)

Aleksandr Kutsan "Managing Dependencies in C++"
Aleksandr Kutsan "Managing Dependencies in C++"Aleksandr Kutsan "Managing Dependencies in C++"
Aleksandr Kutsan "Managing Dependencies in C++"
 
Common primitives in Docker environments
Common primitives in Docker environmentsCommon primitives in Docker environments
Common primitives in Docker environments
 
Are VMs Passé?
Are VMs Passé?Are VMs Passé?
Are VMs Passé?
 
Are VM Passé?
Are VM Passé? Are VM Passé?
Are VM Passé?
 
What is the Secure Supply Chain and the Current State of the PHP Ecosystem
What is the Secure Supply Chain and the Current State of the PHP EcosystemWhat is the Secure Supply Chain and the Current State of the PHP Ecosystem
What is the Secure Supply Chain and the Current State of the PHP Ecosystem
 
La sécurité avec Kubernetes et les conteneurs Docker (June 19th, 2019)
La sécurité avec Kubernetes et les conteneurs Docker (June 19th, 2019)La sécurité avec Kubernetes et les conteneurs Docker (June 19th, 2019)
La sécurité avec Kubernetes et les conteneurs Docker (June 19th, 2019)
 
Dockers and kubernetes
Dockers and kubernetesDockers and kubernetes
Dockers and kubernetes
 
What's New in Docker - February 2017
What's New in Docker - February 2017What's New in Docker - February 2017
What's New in Docker - February 2017
 
Docker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containersDocker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containers
 
Deploy and Scale your PHP App with AWS ElasticBeanstalk and Docker- PHPTour L...
Deploy and Scale your PHP App with AWS ElasticBeanstalk and Docker- PHPTour L...Deploy and Scale your PHP App with AWS ElasticBeanstalk and Docker- PHPTour L...
Deploy and Scale your PHP App with AWS ElasticBeanstalk and Docker- PHPTour L...
 
Getting Started with Docker - Nick Stinemates
Getting Started with Docker - Nick StinematesGetting Started with Docker - Nick Stinemates
Getting Started with Docker - Nick Stinemates
 
The DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkitThe DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkit
 
The DevOps Paradigm
The DevOps ParadigmThe DevOps Paradigm
The DevOps Paradigm
 
HPC Cloud Burst Using Docker
HPC Cloud Burst Using DockerHPC Cloud Burst Using Docker
HPC Cloud Burst Using Docker
 
Demystifying Containerization Principles for Data Scientists
Demystifying Containerization Principles for Data ScientistsDemystifying Containerization Principles for Data Scientists
Demystifying Containerization Principles for Data Scientists
 
Highly available nodejs
Highly available nodejsHighly available nodejs
Highly available nodejs
 
Tampere Docker meetup - Happy 5th Birthday Docker
Tampere Docker meetup - Happy 5th Birthday DockerTampere Docker meetup - Happy 5th Birthday Docker
Tampere Docker meetup - Happy 5th Birthday Docker
 
Docker 101 - all about Docker containers
Docker 101 - all about Docker containers Docker 101 - all about Docker containers
Docker 101 - all about Docker containers
 
Docker Application to Scientific Computing
Docker Application to Scientific ComputingDocker Application to Scientific Computing
Docker Application to Scientific Computing
 
Scaleable PHP Applications in Kubernetes
Scaleable PHP Applications in KubernetesScaleable PHP Applications in Kubernetes
Scaleable PHP Applications in Kubernetes
 

Recently uploaded

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 

Containerizing legacy applications

  • 1. Containerising legacy applications with dynamic file-based configurations and secrets DevOps Days Toronto, May 2019

 Andrew Kirkpatrick
 @magickatt
  • 2. Question 1: How many of you have what you consider to be “legacy” applications at your place of work?
  • 3. Question 2: For those who do have legacy applications, how many of you have no previous maintainers (committers) of that code still employed at your place of work?
  • 4. Question 3: For those who have legacy applications you cannot (easily) maintain, how many of you could not (easily) redeploy those applications on brand new infrastructure in a disaster recovery scenario?
  • 5. This presentation is not about ‣ How wonderful Kubernetes/Nomad/DCOS is (or isn’t…) ‣ What “DevOps” culture should look or feel like ‣ How you can proxy traffic halfway round the moon and produce pretty confusing distributed tracing graphs ‣ How you can observe even more metrics that you’ll probably never look at
  • 6. This is more about the harsh reality that there is always legacy code… http://www.commitstrip.com/en/2017/12/12/the-code-is-always-better-on-the-other-side/ http://www.commitstrip.com/en/2016/02/15/our-companys-greatest-project/
  • 7. So why would anyone want to migrate their application(s) into containers without rewriting code?
  • 8. ‣ Too busy working on newer (micro)services ‣ Avoid touching the nightmare codebase that the long-gone Developer(s) that worked on it left behind ‣ Business wants to fast-track the cost-saving benefits of containerisation
  • 9. https://insights.stackoverflow.com/survey Would you be able to even if you could? ‣Availability of hireable Developers? ‣Prohibitive cost of remaining “experts” ‣Un-decipherable yet crucial business logic ‣Maintenance/ documentation of older libraries/packages/ frameworks/tooling 0 17.5 35 52.5 70 2019 2018 2017 2016 2015 2014 2013 Javascript Java PHP Ruby Node C# Python C
  • 10. Assumptions ‣ You can’t change the code in your application(s) ‣ Familiarity ‣ Time ‣ Permissions/licensing ‣ You don’t want to change the code in your application(s) ‣ Bugs ‣ Liability ‣ You want to take the path of least resistance
  • 11. Steps ‣ Lets find a legacy application ‣ Run your application in a container ‣ (re)deploy your application using templated static configuration ‣ Update your application on-the-fly using dynamic configuration ‣ Avoid things that might catch you out
  • 12. Lets find a legacy application…
  • 13. Old(er) application candidate ‣ Must have been designed pre-containerisation ‣ Must rely on filesystem-based configuration, which it may update itself ‣ Likely makes assumptions about running on a single server (thus a single filesystem, etc.) ‣ Has little/no examples of a quickstart using a container orchestrator
  • 14. Lets pick an old(er) application What about WordPress? Primarily relies on wp-config.php https://codex.wordpress.org/Editing_wp-config.php
  • 15. Lets pick an old(er) application Alright then, perhaps something less obvious but just as ubiquitous? phpBB? Primarily relies on config.php
 https://www.phpbb.com/support/docs/en/3.2/kb/article/rebuilding-your-configphp-file/
  • 16. Lets pick an old(er) application Need to go a bit more obscure… how about another forum, MyBB? Primarily relies on config.php and settings.php
  • 17. Found the right era of application ‣ “Once you’ve uploaded your files you will need to set the permissions on certain files and directories. ‣ Before granting certain files and directories chmod 777, you may want to try chmod 755 or chmod 775.” https://docs.mybb.com/1.8/install/#file-permissions http://www.commitstrip.com/en/2016/06/29/chmod-what/
  • 18. Older applications live on Codebases like these are still being actively used and maintained despite their longevity ‣ MyBB was first released on 9th December 2005 (v1.0.0) ‣ Last released on 27th February 2019 (v1.8.20) Still plenty of issues being reported and resolved 7 days ago, 13 days ago, 15 days ago…
  • 19.
  • 20. MyBB ‣ Majority of the file-based configuration for MyBB resides in 2 files ‣ inc/config.php ‣ inc/settings.php ‣ Note that these are not JSON/YAML or even INI/XML files, but language-specific configuration files
  • 21. config.php Environment configuration such as database connectivity, logging, IP blacklisting, etc. Essential to the bootstrapping of the application
  • 22. settings.php Application-specific settings pre-database configuration load, such as formatting, routing, basic settings and some string replacements. Example of configuration that is traditionally updated by the application itself
 /********************************* 
 DO NOT EDIT THIS FILE, PLEASE USE 
 THE SETTINGS EDITOR 
 *********************************/
  • 23. Demo 0 Run MyBB using the PHP built-in web server
  • 24. Demo 0 key points ‣ Only works on my laptop with a filesystem-based database ‣ Local development server
 php -S localhost:9000 -t application ‣ SQLite database for simplicity https://www.php.net/manual/en/features.commandline.webserver.php
  • 25. Run your application in a container
  • 26. Run your application in a container First, how do you run your application in a container that resembles your existing environment? ‣ Bake a base container image re-using existing provisioning ‣ Modifying your build pipeline to create container images ‣ Additional environment configuration considerations (external libraries, vendor integrations, etc.)
  • 27. What is your “base”? How bespoke is your application runtime environment? ‣ How are you currently provisioning yourVM/bare metal fleet?
 (hopefully using infrastructure automation? 🤞) ‣ Do you compile your own packages/libraries/ drivers and/or maintain your own Debian/RPM/etc. repository? ‣ How many other configuration changes are you making to hosts?
  • 28. Container tutorial assumptions Problem with most containerisation tutorials is that they assume that your application will run with near-vanilla configuration in a vanilla environment
 https://docs.docker.com/samples/library/php/#example
  • 29. Container tutorial assumptions … and to keep the images small, they strip a lot of “extras”
 https://docs.docker.com/samples/library/php/
 https://www.php.net/manual/en/book.mysqli.php
 https://www.php.net/manual/en/book.memcached.php MySQL Improved Extension Memcached Extension
  • 30. Container tutorial assumptions ‣ You may end up with a rather messy Dockerfile ‣ Also AUFS has a 42 layer limit (or used to) ‣ Easier to use multi-stage builds, or “bakes” https://docs.docker.com/samples/library/php/
 https://www.php.net/manual/en/book.mysqli.php
 https://www.php.net/manual/en/book.memcached.php
  • 31. Why not just use Dockerfiles? Developers will often ask…
 “why not just use a Dockerfile to install the bare minimum?” or “can’t we just use a community image?” ‣ “the bare minimum” could be hundreds of packages or binaries, custom configured or compiled ‣ If using a different distribution than the rest of your fleet, do you really know if it’s secure?
  • 32. Why not just use Dockerfiles? CVE-2019-5021 “Alpine Linux Docker Images Shipped for 3 Years with Root Accounts Unlocked Alpine Linux Docker images available via the Docker Hub contained a critical flaw allowing attackers to authenticate on systems using the root user and no password.” “This CVE does not impact Alpine distros that are not delivered as Docker images.” https://threatpost.com/alpine-linux-docker-images-unlocked/144542/
 https://blog.aquasec.com/cve-2019-5021-alpine-docker-image-vulnerability
  • 33. Why not just use Dockerfiles? ‣ If using a community image, are you actually vetting the contents for vulnerabilities or malware?
 “Backdoored images downloaded 5 million times finally removed from Docker Hub” ‣ Also your community image may mysteriously vanish for reasons beyond your control (legal, disagreements or even abandonment)
 "My image with 10M+ pulls has just gone (completely removed) from Docker Hub" https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub https://www.reddit.com/r/docker/comments/ail87k/my_image_with_10m_pulls_has_just_gone_completely
  • 34. Agentless provisioning ‣ All major configuration management tools now have an option to run agentless ‣ This allows you to createVM-esque images (e.g. AMIs) for containers ‣ Same principle as baking images for ASG/MIG/VMSS instance groups
  • 35. Agentless provisioning You will need to run your familiar provisioning tool (ideally) agentless and masterless ‣ Puppet Bolt ‣ Chef Solo (Chef Zero) ‣ Salt Masterless ‣ Ansible 😏 https://puppet.com/products/puppet-bolt https://docs.chef.io/chef_solo.html https://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html
  • 36. Differences between base VM and container images ‣ *nix distributions may not necessarily package the same binaries in their baseVM image and base container image ‣ YourVM provisioning may stumble on OS detection assumptions https://hub.docker.com/_/centos?tab=tags https://app.vagrantup.com/centos/boxes/7/versions/1811.01 CentOS 7.6 via Docker Image
 148 packages installed CentOS 7.6 viaVagrant Box
 318 packages installed
  • 37. Baking container images ‣ HashiCorp Packer with 
 Puppet/Chef/Ansible/SaltStack/etc. ‣ OpenShift Source-To-Image (s2i) ‣ Ansible Bender (formerly Ansible Container) and Buildah ‣ Dockerfile ADD/COPY and RUN ‣ with —squash ‣ https://github.com/jwilder/docker-squash
 https://www.packer.io/docs/builders/docker.html https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/builds_and_image_streams.html#source-build https://github.com/ansible-community/ansible-bender
 https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#add-or-copy
  • 38. Modifying your build pipeline What does the build phase of your deployment pipeline involve? ‣ Version Control pull into active directory ‣ Upgrade via OS package manager (deb, rpm, etc.) ‣ Symlink swap of pre-built artifact on long-running hosts ‣ AutoScalingGroup/ManagedInstanceGroup rolling updates or blue/green
  • 39. Build-phase examples Doesn’t matter if you are… ‣ Fetching build artifacts from a repository ‣ Building build artifacts there-and-then ‣ Installing dependencies from a registry … as long as it ends up in the image
  • 40. Additional environment configuration There are usually other packages/libraries/binaries that will have their own configuration also. How do you configure these? ‣ Bake into image (different images per environment) ‣ Pull from remote source at container start 🤢 ‣ Inject as files using container orchestrator ‣ Set environment variables
  • 41. Example: boto “Boto looks for credentials … through a list of possible locations and stop as soon as it finds credentials” ‣ Passing credentials as parameters in the boto.client() method ‣ Passing credentials as parameters when creating a Session object ‣ Environment variables ‣ Shared credential file (~/.aws/ credentials) ‣ AWS config file (~/.aws/config) ‣ Assume Role provider ‣ Boto config file (/etc/boto.cfg and ~/.boto) ‣ Instance metadata service on an Amazon EC2 instance that has an IAM role configured. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
  • 42. Demo 1 Build and run MyBB container image
  • 43. Demo 1 key points ‣ Single container with port forwarding ‣ Still uses SQLite database ‣ Configuration and database baked into the image ‣Any changes made to settings.php by MyBB itself will be lost when the container stops https://www.php.net/manual/en/features.commandline.webserver.php
  • 44. Docker Compose ‣ “Compose is a tool for defining and running multi-container Docker applications.” ‣ “Traditionally been focused on development and testing workflows” ‣ Limited to a single host (without Swarm)
  • 45. Demo 2 Run MyBB using Docker Compose
  • 46. Demo 2 key points ‣Running MySQL as separate container ‣Volume mounting configuration directory into MyBB container ‣Symlinking configuration files into application include directory ‣Limited to single instance of MyBB https://docs.docker.com/compose/compose-file/#volume-configuration-reference https://docs.docker.com/compose/compose-file/#network-configuration-reference
  • 47. Container Orchestrators ‣ Docker Swarm
 Very basic ‣ CNCF Kubernetes
 De facto standard ‣ Mesosphere Marathon
 Flexible but complicated, separate from Mesos itself ‣ HashiCorp Nomad
 Basic but simpler and can schedule non-container work (plus Consul/Vault integration) ‣ Uber Peloton
 Have fun installing it
  • 48. Docker Swarm Basically Docker Compose for multiple machines ‣ “A swarm is a group of machines that are running Docker and joined into a cluster. ” ‣ “You continue to run the Docker commands you’re used to, but now they are executed on a cluster by a swarm manager” https://docs.docker.com/engine/swarm/ https://docs.docker.com/get-started/part4/
  • 49. Demo 3 Run MyBB using Docker Swarm
  • 50. Demo 3 key points ‣Inject configuration files via the Swarm manager ‣MyBB now load balanced across multiple containers ‣ Specify CPU and memory limits ‣However, once again any changes made to settings.php by MyBB itself will be lost when the container(s) stops https://docs.docker.com/engine/swarm/swarm-mode/
  • 51. Demo 2 key points Demo 2, Docker Compose Demo 3, Docker Compose… …Demo 3, Docker Compose
  • 52. (re)deploy your application using templated static configuration
  • 53. Kubernetes Just in case someone has been frozen like Captain America… “Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery.” https://kubernetes.io
  • 54. Container quickstarts ‣ Almost every container quickstart guide will either demonstrate how to run containers with either no configuration or using environment variables ‣ So how would you inject file-based configuration into a container? https://docs.docker.com/get-started/ https://docs.docker.com/compose/django/ https://cloud.google.com/kubernetes-engine/docs/quickstart
  • 55. Stateful configuration ‣ Bake specific configuration files into the image ‣ Different images required per environment/tenant ‣ Security vulnerability if image is compromised ‣ Potentially dangerous?
  • 56. Stateless configuration ‣ Orchestrator defines what configuration should be injected ‣ Developer and/or deployer only has access to configuration and secrets they need ‣ Assumes that applications are configured once on startup and only reconfigured if/when necessary ‣ Facilitates blue/green and canary deployments
  • 58. Demo 4 Run MyBB using Kubernetes
  • 59. Demo 4 key points ‣ Same as Docker Swarm… for now ‣ Replicated containers for MyBB on single node Kubernetes ‣ Replaced Docker Configs with Kubernetes ConfigMaps https://docs.docker.com/engine/swarm/swarm-mode/
  • 60. Demo 4 key points Demo 4, KubernetesDemo 3, Docker Compose… …Demo 3, Docker Compose https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#generate-configmaps-from-files
  • 61. Templating ‣ Docker Compose files, Kubernetes Manifests, Nomad Job Specifications and Marathon application definitions are just YAML/JSON ‣ Any templating language (Python/Go/etc.) should work ‣ Endless choices these days…
  • 62. Templating ‣ Kubernetes ‣ Helm ‣ ksonnet ‣ Kapitan ‣ Draft ‣ Metaparticle ‣ Nomad ‣ Levant ‣ Generic ‣ jsonnet ‣ Render ‣ etc. https://helm.sh https://ksonnet.io https://github.com/deepmind/kapitan https://draft.sh https://metaparticle.io https://github.com/jrasell/levant https://jsonnet.org/ https://github.com/VirtusLab/render
  • 63. Helm ‣ Go-based templating for Kubernetes manifests ‣ Usually works on a server-client model ‣ Tiller (server) manages releases in the cluster itself ‣ Best suited for off-the-shelf software
  • 64. Helm Can be used just for rendering templates
 (e.g. Spinnaker) https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#generate-configmaps-from-files
  • 65. Kustomize ‣ “Template-free” ‣ Default manifest(s) withYAML “overlays” for environment-specific “variant” values https://github.com/kubernetes-sigs/kustomize https://kubernetes.io/blog/2018/05/29/introducing-kustomize-template-free-configuration-customization-for-kubernetes/
  • 66. Jsonnet/Ksonnet ‣ JSON-based templating DSL ‣ “work on ksonnet will end and the GitHub repositories will be archived” https://jsonnet.org https://ksonnet.io
  • 67. Persisting configuration changes How do we share runtime changes made to configuration files between containers? ‣ Easiest way is a shared volume ‣ Relies on ‣ Application generating it’s own configuration ‣ Pre-populating the volume with configuration https://medium.com/@saliloquy/storage-is-the-achilles-heel-of-containers-97d0341e8d87
  • 68. Kubernetes Volumes ‣ Storage that is mounted in per container in a Pod ‣ Requires a PersistentVolume Claim ‣ Many different storage types supported ‣ If you already use NFS/iSCSI, keep doing so https://kubernetes.io/docs/concepts/storage/volumes/
  • 69. Portworx, StorageOS ‣ Useful for orchestrators with no native persistent storage support (such as Nomad) https://storageos.com https://docs.portworx.com/install-with-other/nomad/
  • 70. Rook ‣ Storage orchestration (orchestrate all the things) ‣ Abstract away storage implementation details ‣ Multiple storage providers* but plays well with Ceph ‣ Ceph offers object, block and file distributed storage *These abstractions have caveats, see Avoid things that might catch you out section at the end https://rook.io https://ceph.com/ceph-storage/
  • 71. Demo 5 Deploy MyBB using Helm and Kubernetes Shared Volume
  • 72. Demo 5 key points https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#generate-configmaps-from-files Demo 4, Kubernetes Demo 5 Helm
  • 73. Demo 5 key points Demo 4, Kubernetes https://helm.sh/docs/chart_template_guide/#values-files Demo 5 HelmDemo 5 Helm
  • 74. Persistent Volume for settings.php
  • 75. Update your application on-the- fly using dynamic configuration
  • 76. ConfigMap updates tl;dr They don’t update consistently ‣ “When a ConfigMap already being consumed in a volume is updated, projected keys are eventually updated as well. Kubelet is checking whether the mounted ConfigMap is fresh on every periodic sync. ‣ However, it is using its local ttl-based cache for getting the current value of the ConfigMap. ‣ As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period + ttl of ConfigMaps cache in kubelet.” https://kubernetes.io/docs/tutorials/configuration/configure-redis-using-configmap/
  • 77. ConfigMap updates … or you Indiana Jones it 😅
 (that feature request has been open since March 2016) https://github.com/kubernetes/kubernetes/issues/22368
  • 78. Dynamic configuration Templating static configuration makes for easier deployments, but means every time you want to update configuration you will need to redeploy. But what if you want to ‣ Update configuration independent of deployments? ‣ Frequently change configuration values/feature toggles and need quick rollbacks? ‣ Use automated short-lived credential rotations? (for databases, etc.)
  • 79. Consul Template ‣ Consul Template is an agent that will (re)generate templated files based on changes detected in Consul and/orVault ‣ Primary use-cases are Key-Value Store updates or Service Discovery registration changes ‣ Built into Nomad Template Stanza, but easy to run otherwise ‣ Integration withVault for secrets injection https://www.consul.io/docs/guides/consul-template.html https://www.hashicorp.com/blog/introducing-consul-template
  • 80. confd ‣ Wider compatibility with data sources beyond just Consul ‣ Primarily etcd ‣ DynamoDB ‣ Redis ‣ Zookeeper ‣ AWS SSM Parameter Store ‣ etc. https://www.consul.io/docs/guides/consul-template.html https://www.hashicorp.com/blog/introducing-consul-template
  • 81. Demo 6 Deploy with Consul Template exec mode entrypoint
  • 82. Demo 6 key points Demo 6, Consul Template https://helm.sh/docs/chart_template_guide/#values-files Demo 5 Helm
  • 83. Demo 6 key points Vault “Key/Value” KV Secrets Engine https://helm.sh/docs/chart_template_guide/#values-files Consul “Key/Value” KV Store
  • 84. Secrets management ‣ Kubernetes Secrets ‣ Torus
 Beta (free for now) cloud service ‣ Confidant by Lyft ‣ Secrethub
 Cloud paid service ‣ Credstash
 AWS-specific, integrates with KMS and DynamoDB ‣ AWS SSM Parameter Store ‣ Keywhiz by Square ‣ Vault by HashiCorp https://kubernetes.io/docs/concepts/configuration/secret/ https://www.torus.sh https://lyft.github.io/confidant https://secrethub.io/docs/reference/inject-command/ https://github.com/fugue/credstash https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications- using-parameter-store-and-iam-roles-for-tasks/ https://square.github.io/keywhiz/ https://www.vaultproject.io
  • 85. Kubernetes Secrets ‣ Are not encrypted, but Base64 encoded in etcd ‣ Is your etcd instance secured? ‣ “With etcd and Kubernetes the setup is all or nothing, there’s no authorisation used, so be very careful” (pre 2.1) ‣ If it’s known to be less-than-secure, what alternatives are there? https://kubernetes.io/docs/concepts/configuration/secret/ https://www.shodan.io/search?query=etcd https://www.jeffgeerling.com/blog/2019/decoding-kubernetes-ingress-auth-secrets https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/
  • 86. HashiCorp Vault “Vault secures, stores, and tightly controls access to tokens, passwords, certificates,API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, auditing, and provides secrets as a service through a unified API.”
 https://www.vaultproject.io
  • 87. Database Secrets Engine ‣ “The database secrets engine generates database credentials dynamically based on configured roles. ‣ Since every service is accessing the database with unique credentials, it makes auditing much easier when questionable data access is discovered. ‣ Vault makes use of its own internal revocation system to ensure that users become invalid within a reasonable time of the lease expiring.” https://www.vaultproject.io/docs/secrets/databases/mysql-maria.html
  • 88. Demo 7 Vault database credential rotation and Consul External Services Monitor
  • 89. Demo 7 key points Password Rotation with short-term credential leases https://helm.sh/docs/chart_template_guide/#values-files Vault Database Secrets Engine
  • 90. Demo 7 key points https://www.hashicorp.com/blog/consul-and-external-services
 https://github.com/hashicorp/consul-esm Consul Nodes and Services (including those registered with External Service Monitor)
  • 91. Demo 7 key points Demo 6, Consul KV and Vault KV Secrets Engine https://helm.sh/docs/chart_template_guide/#values-files Demo 7, Consul Service Discovery and Vault Database Secrets Engine
  • 92. Avoid things that might catch you out
  • 93. Container environment detection Is your application making assumptions about the environment it runs in? https://medium.com/@yortuc/jvm-memory-allocation-in-docker-container-a26bbce3a3f2 https://developers.redhat.com/blog/2017/03/14/java-inside-docker/
  • 94. Forking ‣ Does your application create child processes assuming a normal PID namespace? ‣ ENTRYPOINT will be PID 1 by default ‣ If it forks a child that dies before any grandchildren exit, zombie processes can accumulate ‣ “A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. So, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.” https://docs.docker.com/engine/reference/run/#foreground https://hackernoon.com/the-curious-case-of-pid-namespaces-1ce86b6bc900 https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
  • 95. Restart splay ‣ If your process/container is restarting to reload configuration (or otherwise) ‣ What splay time do you have (0s, 5s, 30s, 5m…) ‣ Given restart delay combined with splay, what is your 0% readiness risk? ‣ Can you buffer incoming requests and replay if 0% ready? (via gateway, traffic manager, etc.)
  • 96. Storage inconsistencies ‣ Are file locks being released upon container termination correctly? ‣ Just because something says it behaves like x, doesn’t mean it actually will e.g. limitations of Ceph RGW NFS ‣ Links, including symlinks, are not supported ‣ NFS ACLs are not supported ‣ Directories may not be moved/ renamed ‣ Only full, sequential write i/o is supported ‣ many typical i/o operations such as editing files in place will necessarily fail as they perform non-sequential stores http://docs.ceph.com/docs/mimic/radosgw/nfs/
  • 97. Pick the simplest orchestrator possible ‣ Don’t use Kubernetes just because everyone else is ‣ Understand the security implications of default networking ‣ If you can roll it out with just Swarm, Fargate or even ECS…
  • 98. Recap ‣ Lets find a legacy application ‣ Run your application in a container ‣ (re)deploy your application using templated static configuration ‣ Update your application on-the-fly using dynamic configuration ‣ Avoid things that might catch you out
  • 99. Acknowledgements ‣ Thanks to my former team of Matthew Wright, Nikolai Orenstrakh and Stefan Kolesnikowiczfor listening to my crazy ideas which led to this talk ‣ Thanks to the ExploreTech Toronto meetup for helping me with my first public speaking opportunity a couple years ago ‣ Thanks to the London (UK) and Toronto (CA) tech communities for both being so welcoming and supportive ‣ Finally, special thanks to the DevOps Days Toronto team for the opportunity to speak today and the support they provided
  • 100. That’s a wrap! • Slides will be posted online • https://github.com/magickatt/ ContainerisingLegacyApplicationsTalk • https://twitter.com/magickatt • https://www.linkedin.com/in/ andrewkirkpatrick/