Today’s best practices for building and deploying Linux containers do not always align with the messy realities of containers used in production. In the wild, the vast majority of containers include entire OS filesystems, applications with complex and often poorly understood dependencies, and a plethora of libraries, with all of their attendant security vulnerability and maintenance concerns. Container orchestration systems like Kubernetes encourage us to treat containers as immutable black boxes. From these building blocks, developers can assemble, integrate, and scale distributed applications with relative ease. Containers as black boxes, is that a brilliant idea or, considering what might be lurking inside, a terrible idea? The answer is both, and after this talk, you will understand why.

1. More Terrible Ideas
for Containers?
The Ideal and the
Real Linux Container
Andy Henroid <andy.henroid@puppet.com>

2. Who am I?

3. This Talk
Linux Containers
Big Promises
Broken Dreams?
Band-Aids & Better Solutions

4. Linux & Open Source Software:
Astounding range & variety of SW
Dependencies accumulate quickly
Yes “DLL Hell” is a Linux problem
See also: RPM Hell, Java JAR Hell, Ruby
Gem Hell, …
And many disparate solutions to these
4
Containers: Escape from Dependency Hell

5. Orchestrators like Kubernetes add:
ü Dynamic Scalability
ü Resource Efficiency
ü High Availability
ü Uniform Service Discovery
ü Portability
And more...
Treating each container/pod as a Black Box,
a homogenous unit to schedule in the cluster
5
Containers + Orchestrator = Massive Scalability

6. Lightweight
Single-purpose
Immutable
Great starting point for
a micro-service
6
The Ideal Linux Container
4.2MB
1
11
Image
Size
Image
Layers
Packages
Alpine 3.5.2 : A minimal Docker image based
on Alpine Linux with a complete package index

7. Heavyweight
General-purpose
Mutable
Flexible, reusable
But fast approaching
VM size
7
The Real Linux Container
Ideal to… far from ideal
4M 11 0.4K
129M 104
8K
190M
140
11K
240M
171
13K
716M
191
25K
Image Size Packages Files
Alpine Debian Nginx (Debian) Fedora GoLang (Debian)

8. Container tooling makes it
trivial to “ADD”
But there is no “REMOVE”
Images grow quickly
And never smaller
8
Image Size & Layers
231M
171
436M
172
Image Size Packages
Fedora Fedora (after "yum update")

9. 9
Riffing on a Cool Idea
~330K Dockerfiles
on GitHub

10. Open-source https://github.com/anchore/anchore
Image size & layers
Image family tree
Linux distro
Packages & files
CVEs
And more…
10
Variations on a Theme: Opening the Black Box
Ideal to.…........... alarming
449 0
8267
707
11191
1435
12792
720
25497
10720
Files Unmanaged Files
Alpine Debian Nginx (Debian) Fedora GoLang (Debian)

11. We analyzed :latest image from
>5000 top repos
136 Official + many more
Wide variety:
Base OS images, services,
middleware, languages, …
600M+ to 10K pulls
11
Docker Hub Survey
† Full data set at https://github.com/ahenroid/docker-hub-survey
Fun Fact: Top 2% public Hub
repos account for >99% of all
Docker pulls
23.3%
2.0% 1.5%
3.1% 1.6%
31.6%
29.5%
6.6%
0.8%
alpine
buildroot
busybox
Misc.
Other
debian
ubuntu
centos
fedora
Lightweight =
31.5% Images
Heavyweight =
68.5% Images
Base Container Image

12. Median stats: Expansive gulf between lightweights & heavyweights
12
Docker Hub Survey
64M
20
577M
245
Image Size Packages
Lightweight Heavyweight
4K
0
23K
5K
Files Unmanaged Files
Lightweight Heavyweight
† Full data set at https://github.com/ahenroid/docker-hub-survey

13. Massive Scalability? Maybe.
Escape from Dependency Hell?
Deferred, but not actually solved.
Ahead for operators: Inventory and
maintenance nightmare
Always Vulnerable
Always Updating
Always Rebuilding
Even the best containers fall short
13
Container Big Promises

14. Labels, embedding Dockerfiles,
and similar band-aids
Pro: Bound to container image
Cons:
Not automatic
Not standardized
No rich data format
Quickly out-of-date
14
Advice (Part I): Know What’s Inside

15. Container Scanning
Static image scanning
Everyone is doing it…
Open source, hosted
services, integrated with
container registries
Live container scanning
Fewer options today
15
Advice (Part I): Know What’s Inside

16. Software development principles
encouraging Fast Delivery
Open Source: Every library you will
ever need is here, ready to integrate
Decomposing traditional monolithic
apps is hard, fraught work
16
Quick Interlude: How Did We Get Here?

17. Start small and build carefully, thoughtfully
FROM scratch, or more realistically, Alpine
Separate & design containers by purpose
Front-end web service, Jenkins build slave, ...
No “Golden Container Image”
Know your dependencies & question over time
Be willing to redesign/refactor as your
containers evolve
Automated tools to identify dependency creep
17
Advice (Part II): Good Engineering Practices
Engineering?!

18. Other Talks
Continuous Delivery to Azure with Docker
Using Redis with other DBs
Creating custom Postgresql packages in Alpine Linux
for use with Docker/Containers/VMs
Libral: towards a system management API for Linux
Spice up your dev environment with containers
Container Images @ FB with Btrfs

19. Questions?

20. The shortest path
to better software.