What’s with all of these container image vulnerabilities? I’m a developer, not a security analyst! Whether you’re a solo dev or a large team embracing DevSecOps, join me to learn practices I’ve seen successful teams using to build safer container images & avoid the mistakes they made along the way.
If you’ve even run a vulnerability scan on a container you’ve probably seen it: the dreaded list with 100s, maybe even 1000s of issues on it. Containers have made life simpler in so many ways, but security sometimes doesn’t feel like one of them. So what can we do about it?
In this talk, I’ll share what I’ve learned working with users and companies and the best practices I’ve picked up along the way to builds safer container images. I’ll also share what not to do, because there are many rabbit holes you can go down that end up wasting time and energy.
I’ll share the processes and patterns that you can use whether you’re working on an individual project, or you’re part of a bigger team embracing DevSecOps.
2. WHAT ARE WE
GOING TO COVER?
• I’ve got a container with a bunch of
vulnerabilities. What do I get started?
• We have a bunch of teams using
containers, is there a good process we
can all use?
• Interesting ideas in the container world
(including some I’d avoid)
8. FIXING CONTAINERS IS ALSO NOT LIKE
FIXING VMS
Did I add
something
with an
issue…
…or did I
inherit this
problem?
The answer changes how we think about a “fix”
10. PICKING A BASE IMAGE ISN’T AS SIMPLE
AS IT SEEMS
ruby
850 MB
414 dependencies
257 vulnerabilities
35 high severity
ruby:3-slim
157 MB
107 dependencies
60 vulnerabilities
10 high severity
There are ~700 tags listed
in Docker Hub’s official ruby
repo…
...which one is “best”?
11. BASE IMAGE BEST PRACTICES
Generic is not what you
want!
• What framework version are you
coding against? Do these
containers match? Will they
match tomorrow?
• Generally: good for playing
around, but don’t use for ”real”
work
But -slim isn’t automatically
the right choice
• Vulns go away! 🥳
• BUT you “get to” manage all the
build dependencies 😒
docker pull [ruby | python | ubuntu...]
RUN apt-get update &&
apt-get install -y build-essential
patch ruby-dev zlib1g-dev liblzma-dev
libpq-dev libsqlite3-dev
probably
^
12. MULTI-STAGE TO THE RESCUE!!!
FROM python:3.8-buster as builder
.
.
.
FROM python:3.8-buster-slim
COPY –from=builder <app stuff> .
13. PICKING A BASE IMAGE ISN’T AS SIMPLE AS
IT SEEMS
ruby
850 MB
414 dependencies
257 vulnerabilities
35 high severity
ruby:3-slim
157 MB
107 dependencies
60 vulnerabilities
10 high severity
ruby:alpine
60 MB
36 dependencies
1 vulnerabilities
0 high severity
14. SO WHY NOT ALPINE
FOR ALL THE THINGS?!?
• Uses musl instead of gnu/libc
• Uses different packages, different package
manager
• They handle security a little differently
https://pythonspeed.com/articles/alpine-docker-python/
15. GENERAL RECOMMENDATIONS
General “make life easier” things:
• A little bit of Sysadmin knowledge is unavoidable - pick an OS
distribution and use it for everything
• Pin to versioned images (at least Major, probably minor)
Getting rid of vulnerabilities
• Learn & love multi-stage builds!
• Let Docker / Red Hat / VMware (Bitnami)… do the heavy lifting!
• Rebuild often (clear your cache or use --no-cache)
• Move your pins every once in a while 📌
Zero vulnerabilities in a container is almost impossible
16. WE HAVE A BUNCH OF TEAMS USING
CONTAINERS, IS THERE A GOOD PROCESS
WE CAN ALL USE?
17. Using base images
FROM ubuntu:latest
Standing on the shoulders of software giants
It’s common with container images to start
building on top of an existing base image that
already has software you want.
This might be an operating system like ubuntu,
alpine or debian or it could be a language like
python, ruby, node or really anything else.
OK, technically often a parent image but hey.
Libraries and underlying software
provided by someone else
Your software
18. Distinct responsibilities
Hardening, common configuration
Your organization might have some common hardening or
configuration changes or maybe metadata it wants to
apply to all images in use by other teams. This is often
intended to be common for all images used. Maybe you
have myorg/base
Libraries and underlying software
provided by someone else
Hardening, common configuration
Common software
Your application
19. Distinct responsibilities
Common software
Some organizations provide a layer of common software
or middleware. This might be language or framework
specific, say a separate image for Java (myorg/java) and
another for Python (myorg/python).
Libraries and underlying software
provided by someone else
Hardening, common configuration
Common software
Your application
20. Distinct responsibilities
Your application
Finally the specifics of your application, whether in source
or binary form. And metadata specific to the application.
Libraries and underlying software
provided by someone else
Hardening, common configuration
Common software
Your application
21. Can you fix vulnerabilities
once?
Base
image
Scale vulnerability management
When considering container
vulnerabilities, you want to be able
to reason about vulnerabilities in
images you’re running, but also
understand the overlap and source
of those vulnerabilities.
Can you address a vulnerability once,
and have it resolved everywhere?
23. Organizing into teams
Libraries and underlying
software provided by someone
else
Hardening, common
configuration
Common software
Your application
Libraries and underlying
software provided by someone
else
Hardening, common
configuration
Common software
Your application
Libraries and underlying
software provided by someone
else
Hardening, common
configuration
Common software
Your application
One team to rule them all
This could be the case when
teams are completely
independent, or when you
have one central image team.
A base image team
A team which provides a
standard set of approved
base images for application
teams to consume.
Separate base/security teams
Larger organizations might have
teams with more distinct
responsibilities, potentially with
even more layers.
24. Pros and cons
One team to rule them all
PROS
Simple to understand
responsibilities.
CONS
Potential for chaos if every
team can do their own thing.
One central team for ALL
images likely to become a
bottleneck.
A base image team
PROS
Able to build strong domain
expertise in the center,
ideally fix/triage issues once.
CONS
Needs some level of
governance in order to
ensure applications teams
benefit from central
expertise.
Separate base/security teams
PROS
Same as having a base image
team, with the added advantage
of deeper specialisms.
CONS
Coordination between
additional teams can slow down
the process.
25. Start with your own base image
$ cat base/Dockerfile
FROM python:3.6.0-slim
RUN apt-get update && apt-get install -y <all our critical hardening pkgs>
$ docker build -t myorg/base -f base/Dockerfile
26. Establish a baseline
What can downstream consumers ignore?
$ <container tool> --test myorg/base
...
Introduced by your base image (python:3.6.0-slim)
Fixed in: 5.28.1-6+deb10u1
✗ High severity vulnerability found in gnutls28/libgnutls30
Description: Out-of-bounds Write
Info: https://snyk.io/vuln/SNYK-DEBIAN10-GNUTLS28-609778
Introduced through: gnutls28/libgnutls30@3.6.7-4+deb10u4, apt@1.8.2.1
From: gnutls28/libgnutls30@3.6.7-4+deb10u4
From: apt@1.8.2.1 > gnutls28/libgnutls30@3.6.7-4+deb10u4
Introduced by your base image (python:3.6.0-slim)
Tested 111 dependencies for known issues, found 178 issues.
27. Watch out for new vulnerabilities
Generally, “fresh” images on Hub, RHT, etc have no FIXABLE vulns
• Your baseline establishes the things
you can ignore
• But things get fixed…
• …and things get broken…
• Rebuild & set a new baseline often!
28. A middleware image
$ cat middleware/Dockerfile
FROM myorg/base
RUN apt-get update && apt-get install -y
gunicorn
sqlite3
&& rm -rf /var/lib/apt/lists/
$ docker build -t myorg/middleware -f middleware/Dockerfile
29. The sum of all vulnerabilities
$ <container tool> test myorg/middleware
...
Tested 127 dependencies for known issues, found 180 issues.
Vulnerabilities from the base image and the new instructions
But a different team is responsible for some of these, so let’s reason about those separately.
30. Where did the issue come from?
$ <container tool> test myorg/middleware
+------------------------------------------------------------------------------------------------------------------------------------------------------+
| Found 180 unique vulnerabilities for myorg/middleware |
+----------------------------------+---------------+------------------+------------------------------------+-------------------+-----------------------+
| Package | Severity | ID | Issue | Installed | Fixed in |
+----------------------------------+---------------+------------------+------------------------------------+-------------------+-----------------------+
| ncurses/libncurses5 | HIGH | CVE-2017-10684 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 |
| ncurses/libncurses5 | HIGH | CVE-2017-10685 | Improper Input Validation | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 |
| sqlite3/libsqlite3-0 | HIGH | CVE-2020-9794 | Out-of-bounds Read | 3.8.7.1-1+deb8u6 | |
| sqlite3/libsqlite3-0 | HIGH | CVE-2019-8457 | Out-of-bounds Read | 3.8.7.1-1+deb8u6 | |
| ncurses/libncurses5 | MEDIUM | CVE-2017-16879 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u3 |
| ncurses/libncurses5 | MEDIUM | CVE-2017-13729 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 |
...
+------------------------------------------------------------------------------------------------------------------------------------------------------+
| Base image vulnerabilities from myorg/base |
+---------------------------+-------------+------------------+--------------------------------------+----------------------+---------------------------+
| Package | Severity | ID | Issue | Installed | Fixed in |
+---------------------------+-------------+------------------+--------------------------------------+----------------------+---------------------------+
| apt/libapt-pkg4.12 | HIGH | CVE-2019-3462 | Arbitrary Code Injection | 1.0.9.8.4 | 1.0.9.8.5 |
| bzip2/libbz2-1.0 | HIGH | CVE-2019-12900 | Out-of-bounds Write | 1.0.6-7+b3 | 1.0.6-7+deb8u1 |
| glibc/libc-bin | HIGH | CVE-2018-1000001 | Out-of-Bounds | 2.19-18+deb8u7 | |
| glibc/libc-bin | HIGH | CVE-2014-9761 | Out-of-Bounds | 2.19-18+deb8u7 | |
Middleware team
Base image team
There are really
only 2 new vulns in
middleware
