What are containers, how do we optimize them, consider the supply chain security, and maybe even use them more securely?

1. Navigating Container Technology for Enhanced Security
Niklas Saari 21.11.2023
1 / 34

2. About speaker
██ Niklas Saari
• Doctoral Researcher in the University of Oulu
◦ Oulu University Secure Programming Group (OUSPG)
◦ The topic is about the impact of open-source software for software security of the
satellite systems
▪ Software supply chain security
▪ Software quality analysis with fuzzing and static methods
██ About containers
In the past, maintained and developed around three years many containerized information security tools as
part of CinCan project. Also, container hobbyist...
2 / 34

3. What is this all about
• What are containers and how do we optimise them?
• How about the role of dependencies for software supply
chain security?
• Focus on how to optimise your image build process and
what it means
• Adapt your thinking of security when you operate with
containers!
The logo of Docker Inc. became
well-known symbol of containers
3 / 34

4. Containers?
• Usually, we use them for isolation
• We like to control resource usages in context basis
• Lightweight; they are actually just processes
• Make application build process and deployment portable, shareable and maybe even reproducible
• With higher usability than traditional solutions
• Usually we mean Linux-based on containers, when we talk about them
4 / 34

5. You can even make Linux container from Windows games
Magic The Gathering: Arena works well when played from the Linux container and many others.
• Managing different runtime dependencies more convenient
• For example, you can have different GPU driver for every game! 5 / 34

6. Evolution of modern containers on Linux
———————————————————————————————————————————————————————————————————————————————————————————————————————————
• 1979 - chroot was introduced
• 2002 - First version of Linux Namespaces
• 2006 - "Process Containers" by Google, merged to Linux Kernel as cgroups, (Control Groups)
• 2008 - LXC (LinuX Containers), first, complete implementation
• 2013-2015 - Let Me Contain That For You (LMCTFY) Google Container Stack - influenced the modern
libcontainer
• 2013 - Docker, initially based on LXC, later created own libcontainer
• NOW - Docker emerged to so-called Open Container Inititiative
———————————————————————————————————————————————————————————————————————————————————————————————————————————
6 / 34

7. Chroot
• In Linux, everything is a file
• Change the root of filesystem to new / for the process
• Isolates on filesystem level but not otherwise
• Usually used for testing and compatibility reasons
7 / 34

8. Namespaces
"A namespace wraps a global system resource in an abstraction that makes it appear to the processes within
the namespace that they have their own isolated instance of the global resource." - Linux Manual
As since the Kernel version of 5.6, 8 kinds of namespaces:
Namespace │ Flag │ Isolates
──────────┼─────────────────┼─────────────────────────────────────
Cgroup │ CLONE_NEWCGROUP │ Cgroup root directory
IPC │ CLONE_NEWIPC │ System V IPC, POSIX message queues
Network │ CLONE_NEWNET │ Network devices, stacks, ports, etc.
Mount │ CLONE_NEWNS │ Mount points
PID │ CLONE_NEWPID │ Process IDs
Time │ CLONE_NEWTIME │ Boot and monotonic clocks
User │ CLONE_NEWUSER │ User and group IDs
UTS │ CLONE_NEWUTS │ Hostname and NIS domain name
8 / 34

9. Control Groups cgroups
• Limits, accounts for, and isolates resource usage (CPU, memory, disk I/O, etc. )
◦ Set process-specific limits for resource usage
◦ Prioritization
◦ Reporting and monitoring of the resources
◦ Control
• Hierarchial management
9 / 34

10. LinuX Containers (LXC)
• Adapts namespaces, control groups and chroot to provide isolation and resource limits between
containers
• Adapts overlay filesystems to provide a union of multiple filesystems
• AppArmor and SELinux profiles
———————————————————————————————————————————————————————————————————————————————————————————————————————————
10 / 34

11. Docker
• Became the de facto container solution to pack your application as single unit and operate it
• Until version 0.9, used LXC, systemd-nspawn and libvirt as execution driver
• After that, used their own libcontainer to operate on Linux kernel, later used just abstraction
for different implementations
• Good user experience (UX) as the major reason for success
11 / 34

12. Basic architecture
+------------------------+
| Client |
| (docker CLI, K8s, etc.)|
+-----------+------------+
|
Docker Engine (HTTP API)
|
+-----------+------------+
| containerd |
+-----------+------------+
|
runC (contains libcontainer)
|
+-----------+------------+
| Linux Kernel |
| (namespaces, cgroups, |
| UnionFS, etc.) |
+------------------------+
12 / 34

13. Open Container Initiative (OCI)
"The Open Container Initiative is an open governance structure for the express purpose of creating open
industry standards around container formats and runtimes."
• https://opencontainers.org/
• Founded in 2015 by Docker, CoreOS and other leaders in the container industry.
• Standardisation of the following container specifications:
◦ The OCI Distribution Specification
◦ The OCI Image Format Specification
◦ The OCI Runtime Specification
◦ https://github.com/opencontainers/distribution-spec
13 / 34

14. The OCI Distribution Specification
• To distribute, push, pull, update and manage all the container images
◦ Multi-arch support
◦ Metadata
◦ Digests and their meaning
• Often known as Docker Hub, GitHub Container Registry, Quay or self-hosted registries and so
on..
Each providers follows the same API and image storage format to describe images, but they usually have some
own additional endpoints for managing them on scale.
14 / 34

15. Understanding the image format
██ Demo
• The digest you see in the container registry, is the digest of the manifest
• Manifest describes the chain of cryptographic hashes of the image layers and the configuration
file
◦ As JSON file
◦ Each layer is TAR file, described as SHA256 digest
• Layers re-usable and cacheable. E.g. base image for some other image.
15 / 34

16. Demo commands to get single layer from container image with cURL
██ Authentication token
curl -sSL "https://ghcr.io/token?service=registry.ghcr.io&scope=repository:nicceboy/testimage:pull" >
auth.json
██ Manifest
curl -L -R GET -sH "Authorization: Bearer `jq -r '.token' auth.json`" -H "Accept:
application/vnd.oci.image.manifest.v1+json" "https://ghcr.io/v2/nicceboy/testimage/manifests/latest" | jq
██ Configuration
curl -L -R GET -sH "Authorization: Bearer `jq -r '.token' auth.json`" -H "Accept:
application/vnd.oci.image.layer.v1.tar+gzip"
"https://ghcr.io/v2/nicceboy/testimage/blobs/sha256:6dee449fabe300f361391e87da6c6204caa7c5cb9569f2c8759679e
d96d08e0f" | jq
██ Binary blob with .txt file
curl -L -R GET -sH "Authorization: Bearer `jq -r '.token' auth.json`" -H "Accept:
application/vnd.oci.image.layer.v1.tar+gzip"
"https://ghcr.io/v2/nicceboy/testimage/blobs/sha256:5897666521003f1bd481840975b2f7368359a8ad382124b4ff0c583
cd3b59f7a" > out.tar
██ Unpack the layer content
tar xvf out.tar
16 / 34

17. Inspecting the image
Using curl, jq, tar and less will get you started. Or just podman inspect.
However, there are better tools...
◦ https://github.com/wagoodman/dive
◦ View all changes of the container image from all layers
◦ Can help you to optimise the image
-> We can conclude that there are a lot dependencies and additional data which might be possible to be
reduced.
-> Also we were leaking some data on purpose, which might happen by accident for someone.
17 / 34

18. Software Bill of Materials (SBOM)
• A detailed catalog how the software is built
• You must known your software dependencies - they can be vulnerable too
• Dependencies must be controllable, listable and trackable
• Software supply chain provides additional attack vector
• 2023 was record breaking again with software supply chain attacks: increase by 742% between
2019 and 2022, according to Sonatype
DALL-E view of SBOM 18 / 34

19. Dependencies and why they do matter
██ The OCI image itself
◦ Do you trust the base image?
◦ If compromised, your build process and new deployment is compromised
██ Application dependencies
◦ How many are not actually needed?
▪ Can increase the attack surface
▪ Compromised package by malicious actor or developer
▪ Vulnerability
▪ Additional noise when identifying outdated or vulnerable software
██ Make things more secure and efficient
-> Build the images with as little dependencies as possible, and by using trusted sources
-> Also reduced disk usage and bandwidth cost
19 / 34

20. Basic optimisation: multi-stage builds with traditional base images
██ Demo
• Build your application in separate stage
• Then copy into the runtime image with runtime dependencies only
◦ Select runtime for you needs (Alpine Linux, Debian slim-based, something else?)
• Avoids leaking build-time secrets to runtime image
• Reduces the image size and dependencies are closer to what are actually required
20 / 34

21. Hardcore: from scratch
██ Demo
• Sometimes your application can be statically build and has only a little of dependencies
• Container is just a process, image can contain only one executable!
• However, not always practical and there are some issues
21 / 34

22. What you usually need in the container
• User management (/etc/passwd and /etc/group files) for namespace isolation
• Some important folders that applications' expect to exist (/var, /tmp, /home)
• libc or musl as C Standard Library
• Timezone database
• CA certificates for TLS to work
22 / 34

23. Entering "distroless"
• Google provides so-called "distroless" base images which attempts solve the previous
• Development in https://github.com/GoogleContainerTools/distroless
• gcr.io/distroless/*
• Different dependency options, depending on the needs
• All images are cryptographically signed
23 / 34

24. Basic relationships
Image credits: Ivan Velichko - https://iximiuz.com/ 24 / 34
▓▓▓ Demo

25. Going further
Distroless images are often enough for your needs. What if your
application is very complex?
• You will need an advanced image builder
◦ Bazel: https://bazel.build
• Declarative package managers are especially useful
◦ Nix package manager: https://nixos.org
◦ Guix: https://guix.gnu.org
◦ Fully reproducible images, with explicit
dependencies and nothing else
25 / 34

26. Tracking your dependencies
Once you have the optimised image - you need to track the lifecycle of your dependencies; are there known
vulnerabilities?
You can integrate different static analysis tools for your container registry or CI build pipelines.
Depending on your application, different ways.
• RedHat's clair: https://github.com/quay/clair
• Trivy: https://github.com/aquasecurity/trivy
• Multi-layered - e.g. use GitHub's dependency bot to check package.json for Node-based
applications
• There are many, depending on the context
26 / 34

27. How about using containers more securely?
Know defaults when running containers. What do you need?
Applications can be very different with different requirements. Find out what you need.
• Kernel capabilities
• Mount points and ports
• User privileges
• Container resources (CPU && RAM?)
• Principle of least privilege (or anything)
27 / 34

28. Kernel capabilities
• Never use --privileged mode
• By default, Docker runs with many capabilities.
• UX over security? 🤨
docker run --cap-drop=all my_container
Select explicitly what you need!
28 / 34

29. Docker default kernel capabilities
• CAP_CHOWN: Allows changing file ownership.
• CAP_DAC_OVERRIDE: Bypasses file read, write, and execute permission checks.
• CAP_FSETID: Don't clear set-user-ID and set-group-ID permission bits when a file is modified.
• CAP_FOWNER: Bypasses permission checks on operations that require the filesystem UID of the
process to match the UID of the file.
• CAP_MKNOD: Allows creation of special files using mknod().
• CAP_NET_RAW: Allows use of RAW and PACKET sockets.
• CAP_SETGID: Allows setting the GID and group membership.
• CAP_SETUID: Allows setting the UID.
• CAP_SETFCAP: Allows setting file capabilities.
• CAP_SETPCAP: Allows transferring or dropping any capability in the calling thread's permitted
capability set.
• CAP_NET_BIND_SERVICE: Allows binding to Internet domain sockets. Ports less than 1024
• CAP_SYS_CHROOT: Allows use of chroot().
• CAP_KILL: Allows sending signals to processes other than those belonging to the current user.
• CAP_AUDIT_WRITE: Allows writing to the Linux kernel's audit log.
29 / 34

30. Mount points
Mount only when there is no other option.
• Only directories which are specifically for that purpose
• Prefer copying instead of mounting if possible
• Usually the only way to break the isolation, excluding runner or kernel vulnerabilities
• Mount explicitly as read-only, if no write access needed
30 / 34

31. Principle of least privilege
• Make an user and make sure that UID/GIDs are unique across all containers
• Run with --security-opt=no-new-privileges to prevent privilege escalation by using setuid or
setgid binaries.
• Disable inter-container communication --icc=false
• Use AppArmor, SELinux or seccomp when possible
• Excellent summary with more details by OWASP:
https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html#rule-9-use-stat
ic-analysis-tools
Give only the resources required for containers in order to prevent exhausting whole system by single
container.
31 / 34

32. Container runners
Architecture can have significant impact for security. Run on rootless mode when possible.
• Vulnerability on the runner does not affect every container or whole system
• Container cannot be run as "real" root by accident
██ Docker vs. Podman
Feature │ Docker │ Podman
────────────────────┼───────────────────────────────┼──────────────────────────────────────────────────────
Daemon │ Runs as a central daemon │ Daemonless, each container runs in a separate process
Root Privileges │ Root by default │ Rootless by default
User Namespaces │ Supported, but not by default │ Supports rootless mode by default
SELinux Integration │ Supported │ Better default integration with SELinux
Seccomp Profiles │ Enabled by default │ Enabled by default, with more restrictive profiles
API Compatibility │ Docker API │ Compatible with Docker CLI, can run docker commands
Default Network │ Bridge │ Host network, with bridge available
32 / 34

33. There are many more
• colima, Rancher, OrbStack (commercial)
• Variations of kubernetes (usually containerd under the hood)
33 / 34

34. Conclusion
• Manage your SBOM from the beginning when creating and operating with containers
◦ Minimal dependencies
◦ Trusted image sources
• Follow your dependencies lifecycle
• Principle of least anything when running and using containers
• Know what happens on default and what your container images contain
34 / 34