Nisha introduces Tern, a utility for software package introspection in containers. This tool allows administrators to have the same level of confidence on what's in their containers as they currently do with VM images, including compliance audits, bill of materials, and exploit detection. Nisha is the primary author of Tern. https://github.com/vmware/tern

1. ©2019 VMware, Inc.
Container BoM
Inspection with Tern
Nisha Kumar
Open Source Engineer @ VMware
CloudNativePDX Apr 2019

2. ©2019 VMware, Inc.
Agenda
2
Automating Compliance for Container Images
- CoW Filesystems
- FROM and RUN in Dockerfiles
- Shrinking Container Images
Tern
- What it does
- How it does
- Features and Future Work

3. 3©2019 VMware, Inc.
The first step in OSS Compliance is finding
the Bill of Materials (BoM) for the artifact
(container image)

4. 4©2019 VMware, Inc.
By knowing the BoM, you also know about
dependencies, build variations and vulnerable
packages

5. 5©2019 VMware, Inc.
Automating Compliance for
Containers
The Implications of using CoW Filesystems

6. ©2019 VMware, Inc. 6
Building Container Image Layers
Reference: https://docs.docker.com/storage/storagedriver/
BaseOS: bin boot etc home lib opt root tmp usr var
Read Only Layer
R/W layer: Add new files or remove existing files
Image
apt-get install ca-certificates…
Copy on Write

7. ©2019 VMware, Inc. 7
Building Container Image Layers
Reference: https://docs.docker.com/storage/storagedriver/
BaseOS: bin boot etc home lib opt root tmp usr var
Read Only Layer
R/W layer: Add new files or remove existing files
Diff Files: etc/ca-certificates/* usr/share/ca-certificates/*
Read Only Layer
Image
Copy on Write
wget -O go.tgz “$url”; tar -C /usr/local -xzf go.tgz; rm go.tgz

8. ©2019 VMware, Inc. 8
An image consists of directory trees - one for each layer
Reference: https://docs.docker.com/storage/storagedriver/
BaseOS: bin boot etc home lib opt root tmp usr var
Read Only Layer
Diff Files: etc/ca-certificates/* usr/share/ca-certificates/*
Read Only Layer
Diff Files: /usr/local/go
Read Only Layer
Image

9. ©2019 VMware, Inc. 9
What layers look like on disk

10. 10©2019 VMware, Inc.
Automating Compliance for
Containers
Base images and shell scripts

11. ©2019 VMware, Inc. 11
Containerizing Apps
Using Docker and Dockerfiles
COPY . .
BaseOS: bin boot etc home lib opt root tmp usr var
Debian, Photon, Alpine
Diff files:
install app dependencies and extra modifications
Diff Files: etc/ca-certificates/* usr/share/ca-certificates/*
OS & language package manager dependencies
Diff Files:
Copy app into container
Somebody
else’s
Image
Your
Changes
FROM <image>
RUN <script>

12. ©2019 VMware, Inc. 12
The downside of FROM
golang:1.11
buildpack-deps:stretch-scm
buildpack-deps:stretch-curl
debian:stretch
https://github.com/docker-library/golang/blob/ed78459fac108dab72556146b759516cc65ee109/1.11/stretch/Dockerfile
https://github.com/docker-library/buildpack-deps/blob/1845b3f918f69b4c97912b0d4d68a5658458e84f/stretch/scm/Dockerfile
https://github.com/docker-library/buildpack-deps/blob/1845b3f918f69b4c97912b0d4d68a5658458e84f/stretch/curl/Dockerfile
https://github.com/debuerreotype/docker-debian-artifacts/blob/ed15c6a0b511d2985ca252f59f4318b1fe2a7a59/stretch/Dockerfile

13. 13©2019 VMware, Inc.
Automating Compliance for
Containers
The Implications of shrinking container images

14. ©2019 VMware, Inc. 14
o Remove “unwanted” files (often source files, build artifacts, license and copyright info)
o Remove everything except the runtime dependencies
o The Builder Pattern
Ways to shrink a container image

15. ©2019 VMware, Inc. 15
FROM golang:1.11 as builder
WORKDIR /go/src/github.com/domain/repo
COPY ./ .
RUN go build -v -o bin/executable ./repo/cmd/
FROM photon:2.0
RUN tdnf update -y
WORKDIR /bin/
COPY --from=builder /go/src/github.com/domain/repo/bin/executable .
ENTRYPOINT [ "/bin/executable" ]
CMD [ "8080" ]
EXPOSE 8080
Multistage Docker Builds
App’s Build
Dependencies
App compiled with dependencies
copied over to new container

16. ©2019 VMware, Inc. 16
o Container Images are tarballs containing tarballs with JSON data on how to assemble them
o Container Images are built by running (often times non-reproducible) scripts in a chroot
environment
o Information about the base image used at each reuse is not maintained
o Shrinking the container image by removing files or the build environment makes the resulting
container unreproducible
Recap

17. 17©2019 VMware, Inc.
Tern
Automates investigation of each container
layer

18. ©2019 VMware, Inc. 18

19. ©2019 VMware, Inc. 19

20. 20©2019 VMware, Inc.
Tern
How it gets its results

21. ©2019 VMware, Inc. 21
o sudo mount -t overlay overlay -o
lowerdir=layer2:layer1,upperdir=emptydir,workdir=workdir mergedir
o sudo mount -t proc /proc /path/to/rootfs/proc
o sudo mount -o bind /sys /path/to/rootfs/sys
o sudo mount -o bind /dev /path/to/rootfs/dev
o sudo cp /etc/resolv.conf /path/to/rootfs/etc/resolv.conf
o sudo chroot rootfs /bin/bash -c “dpkg --get-selections”
Step though a container image

22. ©2019 VMware, Inc. 22

23. 23©2019 VMware, Inc.
Tern
Features (that I like) and Future work

24. ©2019 VMware, Inc. 24
o It’s delightfully easy to enable support for any package manager (even if that package
manager doesn’t support some of the metadata)
o Currently supported package managers: dpkg, apt, rpm, tdnf, pacman, apk
o Once enabling a package manager, it will list the full dependency list for each layer for
components installed with that package manager
o You can define your own keys for the metadata reported (coming soon!)
o If you want to hook up to an external service (say a license scanner or security scanner) you
can as long as that scanner can provide the metadata in a list
o If you want to hardcode the metadata for your one off binary that you have 100% confidence
in, you can
Features I like

25. ©2019 VMware, Inc. 25
o Release 0.3.0
o Custom templates
o SPDX format
o Release 0.4.0
o Enable npm
o Future releases
o Enable pip
o Enable GitHub repositories
o SPDX license identifiers
o Call out to external tools
o Source retrieval…
Future work

26. ©2019 VMware, Inc. 26
https://github.com/vmware/tern
o It’s not boring (even though the word “compliance” is boring)
o It’s young - there’s a lot of work to be done
o It’s in a space that is largely neglected by the cloud native community
o I care about contributor/maintainer diversity
o I care more about enabling contributors to submit code than submitting code myself
Contribute!

27. ©2019 VMware, Inc.
Thank You