This document discusses using Linux Integrity Mechanisms (IMA) to protect container runtime executables from tampering. It begins with an overview of container technology and a motivating example attack (CVE-2019-5736). It then provides background on IMA and how it can be used to measure and appraise the integrity of container runtime executables. Specifically, it details using IMA rules and digital signatures to enforce that the runc binary has not been tampered with. This helps prevent attacks that aim to overwrite the runc binary and escape the container.

1. Linux Integrity Mechanisms -
Protecting Container Runtime as
an Example
Clay Chang
clay.chang@gmail.com
1-Aug-2020, COSCUP’20

2. Who Am I?
• Master Technologist (Linux) at Hewlett Packard Enterprise (HPE)
• Researcher at NTU
• COSCUP’17 - Memory-Style Storage in Linux
• COSCUP’19 - Remote Memory
This talk is a view of my own!

3. Contents
• Introduction to Container Technology
• Motivating Example - CVE-2019-5736
• Background of Linux Integrity Measurement Architecture (IMA)
• Protecting Container Runtime Executables with IMA

4. Container Technology
Docker Engine
(dockerd)
Podman
containerd
runc
Docker CLI Podman CLI
CRI-O
kubelet

5. CVE-2019-5736 aka Container Escape
• Description: … allows attackers to overwrite the host runc binary
(and consequently obtain host root access) by leveraging the ability
to execute a command as root within one of these types of
containers: (1) a new container with an attacker-controlled image, or
(2) an existing container, to which the attacker previously had write
access, that can be attached with docker exec. This occurs
because of file-descriptor mishandling, related to /proc/self/exe.
https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-and.html

6. # git clone https://github.com/q3k/cve-2019-5736-poc
# docker build -t cve .
# strings /usr/bin/runc | tail -n 2
.gnu_debuglink
.gnu_debugdata
# docker run cve
Emulate Docker CLI using podman. Create /etc/containers/
nodocker to quiet msg.
HAX2: argv: /proc/self/fd/3
HAX2: fd: 4
HAX2: res: 13, 0
# strings /usr/bin/runc | tail -n 2
.debug_gdb_scripts
cve-2019-5736

7. Integrity Measurement Architecture
(IMA)
• Since v2.6.30 (9 Jun 2009)
• Key Features of IMA
• Detect if files have been altered accidentally or maliciously, both
remotely or locally
• Appraise a file’s measurement against a “good” value stored as
an extended attributes
• Enforce local file integrity

8. IMA Characteristics
• Policy-based rules
• Actions: measure / dont_measure / appraise / dont_appraise / …
• Conditions:
• File-based operations: BPRM_CHECK / MMAP_CHECK / FILE_CHECK / …
• Masks: MAY_EXEC / MAY_READ / MAY_WRITE / …
• Specific fsmagic / fsuuid / uid / …
• Supports LSM (Linux Security Module, e.g. SELinux): obj_type / obj_user / obj_role / …
• Options: appraise_type / keyring / …

9. # PROC_SUPER_MAGIC
dont_measure fsmagic=0x9fa0
dont_appraise fsmagic=0x9fa0
# SYSFS_MAGIC
dont_measure fsmagic=0x62656572
dont_appraise fsmagic=0x62656572
# DEBUGFS_MAGIC
dont_measure fsmagic=0x64626720
dont_appraise fsmagic=0x64626720
# TMPFS_MAGIC
dont_measure fsmagic=0x01021994
dont_appraise fsmagic=0x01021994
# RAMFS_MAGIC
dont_appraise fsmagic=0x858458f6
# DEVPTS_SUPER_MAGIC
dont_measure fsmagic=0x1cd1
dont_appraise fsmagic=0x1cd1
# BINFMTFS_MAGIC
dont_measure fsmagic=0x42494e4d
dont_appraise fsmagic=0x42494e4d
# SECURITYFS_MAGIC
dont_measure fsmagic=0x73636673
dont_appraise fsmagic=0x73636673
# SELINUX_MAGIC
dont_measure fsmagic=0xf97cff8c
dont_appraise fsmagic=0xf97cff8c
# CGROUP_SUPER_MAGIC
dont_measure fsmagic=0x27e0eb
dont_appraise fsmagic=0x27e0eb
# NSFS_MAGIC
dont_measure fsmagic=0x6e736673
dont_appraise fsmagic=0x6e736673
measure func=BPRM_CHECK
measure func=FILE_MMAP mask=MAY_EXEC
measure func=FILE_CHECK mask=MAY_READ uid=0
appraise fowner=0
Measure all executables in
bprm_check
3
1
2
4
1
Measure all files mapped
executable in file_mmap
2
Measure all files open for read
by root in do_filp_open
3
Appraise all files owned by root4
IMA Policy
Example

10. # cat policy > /sys/kernel/security/ima/policy
# cat /sys/kernel/security/ima/ascii_runtime_measurements
-bash: /usr/bin/cat: Permission denied
# dmesg
-bash: /usr/bin/dmesg: Permission denied
# ls
-bash: /usr/bin/ls: Permission denied
# reboot
-bash: /usr/bin/ls: Permission denied

11. # cat policy2 > /sys/kernel/security/ima/policy
# cat /sys/kernel/security/ima/ascii_runtime_measurements
[root@goldenarmor ima-policy]# cat /sys/kernel/security/ima/ascii_runtime_measurements
10 1d8d532d463c9f8c205d0df7787669a85f93e260 ima-ng sha1:0000000000000000000000000000000000000000 boot_aggregate
10 0000000000000000000000000000000000000000 ima-ng sha1:0000000000000000000000000000000000000000 /tmp/sh-thd.5cBnGN
10 d8ad76de60ab2242cf08bdc1b8bfd0ede36bc983 ima-ng sha1:a8c05f516f368a0654e9e3ae3b3217b6dd40e491 /tmp/sh-thd.5cBnGN
10 0000000000000000000000000000000000000000 ima-ng sha1:0000000000000000000000000000000000000000 /tmp/sh-thd.ccEXk2
10 fb5a5e532147ef93b494ae391022620cf8e2c78b ima-ng sha1:a8c05f516f368a0654e9e3ae3b3217b6dd40e491 /tmp/sh-thd.ccEXk2
10 37a4d23f6ba7f2d5c13747f463f9ff1a2f155be8 ima-ng sha1:359819795e404a69d8a4a52dddccdf2bf1d3859e /usr/bin/gawk
10 a791d8db0076287a372053cbf0882feeea70b57e ima-ng sha1:cf13787cf9f3961d8c94eee05b75727c862cc986 /usr/lib64/ld-2.28.so
10 032d456298ce54f8f4bcd50bd3ddfbe73a16abb7 ima-ng sha1:e337d4fefc5c23e89c57da45bdc48cefed10d547 /etc/ld.so.cache
10 d4bc023942ec7eb0f9093502b7d2234252b2f143 ima-ng sha1:8cb57415f34031b50a7193c6a0176f1a74f2d03c /usr/lib64/libsigsegv.so.2.0.4
10 b7e0667a05d829ac3c3798d9fecc2b8c1921f059 ima-ng sha1:bb4c4b67723c732f12352e5449da1c1f0d2123d7 /usr/lib64/libreadline.so.7.0
10 d085818f8e9e0e7a8d2c24bc2e2b2e5af2d7c426 ima-ng sha1:e028f33771036ece3505a586b4f562992e188996 /usr/lib64/libtinfo.so.6.1
10 fc48f8598aae7723d0c9f7e3035f2eb900d772f0 ima-ng sha1:9403913e901469a4c97219ac9daf2fdb701bb483 /usr/share/locale/locale.alias
10 69f01172cf5cb9c2f6d5c17f3f065f2de924fee7 ima-ng sha1:692a1e79d6e453c99714ec37bb4999abb57f5b44 /usr/lib/locale/en_US.utf8/LC_CTYPE
10 77a0b37e3bc8d8fe55eb1ad3a8fa6f108be73bab ima-ng sha1:a82fb20c382fc9e5dec432ca5ee2f04684536e94 /usr/lib64/gconv/gconv-modules.cache
10 1393afae5534259ab7ad4ee7a759643ccccafc5c ima-ng sha1:aa5248b0ac085665477b2feb1266205b3bed557d /usr/lib/locale/en_US.utf8/LC_ADDRESS
10 eb7c76027b0b21e55693619d884995916d140058 ima-ng sha1:a1a2eabc4a2a2da47d7a90d36b4d0826aac03c73 /usr/lib/locale/en_US.utf8/LC_NAME
...
10 d02d14d189677572ce53d868447b1c9961b9faa6 ima-ng sha1:c5b36af5bfba8854a4209bb6e905c1a7eb47caad /usr/lib64/libidn2.so.0.3.6
10 3477e8cb1563071d5b09695709e7e2d31f567e90 ima-ng sha1:e6dc3965b93be37385e783727fbb0dfdabd4c7c4 /usr/lib64/libattr.so.1.1.2448
10 ce4a6d8b052af0c8d9cff507552d3f5839d9871f ima-ng sha1:47dab9dcb38148a217773302a9384bcfa21ba974 /usr/lib64/libdevmapper.so.1.02
10 f0289ee791a681cf00f512a66351fc295c1d0507 ima-ng sha1:f7259220100a2377ea7fb6ccda3c3037c63debfe /usr/lib64/libssl.so.1.1.1c
10 942fe25b8dd8b66c0f906af71b4c986fb96a67b6 ima-ng sha1:1b533c9e2826d8652e133c979dab4906ec37fdce /usr/lib64/libcrypto.so.1.1.1c
10 0ca2d280e7e3b714caf850203322d7eaa8fae9b8 ima-ng sha1:e6f458475bb286dfdcd98171550fbb0c38917cc6 /usr/lib64/libjson-c.so.4.0.0
10 0e964e27ce222afe78a3e4a030a06ff7eca96048 ima-ng sha1:d3fa4ee3c0ef05ac5b3bf4126facceb7ca59dced /usr/lib64/libpcap.so.1.9.0
10 323e40c5dd45055ece2032058ace3d604e6bc4f8 ima-ng sha1:340c7b865b132a0ef0ea51985435861e347fa2ab /usr/lib64/libunistring.so.2.1.0
10 a042f42bc53ec00abaf8de4efe536b4e50ac3ea9 ima-ng sha1:dec7b77de0dab91f28e9d9f265775c04642d70d5 /usr/lib64/libsepol.so.1
10 8820e6b8ea6d401c8e287b66a8e11bf4b50f435c ima-ng sha1:1802538af67a8306fbf85e12a1a9e18589de475b /usr/lib64/libudev.so.1.6.11
10 7e6a73ee70e9829795945ad3d02d94048cc64f5c ima-ng sha1:442d6bbeb51f089ccb9a0f31e1cf79d9eb8cb42c /usr/lib64/libz.so.1.2.11
10 0000000000000000000000000000000000000000 ima-ng sha1:0000000000000000000000000000000000000000 /tmp/sh-thd.YUyyzn
10 cfb6053df14de2785ef1caf7a77d8e84c7a2406d ima-ng sha1:5f30a9ee5335c8ef0aa17876ddc5d622daec8b15 /tmp/sh-thd.YUyyzn
10 0f60e2884367dee70864927ba04f2e657209355e ima-ng sha1:6207eaf514d465da23059e66759456dfe162349d /usr/bin/cat

12. The Leverage & Solution
• container-selinux: SELinux Policy for Container Runtimes
• New IMA rule:
# ls -Z /usr/bin/runc
system_u:object_r:container_runtime_exec_t:s0 /usr/bin/runc
appraise func=BPRM_CHECK obj_type=container_runtime_exec_t appraise_type=imasig

13. What is appraise_type=imasig?
• Use digital signature of the hash of a file as the “good” value
• Store the digital signature in the xattr (extended attribute) of the file
# evmctl ima_sign -a sha256 -k privkey.pem /usr/bin/runc
# getfattr -m "." -d /usr/bin/runc
getfattr: Removing leading '/' from absolute path names
# file: usr/bin/runc
security.ima=0sAwIEH+VEdQEAkN3YSXNhBLf...UJQ8tk7...LhYUZVy5wmEmtBXSkBHP9yTZFg==
security.selinux="system_u:object_r:container_runtime_exec_t:s0"

14. How IMA appraise the file with imasig?
# keyctl describe %keyring:.ima
574740493: -lswrv--swrv------------ 0 0 keyring: .ima
# evmctl import x509_ima.der 574740493
# keyctl show %keyring:.ima
Keyring
574740493 ---lswrv 0 0 keyring: .ima
4225879 --als--v 0 0 _ asymmetric: IMA Signing Key:
e52f68af018134bfa644cbda02b5c8ef1fe54475
• Load the public key onto the .ima keyring (*)
• IMA uses the public keys loaded onto the .ima keyring to verify the
digsig (security.ima) of the hash of the file being appraised
(*) The public key must be signed by the trusted certificate in the kernel

15. # docker run cve
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
HAX2: argv: /proc/self/fd/3
HAX2: fd: 4
HAX2: res: 13, 0
# strings /usr/bin/runc | tail -n 2
.debug_gdb_scripts
cve-2019-5736
# docker run cve
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
ERRO[0000] Error removing container
6ed3c7ab94169e53352b3df9e461d4406d7851389948c4de39cc2cd0df7b4909 from runtime after
creation failed
Error: container create failed (no logs from conmon): EOF

16. References
• https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-
and.html
• Integrity Verification of Docker Containers for a Lightweight Cloud
Environment, Future Generation Computer System 2019
• Leveraging Kernel Security Mechanisms to Improve Container Security: a
Survey, ARES’19
• Linux Integrity Subsystem Update by Mimi Zohar, Linux Security Summit 2019
• The Keylime Project, https://keylime.dev

17. Q/A

18. Backups

19. BPRM_CHECK
execve syscall
-> exec_binprm (fs/exec.c)
-> search_binary_handler (fs/exec.c)
-> security_bprm_check (security/security.c)
-> ima_bprm_check (security/integrity/ima/ima_main.c)
MMAP_CHECK
mmap_pgoff sys call
-> vm_mmap_pgoff (mm/util.c)
-> security_mmap_file (security/security.c)
-> ima_file_mmap (security/integrity/ima/ima_main.c)
FILE_CHECK
openat2 sys call
-> do_filp_open (fs/namei.c)
-> path_openat (ds/namei.c)
-> do_open (fs/namei.c)
-> ima_file_check (security/integrity/ima/ima_main.c)

20. IMA Keyring: .crs or _crs?
• Kernel config CONFIG_INTEGRITY_TRUSTED_KEYRING determines whether .ima
(trusted keyring) or _crs (regular keyring) is used for IMA
• If .ima was used, only those keys signed by a “kernel-trusted certificate” can be
loaded onto the .ima keyring for appraisal
• We can use CONFIG_SYSTEM_EXTRA_CERTIFICATE to reserve space in the
kernel for embedding own certificate to the kernel
• Then use our own certificate to sign the IMA signing public key
• Details refer to the “Generate trusted keys” in http://linux-ima.sourceforge.net/evmctl.1.html
• Use scripts/insert-sys-certs to insert the certificate into the kernel