Container Security
Salman A. Baset
@salman_baset, sabaset@us.ibm.com
What is a container?
2
What is a container?
According to NIST (National Institute of Standards Technology)
• Virtualization: the simulation of the software and/or hardware upon which
other software runs. (800-125)
• System Virtual Machine: A System Virtual Machine (VM) is a software
implementation of a complete system platform that supports the execution of
a complete operating system and corresponding applications in a cloud. (800-
180 draft)
• Operating System Virtualization (aka OS Container): Provide multiple
virtualized OSes above a single shared kernel (800-190). E.g., Solaris Zone,
FreeBSD Jails, LXC
• Application Virtualization (aka Application Containers): Same shared kernel
is exposed to multiple discrete instances (800-180 draft). E.g., Docker
(containerd), rkt
3
“Container” Security
Orchestrator
Security
Content
Security
Host
Security= + +
2002 – to-date2015 – to-date2016 - date
Implement defense in depth
App
Security +
4
* - date
Container Runtime Stack on a Single Host
Host kernel
Container
runtime
VM kernel
Host kernel
Container
runtime
Application
containers
App
Bins/libs
App
Bins/libs
App
Bins/libs
App
Bins/libs
Application
containers
1. Is host isolated from application container?
2. Is one application container isolated from another application container?
5
Shared kervel vs. separate kernel debate…
CVEs inside Linux kernel
https://www.cvedetails.com/product/47/Linux-Linux-Kernel.html?vendor_id=33 6
The previous chart can be misleading…
• Gain privilege exploits in 2017: 36
• Includes CVEs in previous kernel versions
• 1 eCryptfs 3.18 or earlier
• Drivers: 5 Qualcomm drives, 2 NVIDIA, 1 USB, 1 BROADCOM, 1 GPU,
1 TTY
• KVM: 1 (that is, CVE found in kvm also)
• Not all CVEs are [easily] exploitable, but nevertheless…
Host kernel
Container
runtime
App
Bins/libs
App
Bins/libs
Application
containers
Focus of this talk
7
Usage Model
Host kernel
Container
runtime
App
Bins/libs
App
Bins/libs
Application
containers
Host kernel
Container
runtime
App
Bins/libs
App
Bins/libs
Application
containers
Host kernel
Container
runtime
App
Bins/libs
App
Bins/libs
Application
containers
Multiple apps of same user running on same/different hosts
Multiple apps of distinct users running on same/different hosts
Image
Registry
Orchestrator + +
8
What does host security mean? Informally…
• Isolation from host
• Can never be root
• Should not “see” host processes
• Shout not interfere with host operation
• Isolation from other containers
• Should not “see” other containers
• Should not impact performance of other containers
9
Container on Linux
• is just a process
• wrapper in bunch of “isolation gear”
• to isolate from host and other processes
• the isolation gear was developed independently over time
• docker engine (on a host) + containerd + runC
10
Namespaces
• Linux kernel namespaces provide the isolation
(hence “container”) in which we place one or more
processes
• Introduced ~2002 – inspired from Plan B
• Ok, what about devices?
• docker options
• --userns
• --pid
• --uts
pid mount
ipc
user net
uts
HOST SECURITY
11
Resource isolation - cgroups
• What is a
resource?
• CPU, memory,
disk, network
• PID, file
descriptors
• Devices
• cgroups v2 support introduced in Kernel 4.5.
• Docker has many options for tuning cpu, mem, disk
• cpu: 10
• device: 8
• memory: 5
• PID: configure the max number of PID descriptors
• Device: maximum device bandwidth, in, out
HOST SECURITY
12
Are namespaces and cgroups enough?
• No
• Linux capabilities:
• Fine-grained access capabilities besides root/non-root
• E.g., load a module, mount a file
• Docker container drops most capabilities by default
• chown, dac_override, fsetid, fowner,
mknod, net_raw, setgid, setuid, setfcap,
setpcap, net_bind_service, sys_chroot,
kill, audit_write
• Seccomp
• Restrict the system calls that a system is allowed to execute
• Often, security issues found in system calls pertaining to legacy
devices
• AppArmor / SELINUX
• Mandatory access control (MAC)
Default Docker capabilities
HOST SECURITY
13
Is that enough for host security?
• No
• Other host security best practices still apply!
• Patch management
• Monitoring
• Antivirus
• Malware
• Logging
• Privileged user monitoring
HOST SECURITY
14
Container Images
• Container images are the root of software supply chain
• Containers images are supposed to provide an immutable source
• In reality: secrets needed for running the container are stored outside the
container image, and can impact its run-time
• Different secrets for different run-time environments (stage, prod)
• Container images are often pulled through open source
• Important to vet the base image, malware, anti-virus
• Container images contain OS distro packages + application
packages + application code
• Ubuntu, Node.js
• Check vulnerabilities of all content that goes inside the image, including
scanning application source code
CONTENT SECURITY
App
Bins/libs
secrets
15
Orchestrator Security
• Containers are often deployed through orchestrators
• Docker Machine, Kubernetes
• Sane defaults, TLSv1.2 etc
• Who has access to those APIs just equally as important as other
aspects
• determine what volumes, devices are passed inside container
• who can docker exec into a running container
• Moreover, secrets are typically stored outside container registry
and container run-time. Access to these secrets also important
• Your Jenkins server may be leaking secrets!
ORCHESTRATOR SECURITY
16
App Security
• A developer still needs to configure their app securely.
• However it is still much better to run app in a container than inside a host
APP SECURITY
17
Configurepartitions
Runasnon-root,
utilizinguserand
kernelnamespaces
Configurelog,
monitoring,audit
Configurenetwork,
relatedservices,
disableIPforwarding
Configurepatching
andanti-virusagents
Configuremandatory
accesscontrol
Preventapplication
fromDoSingahost
Furtherdeprivilege
applicationthrough
subsetofLinux
capabilities
FurtherDeprivilege
applicationbylimiting
allowedsystemcalls
Configureapplication
security
Host
Container
Host
Manual
Manual
Manual
per app
One time
(in Docker)
Manual
per app
Manual
One time
Step 1 Step 2 Step 3
Manual
Manual
Step 4
Manual
Manual
Step 5 Step 6
Manual
per app
Out of box
(in Docker)
Out of box
(in Docker)
Step 7
Manual
per app
Step 8
Manual
per app
Out of box
(in Docker)
Manual
per app
Out of box
(in Docker)
Step 9 Step 10
Manual
per app
Manual
Kernel and Container Security Evolution
• 2015
• Content Addressability (image spec, phase 1) – Docker 1.6
• Default ulimits for all containers – Docker 1.6
• Docker Content Trust – (notary) image provenance/signing – Docker 1.8
• 2016
• Full migration to content addressability for images/layers – Docker 1.10
• User namespaces – Docker 1.10
• Secure computing (libseccomp) – Docker 1.10
• --pids-limit (cgroups pid limitation) – Docker 1.11 (kernel 4.3 +)
• cgroups “v2” – (kernel 4.5+)
• --no-new-privileges (limit process escalation) – Docker 1.11
• Storage driver quotas (limited) – Docker 1.12
• Secure by default multi-node orchestration (mutual TLS) – Docker 1.12
• 2017
• Improved resource isolation features in Linux Kernel and in Docker engine
18
Whats Next in Container Security?
• Host Security
• Linux Kit
• Lightweight virtualization (Katacontainers (formerly Intel clear
containers) )
• Unikernels (MirageOS)
• Content Security
• Distribution Specific Project (OCI)
• Project Grafeas
19
Linux Kit
• Secure, portable OSes for Linux containers
• Support Kubernetes, AWS, GCP, Azure etc
• https://github.com/linuxkit/linuxkit
20
Katacontainers
• Based on Intel Clear Containers
• Perform like containers but provide the workload isolation and
security advantages of VMs
• OCI compliant
• https://github.com/kata-containers/
21
MirageOS
• Library operating system for unikernels
• MirageOS3.0 released in Feb 2017
• https://mirage.io/
22
Distribution Specific Project
• Image and run-time formats through OCI
• Image distribution is now defacto Docker Image Registry API
• https://www.opencontainers.org/announcement/2018/04/09/oci-
announces-dist-spec-project
23
Project Grafeas
• Standardize container vulnerability format
• Standardize attestations
• https://grafeas.io/
24
Conclusion
• Container security has significantly evolved from early days of
Docker
• Secure-by-default in all major platforms
• For users, security focus is on securing software supply chain
• New features promise standardization, light-weight, VM
isolation
25

Container Security

  • 1.
    Container Security Salman A.Baset @salman_baset, sabaset@us.ibm.com
  • 2.
    What is acontainer? 2
  • 3.
    What is acontainer? According to NIST (National Institute of Standards Technology) • Virtualization: the simulation of the software and/or hardware upon which other software runs. (800-125) • System Virtual Machine: A System Virtual Machine (VM) is a software implementation of a complete system platform that supports the execution of a complete operating system and corresponding applications in a cloud. (800- 180 draft) • Operating System Virtualization (aka OS Container): Provide multiple virtualized OSes above a single shared kernel (800-190). E.g., Solaris Zone, FreeBSD Jails, LXC • Application Virtualization (aka Application Containers): Same shared kernel is exposed to multiple discrete instances (800-180 draft). E.g., Docker (containerd), rkt 3
  • 4.
    “Container” Security Orchestrator Security Content Security Host Security= ++ 2002 – to-date2015 – to-date2016 - date Implement defense in depth App Security + 4 * - date
  • 5.
    Container Runtime Stackon a Single Host Host kernel Container runtime VM kernel Host kernel Container runtime Application containers App Bins/libs App Bins/libs App Bins/libs App Bins/libs Application containers 1. Is host isolated from application container? 2. Is one application container isolated from another application container? 5
  • 6.
    Shared kervel vs.separate kernel debate… CVEs inside Linux kernel https://www.cvedetails.com/product/47/Linux-Linux-Kernel.html?vendor_id=33 6
  • 7.
    The previous chartcan be misleading… • Gain privilege exploits in 2017: 36 • Includes CVEs in previous kernel versions • 1 eCryptfs 3.18 or earlier • Drivers: 5 Qualcomm drives, 2 NVIDIA, 1 USB, 1 BROADCOM, 1 GPU, 1 TTY • KVM: 1 (that is, CVE found in kvm also) • Not all CVEs are [easily] exploitable, but nevertheless… Host kernel Container runtime App Bins/libs App Bins/libs Application containers Focus of this talk 7
  • 8.
    Usage Model Host kernel Container runtime App Bins/libs App Bins/libs Application containers Hostkernel Container runtime App Bins/libs App Bins/libs Application containers Host kernel Container runtime App Bins/libs App Bins/libs Application containers Multiple apps of same user running on same/different hosts Multiple apps of distinct users running on same/different hosts Image Registry Orchestrator + + 8
  • 9.
    What does hostsecurity mean? Informally… • Isolation from host • Can never be root • Should not “see” host processes • Shout not interfere with host operation • Isolation from other containers • Should not “see” other containers • Should not impact performance of other containers 9
  • 10.
    Container on Linux •is just a process • wrapper in bunch of “isolation gear” • to isolate from host and other processes • the isolation gear was developed independently over time • docker engine (on a host) + containerd + runC 10
  • 11.
    Namespaces • Linux kernelnamespaces provide the isolation (hence “container”) in which we place one or more processes • Introduced ~2002 – inspired from Plan B • Ok, what about devices? • docker options • --userns • --pid • --uts pid mount ipc user net uts HOST SECURITY 11
  • 12.
    Resource isolation -cgroups • What is a resource? • CPU, memory, disk, network • PID, file descriptors • Devices • cgroups v2 support introduced in Kernel 4.5. • Docker has many options for tuning cpu, mem, disk • cpu: 10 • device: 8 • memory: 5 • PID: configure the max number of PID descriptors • Device: maximum device bandwidth, in, out HOST SECURITY 12
  • 13.
    Are namespaces andcgroups enough? • No • Linux capabilities: • Fine-grained access capabilities besides root/non-root • E.g., load a module, mount a file • Docker container drops most capabilities by default • chown, dac_override, fsetid, fowner, mknod, net_raw, setgid, setuid, setfcap, setpcap, net_bind_service, sys_chroot, kill, audit_write • Seccomp • Restrict the system calls that a system is allowed to execute • Often, security issues found in system calls pertaining to legacy devices • AppArmor / SELINUX • Mandatory access control (MAC) Default Docker capabilities HOST SECURITY 13
  • 14.
    Is that enoughfor host security? • No • Other host security best practices still apply! • Patch management • Monitoring • Antivirus • Malware • Logging • Privileged user monitoring HOST SECURITY 14
  • 15.
    Container Images • Containerimages are the root of software supply chain • Containers images are supposed to provide an immutable source • In reality: secrets needed for running the container are stored outside the container image, and can impact its run-time • Different secrets for different run-time environments (stage, prod) • Container images are often pulled through open source • Important to vet the base image, malware, anti-virus • Container images contain OS distro packages + application packages + application code • Ubuntu, Node.js • Check vulnerabilities of all content that goes inside the image, including scanning application source code CONTENT SECURITY App Bins/libs secrets 15
  • 16.
    Orchestrator Security • Containersare often deployed through orchestrators • Docker Machine, Kubernetes • Sane defaults, TLSv1.2 etc • Who has access to those APIs just equally as important as other aspects • determine what volumes, devices are passed inside container • who can docker exec into a running container • Moreover, secrets are typically stored outside container registry and container run-time. Access to these secrets also important • Your Jenkins server may be leaking secrets! ORCHESTRATOR SECURITY 16
  • 17.
    App Security • Adeveloper still needs to configure their app securely. • However it is still much better to run app in a container than inside a host APP SECURITY 17 Configurepartitions Runasnon-root, utilizinguserand kernelnamespaces Configurelog, monitoring,audit Configurenetwork, relatedservices, disableIPforwarding Configurepatching andanti-virusagents Configuremandatory accesscontrol Preventapplication fromDoSingahost Furtherdeprivilege applicationthrough subsetofLinux capabilities FurtherDeprivilege applicationbylimiting allowedsystemcalls Configureapplication security Host Container Host Manual Manual Manual per app One time (in Docker) Manual per app Manual One time Step 1 Step 2 Step 3 Manual Manual Step 4 Manual Manual Step 5 Step 6 Manual per app Out of box (in Docker) Out of box (in Docker) Step 7 Manual per app Step 8 Manual per app Out of box (in Docker) Manual per app Out of box (in Docker) Step 9 Step 10 Manual per app Manual
  • 18.
    Kernel and ContainerSecurity Evolution • 2015 • Content Addressability (image spec, phase 1) – Docker 1.6 • Default ulimits for all containers – Docker 1.6 • Docker Content Trust – (notary) image provenance/signing – Docker 1.8 • 2016 • Full migration to content addressability for images/layers – Docker 1.10 • User namespaces – Docker 1.10 • Secure computing (libseccomp) – Docker 1.10 • --pids-limit (cgroups pid limitation) – Docker 1.11 (kernel 4.3 +) • cgroups “v2” – (kernel 4.5+) • --no-new-privileges (limit process escalation) – Docker 1.11 • Storage driver quotas (limited) – Docker 1.12 • Secure by default multi-node orchestration (mutual TLS) – Docker 1.12 • 2017 • Improved resource isolation features in Linux Kernel and in Docker engine 18
  • 19.
    Whats Next inContainer Security? • Host Security • Linux Kit • Lightweight virtualization (Katacontainers (formerly Intel clear containers) ) • Unikernels (MirageOS) • Content Security • Distribution Specific Project (OCI) • Project Grafeas 19
  • 20.
    Linux Kit • Secure,portable OSes for Linux containers • Support Kubernetes, AWS, GCP, Azure etc • https://github.com/linuxkit/linuxkit 20
  • 21.
    Katacontainers • Based onIntel Clear Containers • Perform like containers but provide the workload isolation and security advantages of VMs • OCI compliant • https://github.com/kata-containers/ 21
  • 22.
    MirageOS • Library operatingsystem for unikernels • MirageOS3.0 released in Feb 2017 • https://mirage.io/ 22
  • 23.
    Distribution Specific Project •Image and run-time formats through OCI • Image distribution is now defacto Docker Image Registry API • https://www.opencontainers.org/announcement/2018/04/09/oci- announces-dist-spec-project 23
  • 24.
    Project Grafeas • Standardizecontainer vulnerability format • Standardize attestations • https://grafeas.io/ 24
  • 25.
    Conclusion • Container securityhas significantly evolved from early days of Docker • Secure-by-default in all major platforms • For users, security focus is on securing software supply chain • New features promise standardization, light-weight, VM isolation 25