Presented at: Open Source 101 at Home Presented by: Veer Muchandi, Red Hat Inc Abstract: While everyone wants to do Containers and Kubernetes, they don’t know what they are getting into from Security perspective. This session intends to take you from “I don’t know what I don’t know” to “I know what I don’t know”. This helps you to make informed choices on Application Security. Kubernetes as a Container Platform is becoming a de facto for every enterprise. In my interactions with enterprises adopting container platform, I come across common questions: - How does application security work on this platform? What all do I need to secure? - How do I implement security in pipelines? - What about vulnerabilities discovered at a later point in time? - What are newer technologies like Istio Service Mesh bring to table? In this session, I will be addressing these commonly asked questions that every enterprise trying to adopt an Enterprise Kubernetes Platform needs to know so that they can make informed decisions.

1. @OpenShift
RHOpenShift
Securing Applications on a
Container Platform
Presenter: Veer Muchandi
Title: Chief Architect, Container Solutions, NACS
Social Handle: @VeerMuchandi
Blogs: https://blog.openshift.com/author/veermuchandi/

2. Session Goals
Considerations for Security on a Container Platform
Containers - A quick look
Container Security at Host Operating System
Container Security at Build time
Container Security at Runtime
Container Platform (K8S Cluster) - Additional Security Features
Application Security

3. Understanding Containers

4. Common Misunderstanding

5. Right Way

6. Can malicious containers steal info?

7. Can bad processes choke others?

8. How can processes with different technologies
coexist?

9. Container separation
● Containers all share the same kernel
● Containers security is about controlling the access to the kernel
● Limit the ability of the container to negatively impact
○ host
○ other containers sharing host

10. Container Security
Importance of
Host OS
And Windows

11. Linux Containers Architecture

12. Isolation with Linux Namespaces
PID namespace
The first process is pid 1 in the container. Namespace destroyed as PID exits.
Cannot see/signal processes in parent pid namespace or other pid namespaces
Network namespace
Allows container to use separate virtual network stack, loopback device and process space
veth pairs are created to add interfaces from initial network namespace to container network namespace
Mount namespace
Isolate the set of file system mount points. Mount
sys directories as read only and nodev
Each process gets its own mount table i.e. view
of the filesystem
UTS namespace
isolate system identifiers – hostname and domain
name
I
IPC namespace
isolate certain interprocess communication (IPC)
resources
User namespace
Allow you to specify a range of host UIDs
dedicated to the container. A process can have
full root privileges for operations inside the
container mapped to non-root user on host.

13. Linux CGroups
Ensures that a single container cannot exhaust a large amount of system
resources
CGroups allocate CPU time, system memory, network bandwidth, or
combinations of these among user-defined groups of tasks
In a container:
# ls /sys/fs/cgroup/
blkio cpu cpuacct cpuacct,cpu cpuset devices freezer hugetlb memory net_cls
net_prio net_prio,net_cls perf_event pids systemd
# cat /sys/fs/cgroup/cpu/cpu.shares
2
# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712
Device CGroups
● specify which device nodes can be used within the container
● blocks the processes from creating and using device nodes that could be
used to attack the host.

14. 14
SELinux and Type Enforcement
● Implements Mandatory Access Control
● A LABELING system
● Every process has a LABEL. Every file, directory, and system object has a
LABEL
● Policy rules control access between labeled processes and labeled objects
● The kernel enforces the rules.
● Type Enforcement protects the host system from container processes
All container processes run with type svirt_lxc_net_t
Content within the container is labeled with type svirt_sandbox_file_t
svirt_lxc_net_t
● allowed to manage any content labeled svirt_sandbox_file_t
● able to read/execute most labels under /usr on the host
● not allowed to open/write to any other labels on the system
● not allowed to read any default labels in /var, /root, /home etc

15. MCS protects Containers from each other
Based on Multi Level Security (MLS)
Container Runtime picks out unique random MCS Label s0:c1,c2
Assigns MCS Label to all content
Launches the container processes with same label
Container Processes can only read/write their own files - Kernel enforces this
Multi Category Security (MCS) Enforcement

16. https://people.redhat.com/duffy/selinux/seli
nux-coloring-book_A4-Stapled.pdf

17. 17
CAP_SETPCAP
CAP_SYS_MODULE
CAP_SYS_RAWIO
CAP_SYS_PACCT
CAP_SYS_NICE
CAP_SYS_RESOURCE
CAP_SYS_TIME
CAP_SYS_TTY_CONFIG
CAP_AUDIT_WRITE
CAP_AUDIT_CONTROL
CAP_MAC_OVERRIDE
CAP_MAC_ADMIN
CAP_SYSLOG
CAP_NET_ADMIN
CAP_SYS_ADMIN
Modify process capabilities
Insert/Remove kernel modules
Modify Kernel Memory
Configure process accounting
Modify Priority of processes
Override Resource Limits
Modify the system clock
Configure tty devices
Write the audit log
Configure Audit Subsystem
Ignore Kernel MAC Policy
Configure MAC Configuration
Modify Kernel printk behaviour
Configure the network:
Catch all
- Setting the hostname/domainname
- mount(),unmount()
- nfsservctl
- ….
Linux Capabilities
Kernel 2.2 divides root privileges into 32 distinct capabilities (now 37)
Root user with all
capabilities is all
powerful!!

18. Drop Capabilities
● CHOWN the ability to make arbitrary
changes to file UIDs and GIDs
● DAC_OVERRIDE allows root to bypass
file read, write, and execute permission
checks
● FOWNER ability to bypass permission
checks on operations where the filesystem
process UID should match file UID
● FSETID override file owner and group
reqmts when setting setuid or setgid bits
on a file
● KILL root owned process can send kill
signals to non root processes
● SETGID and SETUID
● SETPCAP a process can change its
current capability set within its bounding
set
● NET_BIND_SERVICE you can bind to
privileged ports (e.g., those below 1024)
● NET_RAW allows a process to spy on
packets on its network
● SYS_CHROOT allows your processes to
chroot into a different rootfs
● MKNOD allows your processes to create
device nodes
● AUDIT_WRITE you can write a message
to kernel auditing log
● SET_FCAP allows you to set file
capabilities on a file system
14 Default capabilities available in a Docker Container
Do really think containers need these
default capabilities in production ??
A good strategy is to drop all capabilities and just add the
needed ones back.. Example
# podman run -d --cap-add SYS_TIME ntpd
Dockerfile
LABEL “io.containers.capabilities=SETUID,SETGID”

19. Secure Computing Mode - Seccomp
Filters system calls from a container to the kernel. Uses Berkeley Packet Filter
(BPF) system
Children to a container process will inherit the BPF filter
Provides more fine-grained control than capabilities, giving an attacker a limited
number of syscalls from the container
Restricted and allowed calls are arranged in profiles, pass different profiles to
different containers
Specify your own policy
# podman run --security-opt seccomp=/path/to/custom/profile.json <container>
Seccomp Filter generator: https://github.com/containers/oci-seccomp-bpf-hook

20. 20
Read Only Mounts
Linux kernel file systems have to be mounted in a container environment or
processes would fail to run
Fortunately, most of these file systems can be mounted as "read-only" on RHEL.
/sys
/proc/sys
/proc/sysrg-trigger
/proc/irq
/proc/bus
Some platforms (OpenShift) also block the ability of the privileged container
processes from re-mounting the file systems as read/write

21. Container Build Time Security

22. ● Red Hat Container
Registry
● Policies to control who
can deploy which
containers
● Certification Catalog
● Trusted content with
security updates
HOST OS
CONTAINER
OS
RUNTIME
APP
HOST OS
CONTAINER
OS
RUNTIME
APP
22
Image governance and private
registries
● What security meta-data is
available for your images?
● Are the images in the
registry updated regularly?
● Are there access controls
on the registry? How
strong are they?
Trusting Container Content

23. 23
Container Health Index
https://access.redhat.com/articles/2803031

24. Container Image Provenance
● Did the image come from a trusted source?
● Are you verifying image signatures?
● Are you signing container images?
$ podman image sign --sign-by foo@bar.com --directory /tmp/signatures
docker://privateregistry.example.com/foobar
Example: Image signing with Podman

25. Private Registries
Enterprise security policies may not allow your Container Images to be pushed to
a registry outside.
Or what if public registry (like DockerHub) is down?
Caching images in Private Registry
- Red Hat’s Quay
- JFrog Artifactory
- Docker Trusted Registry

26. 26
Restrict Registry Sources
$ oc edit images.config.openshift.io cluster
spec:
registrySources:
blockedRegistries:
- docker.io
Example: Restricting Container Registries in OpenShift
● Restrict pulling images from specific registries of your choice

27. Container Image Management Responsibilities
Operations/ Infrastructure admin
- Maintain Trusted OS Base Images (RHEL, UBI)
- Ensure these are good with Linux Kernel
Middleware Engineers/Architects
- Maintain Middleware Images
- Control build process - S2I, CICD
- Reference Architectures with Trusted middleware
Development teams
- Write code that layers on approved images
- Ensure code is clean, open source software vulnerabilities are handled

28. Scanning Containers

29. Container Scanning
Frequency
Scan containers as soon as they are created
Scan containers that get into enterprise registry
Ongoing basis- Identify any new vulnerabilities

30. 30
Scanning Containers at Creation
https://www.youtube.com/watch?v=65BnTLcDAJI
Git Repo Nexus
Sonar
Qube
Quay
Registry
Claire scan

31. Scanning Tools
Claire scanner with Red Hat’s Quay
BlackDuck
JFrog XRay
Twistlock
Sysdig

32. Additional Container Platform Security
Features

33. Automating Security Compliance for Infrastructure
OpenSCAP Compliance

34. 3434
Policy based Governance, Risk and Compliance
• Centrally set & enforce policies
for security, applications, &
infrastructure
• Quickly visualize detailed
auditing on configuration of
apps and clusters
• Built-in CIS compliance policies
and audit checks
• Immediate visibility into your
compliance posture based on
your defined standards
Don’t have to wait for your security team to tap you on the shoulder

35. •
–
–
•
–
–
–
•
•
…

37. In Kubernetes, containers in
pods run with the role
assigned to a special user
called Service Account

38. 38
Container Deployment Permissions with Pod Security
Policies (PSP) or with Security Context Constraints (SCC)

39. Securing Applications
(running as containers)

40. Microsegmentation with Network Policy Objects
● Lock everything by
default
● Add Network
Policies to allow
specific ingress
traffic

41. SSL at Ingress
Edge termination
Passthrough termination
Reencrypt

42. Specific IP Whitelists at Ingress
- Restrict access to a route to a select IP address(es)
- Annotate the route with the whitelisted/allowed IP addresses
- Connections from any other IPs are blocked
metadata:
annotations:
haproxy.router.openshift.io/ip_whitelist: 192.168.1.10 192.168.1.11

43. Istio Service Mesh on OpenShift
Connect, Manage, and Secure
Microservices, transparently
● Intelligent Routing
● Load Balancing
● Service Resilience
● Telemetry and Reporting
● Policy Enforcement
● Content based Filtering
(Layer 7)
● mTLS between services
● East-West traffic control

44. End to End Encryption with Istio Service mesh
Uses Service Account as
Identity. SPIFFE Id format
spiffe://<domain>/ns/<namespace>/sa/<serviceaccount>
Mutual TLS between sidecars
Istio CA
- Generate cert pair and
SPIFFE key for each SA
- Distribute key and cert pairs
- Rotate keys and certs
periodically
- Revoke key and cert when
need

45. Secrets to store sensitive data
kubernetesMasterConfig:
apiServerArguments:
experimental-encryption-provider-config:
- /path/to/encryption.config

46. Egress IP for all traffic from a Project

47. Egress Firewall to Limit Access
Cluster admin can limit the external addresses accessed by some or all pods
from within the cluster Examples:
A pod can talk to hosts (outside
OpenShift cluster) but cannot
connect to public internet
A pod can talk to public internet, but
cannot connect to hosts (outside
OpenShift cluster)
A pod cannot reach specific
subnets/hosts

48. API Management
SSL
Secrets
Connecting to external services
Summary: Application Security

49. Questions

50. @OpenShift
RHOpenShift
Thank you!!

57. 57
SELINUX - MAC - MCS - Process
system_u:system_r:container_runtime_t:s0
SElinux Policy module for the container
The OOTB SElinux policy container.te defines
what you can execute and access with the label
container_runtime_t
[root@osemaster ~]# ps -efZ | grep docker-containerd-shim-current
system_u:system_r:container_runtime_t:s0 root 3035 1479 0 Feb15 ? 00:00:01
/usr/bin/docker-containerd-shim-current
4d254785cbc6ee7aae8facc48555251e2385f65d89553b319b6324b1501e4b16
/var/run/docker/libcontainerd/4d254785cbc6ee7aae8facc48555251e2385f65d89553b319b6324b1501e4b16
/usr/libexec/docker/docker-runc-current

58. 58
SELINUX - MAC - MCS - Files
container_var_lib_t / svirt_sandbox_file_t
SElinux Policy module for the container
[root@osemaster ~]# ls -lZ
/var/lib/docker/containers/97de4217a04b6532e312cfb3e4638529aeb7dfa281a2cc067e092fcee82e6737
/
-rw-r-----. root root system_u:object_r:container_var_lib_t:s0
97de4217a04b6532e312cfb3e4638529aeb7dfa281a2cc067e092fcee82e6737-json.log
-rw-rw-rw-. root root system_u:object_r:container_var_lib_t:s0 config.v2.json
-rw-rw-rw-. root root system_u:object_r:container_var_lib_t:s0 hostconfig.json
-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0 hostname
-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0:c0,c1 hosts
-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0 resolv.conf
-rw-r--r--. root root system_u:object_r:container_var_lib_t:s0 resolv.conf.hash
drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c0,c1 secrets
drwx------. root root system_u:object_r:container_var_lib_t:s0 shm