Tools and Techniques to Kubernetes Network debugging easier

1. Kubernetes Network Debugging
Tools and Techniques to make one of the most frustrating
parts of Kubernetes easier
Konrad F. Heimel, 2023-10-19 1

2. Agenda
mirrord
Transfer your IDE into the Kubernetes cluster
ephemeral containers
Using debug tools without bloating your images
inspector gadget
Cloud-native debugging using eBPF
kubeshark
API traffic analyzer for Kubernetes
Konrad F. Heimel, 2023-10-19 2

3. Containers and Networking Challenges for
Developers
Consistency in Containers
Application + Dependencies = Consistent file environment.
Networking Issues
Containers don't ensure networking consistency.
Network differences Main source of errors.
Kubernetes Locally?
Replicating full Kubernetes = Resource intensive & confusing.
Not feasible for every developer.
Konrad F. Heimel, 2023-10-19 3

4. What is mirrord?
Connects a local process to your Kubernetes cluster.
Comes with CLI & plugins for IntelliJ and VS Code.
Debug in the cloud, without deploying.
Test locally in cloud conditions:
Without local deployment
Without CI/CD
Without deploying untested code
Konrad F. Heimel, 2023-10-19 4

5. 🎥Live Demo
Konrad F. Heimel, 2023-10-19 5

6. How does it work?
1. Creates a mirrord-agent in the cluster:
Clones/steals & forwards traffic
2. Overrides local process' syscalls to:
Listen to agent's incoming traffic.
Send out traffic from remote pod.
Access remote file system.
Merge pod's environment with local.
Konrad F. Heimel, 2023-10-19 6

7. Language/Framework Support
Hooks libc , supporting:
Rust
Node
Python
Java
Kotlin
Ruby
... and others!
Also supports Go, not using libc .
Konrad F. Heimel, 2023-10-19 7

8. Installation on Cluster?
Nothing persistent.
Short-lived pod/container for
proxy.
Only needs kubectl
configured.
Incompatible with Pod
Security Standards.
apiVersion: v1
kind: Pod
metadata:
name: mirrord-agent-lgfcl4ujer-mxbgp
spec:
containers:
- image: app:1.0.0
name: greenfield
ephemeralContainers:
- command:
- ./mirrord-agent
- -l
- "49332"
- -e
image: ghcr.io/metalbear-co/mirrord:3.56.1
imagePullPolicy: IfNotPresent
name: mirrord-agent-mszkpupjeb
securityContext:
capabilities:
add:
- SYS_ADMIN
- SYS_PTRACE
- NET_RAW
- NET_ADMIN
runAsGroup: 44448
Konrad F. Heimel, 2023-10-19 8

9. Configuration
target : Pod/group you connect to.
env : Merge pod's and local's environment.
networking.mode :
steal : Capture incoming
mirror : Sniff & forward a copy
fs.mode :
localwithoverrides : Read Kubelet-
generated files.
{
"kube_context": "my-cluster",
"accept_invalid_certificates": false,
"target": {
"path": "deploy/spring-demo-chart",
"namespace": "mirrord-demo"
},
"feature": {
"network": {
"incoming": {
"mode": "mirror",
"outgoing": true
},
"dns": true
},
"fs": {
"mode": "localwithoverrides"
},
"env": true
},
"telemetry": false
}
Konrad F. Heimel, 2023-10-19 9

10. Advantages of mirrord
Mirrors traffic ensuring safety.
Flexibly manage traffic and file operations.
Superior to local clusters: Handles complex
environments.
No installation of infrastructure on cluster required.
No cluster deployments: Stable code remains.
Connects specific services to the cloud.
Konrad F. Heimel, 2023-10-19 10

11. mirrord vs. Telepresence
Process-level operation (no
daemons).
Run multiple services concurrently.
No cluster installation needed.
Duplicates traffic by default.
IDE extensions available!
vs
Konrad F. Heimel, 2023-10-19 11

12. Introduction to Ephemeral Containers
Not part of the pod's spec, but born as needed.
Share namespaces with containers in the same pod.
Powerful for understanding & diagnosing app behavior.
Stable since Kubernetes v1.25
Once created, they can't be deleted.
Konrad F. Heimel, 2023-10-19 12

13. Using Ephemeral Containers for Debugging
Inspect a running pod without halting its operation.
Don't need tools outside your production image.
Communicate over localhost, use IPC, inspect processes, and access shared
volumes.
Debug command example:
k debug --image=nicolaka/netshoot -it -- /bin/bash
Konrad F. Heimel, 2023-10-19 13

14. Basic Network Debugging Commands
1. ping <hostname/IP> - Check network connectivity.
2. netstat -tuln - Display listening ports.
3. nslookup <hostname> - DNS lookup.
4. traceroute <hostname/IP> - Trace network path.
5. ifconfig or ip a - Display network interfaces.
6. nc -zv <hostname/IP> <port> - Check if port is open.
Konrad F. Heimel, 2023-10-19 14

15. Kubernetes and Linux Namespaces
Linux namespaces provide isolated
environments with unique resources
The container runtime leveraged by
Kubernetes creates a new set of
namespaces for each pod, ensuring
isolated network, IPC, UTS, and PID
environments.
Sidecar containers, deployed in the
same pod as the primary container,
share most namespaces, enabling
inter-container communication.
Network (net)
Inter-process Communication (ipc)
User ID(user)
Process ID(pid)
* spec.shareProcessNamespace: true
UNIX Time-Sharing(uts) UNIX Time-Sharing(uts)
Mount (mnt) Mount (mnt)
__ _ _______ _______ _______ __ __ _______ _______ _______
| | | || || || || | | || || || |
| |_| || ___||_ _|| _____|| |_| || _ || _ ||_ _|
| || |___ | | | |_____ | || | | || | | | | |
| _ || ___| | | |_____ || || |_| || |_| | | |
| | | || |___ | | _____| || _ || || | | |
|_| |__||_______| |___| |_______||__| |__||_______||_______| |___|
Konrad F. Heimel, 2023-10-19 15

16. 🎥Live Demo
Konrad F. Heimel, 2023-10-19 16

17. Collection of eBPF-based tools for Kubernetes apps.
Collects low-level kernel data.
Enriches with Kubernetes metadata.
Mechanism to deploy eBPF tools to Kubernetes clusters.
CLI tool ig for tracing containers.
Prometheus metrics endpoint.
Konrad F. Heimel, 2023-10-19 17

18. Linux kernel technology.
Restricted C subset programs.
Compiled to special bytecode.
Validated before kernel execution.
from __future__ import print_function
from bcc import BPF
from bcc.utils import printb
# load BPF program
b = BPF(text="""
TRACEPOINT_PROBE(random, urandom_read) {
// args is from /sys/kernel/debug/tracing/events/random/urandom_read/format
bpf_trace_printk("%dn", args->got_bits);
return 0;
}
""")
# header
print("%-18s %-16s %-6s %s" % ("TIME(s)", "COMM", "PID", "GOTBITS"))
# format output
while 1:
try:
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
except ValueError:
continue
except KeyboardInterrupt:
exit()
printb(b"%-18.9f %-16s %-6d %s" % (ts, task, pid, msg))
Konrad F. Heimel, 2023-10-19 18

19. eBPF Overview
Source: https://www.brendangregg.com/ebpf.html 19

20. Inspektor Gadget Overview
Provides a trace Custom Resource
Definition (CRD) for control.
Interaction through kubectl gadget
CLI.
Gadget pod has a Kubernetes
controller to perform CR actions.
eBPF program installation via tracers
from trace CRD.
eBPF: Inbuilt kernel VM allowing
userspace scripts in kernel space.
Konrad F. Heimel, 2023-10-19 20

21. 🎥Live Demo
Konrad F. Heimel, 2023-10-19 21

22. The Gadgets
Konrad F. Heimel, 2023-10-19 22

23. Installing Inspector Gadget
Install Inspector Gadget using Krew kubectl plugin manager:
$ kubectl krew install gadget
Deploy Inspector Gadget on Kubernetes:
$ kubectl gadget deploy
Creating Namespace/gadget...
...
Creating DaemonSet/gadget...
...
Inspektor Gadget successfully deployed
Konrad F. Heimel, 2023-10-19 23

24. Kubeshark – API Traffic Analyzer for Kubernetes
Think Wireshark re-invented for Kubernetes
Real-time, identity-aware, protocol-level visibility into K8s API traffic
Konrad F. Heimel, 2023-10-19 24

25. Kubeshark in Action
1. Network Analysis: Real-time protocol-level visibility & troubleshooting
2. Investigating Traffic: Filter and inspect pod-to-pod communication
3. Security Auditing: Identify suspicious traffic patterns & threats
4. Historic Traffic Analysis: Analyze past traffic snapshots
5. Connectivity Troubleshooting: Diagnose network errors & latency issues
Konrad F. Heimel, 2023-10-19 25

26. Konrad F. Heimel, 2023-10-19 26

27. Konrad F. Heimel, 2023-10-19 27

28. Kubeshark: Tracing Encrypted Traffic
How? Using eBPF to sniff encrypted TLS traffic without decryption.
Hooks into OpenSSL & Go's crypto/tls package.
Deploy Kubeshark with TLS Sniffing:
kubeshark tap --tls -n kh-demo
Supports OpenSSL & Go's crypto/tls.
Konrad F. Heimel, 2023-10-19 28

29. Further Resources
mirrord
inspector gadget
eBPF Basics
kubeshark Related Tools
Krew kubectl Plugin Manager
BPF Compiler Collection (BCC)
Konrad F. Heimel, 2023-10-19 29