Presentation about two tools: - mirrord Transfer your IDE into the - inspector gadget Cloud-native debugging using eBPF

1. Kubernetes Debugging Tools
mirrord and Inspector Gadget
Konrad F. Heimel, 2023-08-17 1

2. Agenda
mirrord
Transfer your IDE into the Kubernetes cluster
inspector gadget
Cloud-native debugging using eBPF
Konrad F. Heimel, 2023-08-17 2

3. What is mirrord?
Connects a local process to your Kubernetes cluster.
Comes with CLI & plugins for IntelliJ and VS Code.
Debug in the cloud, without deploying.
Test locally in cloud conditions:
Without local deployment
Without CI/CD
Without deploying untested code
Konrad F. Heimel, 2023-08-17 3

4. 🎥Live Demo
Konrad F. Heimel, 2023-08-17 4

5. How does it work?
1. Creates a mirrord-agent in the cluster:
Clones/steals & forwards traffic
2. Overrides local process' syscalls to:
Listen to agent's incoming traffic.
Send out traffic from remote pod.
Access remote file system.
Merge pod's environment with local.
Konrad F. Heimel, 2023-08-17 5

6. Language/Framework Support
Hooks libc , supporting:
Rust
Node
Python
Java
Kotlin
Ruby
... and others!
Also supports Go, not using libc .
Konrad F. Heimel, 2023-08-17 6

7. Installation on Cluster?
Nothing persistent.
Short-lived pod/container for proxy.
Only needs kubectl configured.
Incompatible with Pod Security
Standards.
apiVersion: v1
kind: Pod
metadata:
name: mirrord-agent-lgfcl4ujer-mxbgp
spec:
containers:
- image: ghcr.io/metalbear-co/mirrord:3.56.1
name: mirrord-agent
securityContext:
capabilities:
add:
- SYS_ADMIN
- SYS_PTRACE
- NET_RAW
- NET_ADMIN
runAsGroup: 7318
volumeMounts:
- mountPath: /host/run
name: hostrun
- mountPath: /host/var
name: hostvar
hostPID: true
volumes:
- hostPath:
path: /run
name: hostrun
- hostPath:
path: /var
name: hostvar
Konrad F. Heimel, 2023-08-17 7

8. Configuration
target : Pod/group you connect to.
env : Merge pod's and local's environment.
networking.mode :
steal : Capture incoming
mirror : Sniff & forward a copy
fs.mode :
localwithoverrides : Read Kubelet-
generated files.
{
"kube_context": "my-cluster",
"accept_invalid_certificates": false,
"target": {
"path": "deploy/spring-demo-chart",
"namespace": "mirrord-demo"
},
"feature": {
"network": {
"incoming": {
"mode": "mirror",
"outgoing": true
},
"dns": true
},
"fs": {
"mode": "localwithoverrides"
},
"env": true
},
"telemetry": false
}
Konrad F. Heimel, 2023-08-17 8

9. Advantages of mirrord
Mirrors traffic ensuring safety.
Flexibly manage traffic and file operations.
Superior to local clusters: Handles complex
environments.
No installation of infrastructure on cluster required.
No cluster deployments: Stable code remains.
Connects specific services to the cloud.
Konrad F. Heimel, 2023-08-17 9

10. mirrord vs. Telepresence
Process-level operation (no
daemons).
Run multiple services concurrently.
No cluster installation needed.
Duplicates traffic by default.
IDE extensions available!
vs
Konrad F. Heimel, 2023-08-17 10

11. Collection of eBPF-based tools for Kubernetes apps.
Collects low-level kernel data.
Enriches with Kubernetes metadata.
Mechanism to deploy eBPF tools to Kubernetes clusters.
CLI tool ig for tracing containers.
Prometheus metrics endpoint.
Konrad F. Heimel, 2023-08-17 11

12. Linux kernel technology.
Restricted C subset programs.
Compiled to special bytecode.
Validated before kernel execution.
from __future__ import print_function
from bcc import BPF
from bcc.utils import printb
# load BPF program
b = BPF(text="""
TRACEPOINT_PROBE(random, urandom_read) {
// args is from /sys/kernel/debug/tracing/events/random/urandom_read/format
bpf_trace_printk("%dn", args->got_bits);
return 0;
}
""")
# header
print("%-18s %-16s %-6s %s" % ("TIME(s)", "COMM", "PID", "GOTBITS"))
# format output
while 1:
try:
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
except ValueError:
continue
except KeyboardInterrupt:
exit()
printb(b"%-18.9f %-16s %-6d %s" % (ts, task, pid, msg))
Konrad F. Heimel, 2023-08-17 12

13. eBPF Overview
Source: https://www.brendangregg.com/ebpf.html 13

14. Inspektor Gadget Overview
Provides a trace Custom Resource
Definition (CRD) for control.
Interaction through kubectl gadget
CLI.
Gadget pod has a Kubernetes
controller to perform CR actions.
eBPF program installation via tracers
from trace CRD.
eBPF: Inbuilt kernel VM allowing
userspace scripts in kernel space.
Konrad F. Heimel, 2023-08-17 14

15. 🎥Live Demo
Konrad F. Heimel, 2023-08-17 15

16. The Gadgets
Konrad F. Heimel, 2023-08-17 16

17. Installing Inspector Gadget
Install Inspector Gadget using Krew kubectl plugin manager:
$ kubectl krew install gadget
Deploy Inspector Gadget on Kubernetes:
$ kubectl gadget deploy
Creating Namespace/gadget...
...
Creating DaemonSet/gadget...
...
Inspektor Gadget successfully deployed
Konrad F. Heimel, 2023-08-17 17

18. Further Resources
mirrord
inspector gadget
eBPF Basics
Related Tools
Krew kubectl Plugin Manager
BPF Compiler Collection (BCC)
Konrad F. Heimel, 2023-08-17 18