Pods are the fundamental building block of Kubernetes applications. Since Pods are intended to be disposable and replaceable, you cannot add a container to a Pod once it has been created. Instead, you usually delete and replace Pods in a controlled fashion using deployments. Sometimes it's necessary to inspect the state of an existing Pod, however, for example to troubleshoot a hard-to-reproduce bug. In these cases you can run an ephemeral container in an existing Pod to inspect its state and run arbitrary commands. Ephemeral containers are useful for interactive troubleshooting when `kubectl exec` is insufficient because a container has crashed or a container image doesn't include debugging utilities. Faheem Memon, Principal Architect @ Amobee, will review the ephemeral containers feature (in beta since 1.23) and demonstrate potential practical implementations.

1. Debugging Live Apps in k8s
Faheem Memon

2. Current Troubleshooting Options
● Application logs
● Application metrics
● Pod description and status
● Pod events
● Pod shell aka exec
○ Requires debug tooling to be available in the containers
○ Sometimes requires higher tooling
○ Harder to control through governance

3. Ephemeral Containers
● Motivation
○ Minimal images lower operational burden and reduce
attack vectors.
○ Immutable images improve correctness and reliability.
○ Smaller image size reduces resource usage and speeds
deployments.
● Graduated to beta with v1.23
○ alpha since v1.16)
● Adds new containers to running pods
● Follow ContainerSpec
○ Ephemeral containers may not have ports
○ Lack guarantees for resources or execution
○ Never restarted

4. Ephemeral Containers Play Nice
● Allow access to namespaces and the file systems
of individual containers
● Fetch container images at run time rather than at
the time of pod or image creation
● Respect admission controllers and audit logging
● Be discoverable via the API
● Support arbitrary runtimes via the CRI (possibly
with reduced feature set)
● Require no administrative access to the node
● Have no inherent side effects to the running
container image
● Define a v1.Container available for inspection by
admission controllers

5. Kubectl debug
# Create an interactive debugging session in pod mypod and immediately attach to it.
kubectl debug mypod -it --image=busybox
# Create a debug container named debugger using a custom automated debugging image.
kubectl debug --image=myproj/debug-tools -c debugger mypod
# Create a copy of mypod adding a debug container and attach to it
kubectl debug mypod -it --image=busybox --copy-to=my-debugger
# Create a copy of mypod changing the command of mycontainer
kubectl debug mypod -it --copy-to=my-debugger --container=mycontainer -- sh
# Create a copy of mypod changing all container images to busybox
kubectl debug mypod --copy-to=my-debugger --set-image=*=busybox
# Create a copy of mypod adding a debug container and changing container images
kubectl debug mypod -it --copy-to=my-debugger --image=debian --set-image=app=app:debug,sidecar=sidecar:debug
# Create an interactive debugging session on a node and immediately attach to it
kubectl debug node/mynode -it --image=busybox

6. Potential Use Cases
● Live debugging for hard to find issues
○ Process start – init containers
○ Networking issues
○ File-system issues
○ Hard to replicate issues
● Run security and inventory scans
● Information gathering for tech support

7. Creating Ephemeral Containers
● Fetch a Pod object from the /pods resource.
● Modify spec.ephemeralContainers and write it back to the Pod's
/ephemeralcontainers subresource,
● The apiserver discards all changes except those to
spec.ephemeralContainers.
● The apiserver validates the update.
○ Pod validation fails if container spec contains fields disallowed
for Ephemeral Containers or the same name as a container in
the spec or EphemeralContainers.
○ Registered admission controllers receive an AdmissionReview
request containing the entire Pod.
● The kubelet's pod watcher notices the update and triggers a
syncPod().
● syncPod() finishes a regular sync, publishing an updated PodStatus
● The client performs an attach to the debug container's console.

8. Live Demo

9. References
● Proposal -
https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/277-
ephemeral-containers/README.md
● Debugging Runnign Pods - https://kubernetes.io/docs/tasks/debug/debug-
application/debug-running-pod
● Ephemeral Containers -
https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/
● Introduction Ephemeral Containers -
https://opensource.googleblog.com/2022/01/Introducing%20Ephemeral%20C
ontainers.html