How to debug Go application running on remote Kubernetes cluster? Slides from #GopherCon Israel 2019

1. Debugging
Go on Kubernetes
GopherCon 2019, Israel
@alexeiled

2. Intro
Alexei Ledenev
@alexeiled
alexei-led/pumba

3. Quick Survey
1. Who is using Docker today?
2. Who is using Kubernetes?

5. What is Kubernetes?

6. Kubernetes Objects
wordpress
service
210.14.128.31

8. Core Kubernetes

9. Debugging Go
🤔
👍
🚫

11. K8s Development Flow

12. K8s Debug Models
100% local 100% remote
Realism Low High
Feedback Fast Slow
Resource Footprint Heavy Lightweight

13. 100% Remote

14. Build image:x.x.x-debug
Go App Static Binary
(built with `go build --gcflag "-N -l" ...`)
Delve Binary (COPY --from builder OR go get)
EXPOSE 2345 (debugger port)
Base Image (alpine/scratch OR golang)
CMD ["dlv", "exec", "--headless", "--api-version=2",
"--listen=:2345", "--log", "./todo-app"]

15. Deploy image:x.x.x-debug
• Patch/Re-deploy K8s Deployment: image, SYS_PTRACE
• Expose Delve server port (kubectl port-forward)
• Rollback after debug session
Every
Commit

16. # build Docker image with delve debugger
docker build -t alexeiled/todo-app-web:debug -f debug/Dockerfile .
# push Docker image to registry
docker push alexeiled/todo-app-web:debug
# patch Kubernetes deployment and service
kubectl patch deploy todo-app --type="json" -ntodo-app
--patch "$(/bin/cat dbg/patch-debug.json)"
# forward debugger port to the local machine
kubectl port-forward pod/app-pod 2345
# DEBUG SESSION .......
# rollback the patch
kubectl patch deploy todo-app --type="merge"
--patch "$(/bin/cat k8s-deployment/todo-app-deployment.yaml)" -ntodo-app

17. Local + Remote

18. Telepresence (Datawire 👍)
Debug
Locally
Instant
Feedback
Realistic
Environment
Minimal
Resource
Footprint
Use your
own tools

19. reverse tunnel (sshd) over a
kubectl port-forward

20. Telepresence Demo

21. # open VS Code editor
code .
# [code.terminal.#1]: swap todo-app deployment with bash
telepresence --namespace=todo-app --swap-deployment=todo-app
--expose 3000 --run bash
# [code.terminal.#1] run debugger (with .debug/runDelve.sh)
dlv debug --listen=localhost:2345
--headless=true --api-version=2 --log=true ./
-- -config-file="$TELEPRESENCE_ROOT/etc/todo-app/redis.config"
# [code.terminal.#2] invoke todo-app service in K8s cluster
http http://a8d6ad3e6169111e98c1d02b9ca3b647-624533560.us-
west-2.elb.amazonaws.com:9080/read/todo
# DEBUG SESSION ....... stop debug (dlv); exit (swap back)

22. Debug Go in K8s
https://telepresence.io
TODO: check this too

23. @alexeiled