In an ideal world, you would write Python code and then it would work perfectly. But unfortunately, it doesn't work in this manner. In my talk, I'll cover how to efficiently debug your programs, especially in cloud environments or inside Kubernetes.

1. Debugging
Python apps
inside k8s

2. andrii@hachyderm.io
# cat ./misc/about-me.yaml
---
name: Andrii Soldatenko ⎈
work: Dynatrace
about:
- pythonista: and OSS contributor
- fan: of debuggers
- speaker: At many conferences
links:
- https://asoldatenko.org/about-me/

3. andrii@hachyderm.io

4. HowcanI debugPython
program?
andrii@hachyderm.io

5. viaprints> ?

6. logs
import logging
…
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')

7. logs
2024-04-18 13:04:04,653 - simple_example -
2024-04-18 13:04:04,653 - simple_example - DEBUG
INFO -
- debug message
info message
2024-04-18 13:04:04,653 - simple_example -
2024-04-18 13:04:04,653 - simple_example - WARNING - warn message
ERROR - error message
2024-04-18 13:04:04,653 - simple_example - CRITICAL - critical message

8. viaREPL?
andrii@hachyderm.io
(main, Mar 14 2024, 15:09:12)…
"copyright", "credits" or "license" for more
Python 3.12.2
Type "help",
information.
>>>

9. viaREPL?
andrii@hachyderm.io
(main, Mar 14 2024, 15:09:12)…
"copyright", "credits" or "license" for more
Python 3.12.2
Type "help",
information.
>>>

10. viaREPL?
andrii@hachyderm.io
$ python -i cycle_length.py
>>> cycle_length
<functools._lru_cache_wrapper
>>> pprint.pprint({l for l in
{'cycle_length', 'functools',
>>>
object at
locals()
'pprint',
0x104328bf0>
if not l.startswith(' ')})
'sys'}

11. usingduckdebugging?

12. Usingdebuggers!

14. DAP

15. Pythondebuggers

16. Debugging ondevenv
from enum import Enum
from fastapi import FastAPI
...
ModelName):
@app.get("/models/{model_name}")
async def get_model(model_name:
import pdb; pdb.set_trace()
if model_name is ModelName.alexnet:
return {"model_name": model_name,
if model_name.value == "lenet":
return {"model_name": model_name,
"message": "Deep Learning FTW!"}
return {"model_name": model_name, "message": "Have some
"message": "LeCNN all the images"}
residuals"}

17. from enum import Enum
from fastapi import FastAPI
...
import pdb; pdb.set_trace()
"Deep Learning FTW!"}
return {"model_name": model_name, "message": "Have some
"LeCNN all the images"}
residuals"}
Debugging ondevenv
@app.get("/models/{model_name}")
async def get_model(model_name: ModelName):
import pdb; pdb.set_trace()
if model_name is ModelName.alexnet:
return {"model_name": model_name, "message":
if model_name.value == "lenet":
return {"model_name": model_name, "message":

18. Debugging ondevenv
INFO: 127.0.0.1:51061 - "GET /openapi.json HTTP/1.1" 200 OK
> /Users/andrii/work/debugging-containerized-python-applications/main.py(27)get_model()
-> if model_name is ModelName.alexnet:
(Pdb) l
22
23
24 @app.get("/models/{model_name}")
25 async def get_model(model_name: ModelName):
26 import pdb; pdb.set_trace()
if model_name is ModelName.alexnet:
return {"model_name": model_name,
27 ->
28 "message": "Deep Learning FTW!"}
29
30 if model_name.value == "lenet":
31 return {"model_name": model_name, "message": "LeCNN all the images"}
32
(Pdb) n

19. You buildityouownit!

20. DOCKERFILE
# temp stage
FROM python:3.12.2-slim
WORKDIR /app
as builder
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN apt-get update &&
apt-get install -y --no-install-recommends gcc
COPY requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /app/wheels -r requirements.txt
# final stage
FROM python:3.12.2-slim
WORKDIR /app
COPY --from=builder /app/wheels /wheels
COPY --from=builder /app/requirements.txt .
RUN pip install --no-cache /wheels/*
COPY main.py .
RUN ["uvicorn", "main:app", "--port", "8000", "--host", "0.0.0.0"]

21. Define aproblem

22. Debugging containerizedpython
apps

23. Remotedebugging
containerizedpythonapps
andrii@hachyderm.io

24. VSCODE

25. Step1:Addattachconfig
$ cat .vscode/launch.json
{
"name": "Python Debugger: Attach",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5678
}
}

26. Step2:AdjustDockerfile
FROM python:3.12-slim-bookworm
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN pip install --no-cache-dir
EXPOSE 5678 8000
-r requirements.txt
CMD ["python", "-m", "debugpy", "--wait-for-client", "--listen", "0.0.0.0:5678", "-m",
"uvicorn", "main:app", "--port", "8000", "--host", "0.0.0.0"]

27. andrii@hachyderm.io
Step3:Build
docker build -f Dockerfile-debugpy -t asoldatenko/py-sample-app-debug .
python:3.12-slim-bookworm
1/6: FROM
STEP 2/6: WORKDIR /usr/src/app
--> Using cache abcb4b08ab5f9df1a0e10bfa0dec85461cb6524d829ca61045310b8241c10022
--> abcb4b08ab5f
STEP 3/6: COPY . /usr/src/app
--> 8c9b8a5ce997
...
Successfully tagged localhost/asoldatenko/py-sample-app-debug:latest
875242325d5b07fbc223a33832df6bdf625dda750464d1be5fa088dd0807d0b0

28. andrii@hachyderm.io
Step4:Run
docker run -p 8000:8000 -p 5678:5678 asoldatenko/py-sample-app-debug
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to
disable this validation.

29. andrii@hachyderm.io
Step5:Connect

30. PDB CLONE Doesn’t work
with 3.12
def main():
import pdb_clone.pdb as pdb
pdb.set_trace_remote(host=b'127.0.0.1', port=7935, frame=None)
a = 2
b = 3
c = a + b
$ python samples/pdb_clone_example.py

31. PDB CLONE
$ telnet localhost
Trying ::1...
…
7935
PROCESS_PID:43030
PROCESS_NAME:samples/pdb_clone_example.py
> /users/andrii.soldatenko/work/debugging-containerized-python-applications/samples/
pdb_clone_example.py(4)main()
-> a = 2
(Pdb) n
> /users/andrii.soldatenko/work/debugging-containerized-python-applications/samples/
pdb_clone_example.py(5)main()
-> b = 3

32. PDB CLONE
-> b = 3
(Pdb) l
1 def main():
2 import pdb_clone.pdb as pdb
3 pdb.set_trace_remote(host=b'127.0.0.1',
a = 2
b = 3
c = a + b
port=7935, frame=None)
4
5 ->
6
7
8 main()
[EOF]
(Pdb)

33. PuDB

34. PuDB
FROM python:3.12-slim-bookworm
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app
COPY ./samples/pydb_example.py /usr/src/app
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install pudb
EXPOSE 6899
CMD ["python", "pydb_example.py"]

35. PuDB
def main():
from pudb.remote import
set_trace(term_size=(80,
set_trace
24), host="0.0.0.0")
a = 1
b = 2
c = a + b
if name == " main ":
main()

36. PuDB
def main():
from pudb.remote import
set_trace(term_size=(80,
set_trace
24), host="0.0.0.0")
a = 1
b = 2
c = a + b
if name == " main ":
main()

37. PuDB
python samples/pydb_example.py
pudb:6899: Please start a telnet session
telnet 0.0.0.0 6899
pudb:6899: Waiting for client...
using a command like:

38. PuDB: telnetlocalhost6899

39. PuDB: telnetlocalhost6899

40. andrii@hachyderm.io

41. docker-entrypoint.sh
https://github.com/ashleymcnamara/gophers
#!/bin/sh
...
if [ "$REMOTE_DEBUG_PORT" ]; then
echo "Starting application with remote
debugFlags="--wait-for-client --listen
main:app --port 8000 --host 0.0.0.0"
debugging on port $REMOTE_DEBUG_PORT"
0.0.0.0:$REMOTE_DEBUG_PORT -m uvicorn
# Simply setting this environment variable is enough to force the debugger to
# pause on start --- we don't care about the value.
cmd="python -m debugpy $debugFlags"
fi
…

42. docker-entrypoint.sh
#!/bin/sh
...
if [ "$REMOTE_DEBUG_PORT" ]; then
debugging on port $REMOTE_DEBUG_PORT"
0.0.0.0:$REMOTE_DEBUG_PORT -m uvicorn
echo "Starting application with remote
debugFlags="--wait-for-client --listen
main:app --port 8000 --host 0.0.0.0"
# Simply setting this environment variable is enough to force the debugger to
# pause on start --- we don't care about the value.
cmd="python -m debugpy $debugFlags"
fi
…

43. FinalDockerfile
FROM python:3.12-slim-bookworm
WORKDIR /usr/src/app
COPY . /usr/src/app
COPY docker-entrypoint.sh /entrypoint
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5678 8000
ENTRYPOINT ["/entrypoint"]

44. kubectlapply-f ./config/sample-
pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: sample-app
namespace: demo
spec:
containers:
- name: sample-debug
image: asoldatenko/py-sample-app-debug
env:
- name: REMOTE_DEBUG_PORT
value: "5678"

45. LIVE DEMO B
./start.sh

46. kubectl-ndemoport-forward
sample-app5678:5678

48. TIP: Debuggingwithan
ephemeraldebugcontainer
kubectl debug -it -n demo
sample-app --target=sample-debug
—image=busybox
--share-processes -- sh

49. TIP: Debuggingusingacopyof
POD
kubectl debug -it -n demo
sample-app --copy-to=sample-debug
—image=asoldatenko/debug:0.0.2
--share-processes -- sh
NAME
sample-app
sample-debug
3m43s
READY
1/1
2/2
STATUS
Running
Running
RESTARTS
2 (40m ago)
0
AGE
43m

50. Burngit commitSHA intodocker
image
FROM python:3.12-slim-bookworm
WORKDIR /usr/src/app
COPY . /usr/src/app
COPY docker-entrypoint.sh /entrypoint
ARG GIT_HASH
ENV GIT_HASH=${GIT_HASH:-dev}
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5678 8000
ENTRYPOINT ["/entrypoint"]

51. Burngit commitSHA intodocker
image
docker build -f Dockerfile-debugpy-sha --build-arg GIT_HASH=${GIT_HASH::7} -t
asoldatenko/py-sample-app-debug .
docker run --rm asoldatenko/py-sample-app-debug env | grep GIT_HASH
GIT_HASH=80907a7

52. K9s

53. K9s: plugin.yml
plugin:
a plugin to provide a `ctrl-l` shortcut to tail the logs while
# Defines
in pod view
#--- Create debug container for selected pod in current namespace
# See https://kubernetes.io/docs/tasks/debug/debug-application/debug-
running-pod/#ephemeral-container
debug:
shortCut: Ctrl-H
description: Add debug container
scopes:
- containers
command: bash
background: false
confirm: true
args:
- -c
- "kubectl debug -it -n=$NAMESPACE $POD --target=$NAME --
image=nicolaka/netshoot:v0.11 --share-processes -- bash"

54. Summary
The debugger doesn't fix your bugs, it just runs them in slow motion.
Don’t forget to add some details to your python apps inside images
Of course, you have to scale down traffic to 1 instance to avoid misses of
requests to your pod
Use debugger as much as possible
Automate you rotine with hotkeys/plugins

55. FutureReading:
https://github.com/andriisoldatenko/debug
https://github.com/andriisoldatenko/debugging-containerized-go-applications
https://github.com/andriisoldatenko/debugging-containerized-python-applications

56. Questions

57. Slides:
andrii@hachyderm.io