A short, introductory talk to the world of debuggers. During the talk, we write a simple debugger application in Rust. Video at: https://www.youtube.com/watch?v=qS51kIHWARM

1. Let’s write a Debugger!
Levente Kurusa <levex@linux.com>
Imperial College London
linux.conf.au 2018, Sydney, Australia January 25, 2018

2. Who am I?
• Final year undergraduate at Imperial College London
• Previously at Apple and Red Hat
• Mostly working on operating systems and low-level
performance
• Writes in C, Haskell, Rust, Go.
1

3. History of debuggers

4. Single user machines
• One of the first computers
in the world
• Small application was loaded
at the top of the memory
• single step
• examine registers
• read/write memory
TX-0 at MIT
2

5. Batch processing machines
3

6. Batch processing machines
Debugged by putting macro call in the punch card and generating:
• Snapshots (register dump)
• Core dumps (contents of memory)
3

7. printf
Then came CTSS (Compatible Time-Sharing System), one of the
first time-sharing operating systems!
Debugging suddenly became interactive.
printf-debugging
*ptr = 1337;
printf("Did we crash at line %d?n", __LINE__);
*((int *) 0) = 1337;
printf("Did we crash at line %d?n", __LINE__);
4

8. Unix-es
• The first version of UNIX had a debugger called, DB
• GNU had GDB and LLDB
• For Plan 9, ADB was created
These debuggers should be familiar!
5

9. Tracing processes

10. ptrace
Most debuggers heavily rely on a system call known as ptrace.
The prototype of ptrace(2)
#include <sys/ptrace.h>
long ptrace(enum __ptrace_request request, pid_t pid,
void *addr, void *data);
6

11. Signals
How does int a = 3, b = 0, c = a / b result in a SIGFPE?
7

12. Signals
How does int a = 3, b = 0, c = a / b result in a SIGFPE?
1. Division by zero is noticed by the CPU
7

13. Signals
How does int a = 3, b = 0, c = a / b result in a SIGFPE?
1. Division by zero is noticed by the CPU
2. The CPU raises a divide-by-zero error (#DE)
7

14. Signals
How does int a = 3, b = 0, c = a / b result in a SIGFPE?
1. Division by zero is noticed by the CPU
2. The CPU raises a divide-by-zero error (#DE)
3. A handler in the kernel is eventually called
7

15. Signals
How does int a = 3, b = 0, c = a / b result in a SIGFPE?
1. Division by zero is noticed by the CPU
2. The CPU raises a divide-by-zero error (#DE)
3. A handler in the kernel is eventually called
4. The kernel sends a SIGFPE to the offending process
7

16. Signals
How does int a = 3, b = 0, c = a / b result in a SIGFPE?
1. Division by zero is noticed by the CPU
2. The CPU raises a divide-by-zero error (#DE)
3. A handler in the kernel is eventually called
4. The kernel sends a SIGFPE to the offending process
5. Your signal handler is called (or not if it is SIGKILL)
7

17. Implementation
• Enable tracing
• Run until system call
• Monitoring registers
• Single stepping
• Memory manipulation
8

18. Implementation
• Enable tracing
• Run until system call
• Monitoring registers
• Single stepping
• Memory manipulation
8

19. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• Monitoring registers
• Single stepping
• Memory manipulation
8

20. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• Monitoring registers
• Single stepping
• Memory manipulation
8

21. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• PTRACE SYSCALL
• PTRACE SYSEMU
• Monitoring registers
• Single stepping
• Memory manipulation
8

22. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• PTRACE SYSCALL
• PTRACE SYSEMU
• Monitoring registers
• Single stepping
• Memory manipulation
8

23. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• PTRACE SYSCALL
• PTRACE SYSEMU
• Monitoring registers
• PTRACE PEEKUSER / PTRACE POKEUSER
• PTRACE GETREGS / PTRACE SETREGS
• Single stepping
• Memory manipulation
8

24. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• PTRACE SYSCALL
• PTRACE SYSEMU
• Monitoring registers
• PTRACE PEEKUSER / PTRACE POKEUSER
• PTRACE GETREGS / PTRACE SETREGS
• Single stepping
• Memory manipulation
8

25. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• PTRACE SYSCALL
• PTRACE SYSEMU
• Monitoring registers
• PTRACE PEEKUSER / PTRACE POKEUSER
• PTRACE GETREGS / PTRACE SETREGS
• Single stepping
• PTRACE SINGLESTEP
• Memory manipulation
8

26. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• PTRACE SYSCALL
• PTRACE SYSEMU
• Monitoring registers
• PTRACE PEEKUSER / PTRACE POKEUSER
• PTRACE GETREGS / PTRACE SETREGS
• Single stepping
• PTRACE SINGLESTEP
• Memory manipulation
8

27. Implementation
• Enable tracing
• PTRACE TRACEME
• Run until system call
• PTRACE SYSCALL
• PTRACE SYSEMU
• Monitoring registers
• PTRACE PEEKUSER / PTRACE POKEUSER
• PTRACE GETREGS / PTRACE SETREGS
• Single stepping
• PTRACE SINGLESTEP
• Memory manipulation
• PTRACE PEEKTEXT / PTRACE POKETEXT
• PTRACE PEEKDATA / PTRACE POKEDATA
8

28. Architectural support

29. Interrupting a process
PTRACE SINGLESTEP
9

30. Interrupting a process
PTRACE SINGLESTEP
%EFLAGS
9

31. Interrupting a process
PTRACE SINGLESTEP
%EFLAGS.TF
9

32. Interrupting a process
PTRACE SINGLESTEP
Trap flag (Sometimes referred to as ”Trace flag”)
9

33. Interrupting a process
PTRACE SINGLESTEP
Trap flag (Sometimes referred to as ”Trace flag”)
• After each instruction, trap into #DB interrupt
• The kernel delivers a SIGTRAP
• This fact is delivered via a wait-event to the debugger
9

34. Interrupting a process
Breakpoints
10

35. Interrupting a process
Breakpoints
• ud2 (machine code: 0x0F 0x0B)
• Triggers #UD – Undefined instruction exception
10

36. Interrupting a process
Breakpoints
• ud2 (machine code: 0x0F 0x0B)
• Triggers #UD – Undefined instruction exception
• int $3 (machine code: 0xCC)
• Triggers #BP – Breakpoint exception
10

37. Breakpoint implementation
• Replace with ”int $3”
11

38. Breakpoint implementation
• Replace with ”int $3”
• Take note of the previous instruction
11

39. Breakpoint implementation
• Replace with ”int $3”
• Take note of the previous instruction
• When the breakpoint is hit, replace with the previous
instruction
11

40. Breakpoint implementation
• Replace with ”int $3”
• Take note of the previous instruction
• When the breakpoint is hit, replace with the previous
instruction
• Try executing the instruction again
Nota bene:
• Could have used ”ud2”
11

41. Debug registers
• DR0 - DR3: Linear addresses
12

42. Debug registers
• DR0 - DR3: Linear addresses
• DR6: Debug control
Contains bitmasks for:
• 1, 2, 4 or 8 bytes monitored
• Break on read, write, execute, or read+write
12

43. Debug registers
• DR0 - DR3: Linear addresses
• DR6: Debug control
Contains bitmasks for:
• 1, 2, 4 or 8 bytes monitored
• Break on read, write, execute, or read+write
• DR7: Debug status
Bitmask showing which of DR0-DR3 triggered the #DB
12

44. Debug registers
• DR0 - DR3: Linear addresses
• DR6: Debug control
Contains bitmasks for:
• 1, 2, 4 or 8 bytes monitored
• Break on read, write, execute, or read+write
• DR7: Debug status
Bitmask showing which of DR0-DR3 triggered the #DB
• DR4 & DR5: Obsolete aliases to DR6 & DR7
12

45. Thanks!
Thank you for your attention!
Twitter: @iLevex Email: <levex@linux.com>
GitHub: levex Website: http://osdev.me/
The LATEX theme is available at github.com/matze/mtheme
Both the theme and the talk are licensed under the CC-BY-SA 4.0
International license.
13