1. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Porting Valgrind to OpenBSD and NetBSD
Masao Uebayashi
Tombi Inc.
Sep. 28, 2014

2. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Agenda
1. Prologue
2. Introduction
3. Development
4. Internal: System call
5. Internal: execve(2)
6. Internal: Signal
7. Issues
8. Result
9. Summary
10. Future

3. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Prologue: Questions
1. Have you heard Valgrind?
2. Have you used Valgrind?
3. Do you want to use Valgrind?
4. Do you want to understand Valgrind?
5. Do you want to fix Valgrind?
6. Do you want to maintain Valgrind?
7. Do you want to write Valgrind (from scratch)?

4. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Prologue: Audience
Use *BSD?
Write code?
Use (want) Valgrind?
Report problems?
Read Valgrind code?
Write
platform dependent
code?
Write
platform independent
code?
Yes
No
User
Yes
Yes
Yes
Yes
Yes
Yes
Advanced user
Valgrind user
Helpful Valgrind user
Luser
Advanced helpful Valgrind user
Valgrind porting developer
Valgrind architect

5. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Prologue: Goals (of this presentation)
1. Involve more people to help development
2. Introduce design & explain internal (esp. syscall + execve
+ signal)
3. Convince {Open,Net}BSD developers for necessary
changes

6. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Introduction: Authors' words
From ``Valgrind: A Framework for Heavyweight Dynamic
Binary Instrumentation'':
▶ A dynamic binary instrumentation (DBI) framework
▶ Designed for building heavyweight dynamic binary
analysis (DBA) tools
▶ ``Shadow values'' (register and memory)

7. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Introduction: Wikipedia
From Wikipedia (https://en.wikipedia.org/wiki/Valgrind)
``Overview'':
Valgrind is in essence a virtual machine using
just-in-time (JIT) compilation techniques, including
dynamic recompilation.

8. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Introduction: Design: Emulator
48 89 d1
48 8b 3c 24
48 8d 54 fc 10
:
Client Code (.text)
%rdx, %rcx)
(%rsp), %rdi)
0x10(%rsp, %rdi, 8), %rdx)
(mov
(mov
l(ea
RAX
RBX
RCX
RDX
RSP
RBP
RSI
RDI
R8
R9
R10
R11
R12
R13
R14
R15
Guest State
Flags
YMM
Registers
FPU
Registers
Scheduler
Client Data (.data)
Client Address Space

9. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Introduction: Design: Address space
0x400000
0x38000000
Client
text + data
Valgrind
text + data
VM_MAXUSER_ADDRESS
(0x00007f7fffffc000)
VM_MIN_ADDRESS
(PAGE_SIZE == 0x1000)
Valgrind
Stack
Client + Valgrind
Heap
Address
Space
Mng.
Guest
State
Client
Stack
Valgrind
Data

10. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Development: Project
▶ valgrind.org
▶ Unofficial FreeBSD branch @ Bitbucket
(https://bitbucket.org/stass/valgrind-freebsd)
OpenBSD/NetBSD branches are forked from valgrind-freebsd

11. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Development: Code: Structure
Valgrind tool process
Userland
Emulator (VEX)
OS / Architecture
Dependent
Code
Host
System
Call
Handling
Client
System
Call
Handling
Address
Space
Management
Signal
Handling
Start-Up
Code
initimg
(execve)
Kernel
theoretical
abstract
realistic
concrete

12. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Development: Code: Files and lines
Files:
▶ 1,032 *.c files
▶ 81 *.c files containing ``freebsd''
▶ 48 *.[cS] files containing ``#if .*freebsd''
▶ 8 *freebsd*.c files
▶ 4 *freebsd*.S files
Lines:
▶ 499 lines containing ``freebsd'' in *.[cS] files
▶ 340 lines containing ``#if .*freebsd'' in *.[cS] files

13. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Development: Code: #ifdef's
OS and architecture conditional macros:
▶ VGO_*: OS
▶ VGA_*: Architecture
▶ VGP_*: Platform (== Architecture + OS)
How many for FreeBSD:
▶ VGO_freebsd: 196 lines
▶ VGA_amd64: 53 lines
▶ VGP_amd64_freebsd: 56 lines

14. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Development: Code: portability
From ``Porting Plans'' in
http://valgrind.org/info/platforms.html:
...
Unlike NetBSD or GCC, we are not interested in
having Valgrind work on every platform in the
known universe: the maintenance burden is too high.
...
Portable, but not very portable.
From ``Out Of Tree'':
x86/FreeBSD: Doug Rabson and others haved done a
fairly complete port of Valgrind 3.X. The FreeBSD
porting team actively maintains the port. Snapshots
of the work in progress are at this FreeBSD page.

15. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Development: Phases
Start
Build
(auto* tools, #ifdef)
No client
(internal initialization)
Empty client
(client initialization)
Run
test program
Debug & fix
Valgrind
Debug & fix
test program
Beta
Release
Initial
Development
(Not Working)
Alpha
Quality
(Somewhat
Working)
(Developer
Only)
Release
Beta
Quality
(Almost
Working)
Release
Quality
(Working)
For
Valgrind
Developers
For
Valgrind
Users

16. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Development: Debugging
gdb
(vgdb)
gdb
gdb ls
valgrind
valgrind ls
gdbserver
gdb serial protocol
ptrace(2)
ptrace(2)

17. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: System call
▶ Interface for userland to interact with kernel
▶ open(2), read(2), write(2), ...
▶ Implemented as system call instruction
▶ Entering kernel mode

18. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
% objdump -d /usr/lib/libc.a | sed -ne '/<_thread_sys_read>:/,/^$/p'
0000000000000010 <_thread_sys_read>:
10: b8 03 00 00 00 mov $0x3,%eax
15: 49 89 ca mov %rcx,%r10
18: 0f 05 syscall
1a: 72 e4 jb 0 <_thread_sys_read-0x10>
1c: c3 retq

19. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
% objdump -d /usr/lib/libc.a | sed -ne '/<_thread_sys_mmap>:/,/^$/p'
0000000000000000 <_thread_sys_mmap>:
0: 48 83 ec 28 sub $0x28,%rsp
4: 31 c0 xor %eax,%eax
6: 4c 89 4c 24 08 mov %r9,0x8(%rsp)
b: 45 89 c1 mov %r8d,%r9d
e: 41 89 c8 mov %ecx,%r8d
11: 89 d1 mov %edx,%ecx
13: 48 89 f2 mov %rsi,%rdx
16: 48 89 fe mov %rdi,%rsi
19: bf c5 00 00 00 mov $0xc5,%edi
1e: c7 04 24 00 00 00 00 movl $0x0,(%rsp)
25: e8 00 00 00 00 callq 2a <_thread_sys_mmap+0x2a>
2a: 48 83 c4 28 add $0x28,%rsp
2e: c3 retq

20. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: System call: *BSD
syscall
mov $123, $rax
:
trap vector
User Text
User Stack
Kernel Text
syscall
mov $123, $rax
:
User Text
User Stack
Kernel Text
SS / CS
syscall
mov $123, $rax
:
User Text
User Stack
Kernel Text
SS / CS
frame reg
trap vector
syscall syscall
trap vector
syscall
syscall instruction
saves SS/CS on stack
entering kernel mode
trap vector
saves registers on stack
(frame reg)
syscall
mov $123, $rax
:
User Text
User Stack
Kernel Text
SS / CS
frame reg
trap vector
syscall
syscall code runs
saving retval
into frame reg
syscall
mov $123, $rax
:
User Text
User Stack
Kernel Text
trap vector
syscall
ret instruction
restores context w/ retval
entering user mode
return values
(RAX, RDX)

21. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: System call: Valgrind
▶ Valgrind does not rely on host libc
▶ Instead, Valgrind is built on top of POSIX-like ``real''
system calls
▶ Two purposes: for Valgrind itself, and for client (``client
syscall'')

22. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: System call: Valgrind: Client
(Next page)

23. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Valgrind scheduler
executes syscall instruction
then syscall handler
is called
Client syscall handler
checks syscall number
and dispatches
If read sycall is needed
set syscall arguments
reading saved
machine state
Enter kernel
trap vector saves SS/CS
syscall handler saves
frame reg
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
SS/CS
frame reg
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
SS/CS
frame reg
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
Syscall code runs
saving retval
into frame reg
Return to userland
WRK asm wrapper
saves results
into machine state
Return values
(RAX, RDX)
Return values
(RAX, RDX)
syscall handler
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
syscall
mov $123, $rax
:
Client Text
Client Stack
client_syscall()
ML_(do_syscall_
for_client_WRK)()
Client
Machine State
Resume Valgrind scheduler
Client sees
syscall results
in machine state

24. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
.globl ML_(do_syscall_for_client_WRK)
ML_(do_syscall_for_client_WRK):
/* save callee-saved regs */
pushq %rbp
movq %rsp, %rbp
pushq %rdi // -8(%rbp) syscallno
pushq %rsi // -16(%rbp) guest_state
pushq %rdx // -24(%rbp) sysmask
pushq %rcx // -32(%rbp) postmask
pushq %r8 // -40(%rbp) sigsetSzB
1: /* Even though we can't take a signal until the sigprocmask completes,
start the range early.
If eip is in the range [1,2), the syscall hasn't been started yet */
/* Set the signal mask which should be current during the syscall. */
/* Save and restore all 5 arg regs round the call. This is easier
than figuring out the minimal set to save/restore. */
movq $__NR_sigprocmask, %rax // syscall #
movq $VKI_SIG_SETMASK, %rdi // how
movq (%rdx), %rsi // sysmask
syscall
jb 7f /* sigprocmask failed */
movq -32(%rbp), %rdx
testq %rdx, %rdx
jz 10f
movl %eax, (%rdx) // postmask

25. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
10:
/* OK, that worked. Now do the syscall proper. */
/* 6 register parameters */
movq -16(%rbp), %r11 /* r11 = VexGuestAMD64State * */
movq OFFSET_amd64_RDI(%r11), %rdi
movq OFFSET_amd64_RSI(%r11), %rsi
movq OFFSET_amd64_RDX(%r11), %rdx
movq OFFSET_amd64_R10(%r11), %r10
movq OFFSET_amd64_R8(%r11), %r8
movq OFFSET_amd64_R9(%r11), %r9
/* 2 stack parameters plus return address (ignored by syscall) */
movq OFFSET_amd64_RSP(%r11), %r11 /* r11 = simulated RSP */
movq 16(%r11), %rax
pushq %rax
movq 8(%r11), %rax
pushq %rax
/* (fake) return address. */
movq 0(%r11), %rax
pushq %rax
/* syscallno */
movq -8(%rbp), %rax
/* If rip==2, then the syscall was either just about
to start, or was interrupted and the kernel was
restarting it. */
2: syscall
3: /* In the range [3, 4), the syscall result is in %rax,
but hasn't been committed to RAX. */

26. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
/* stack contents: 3 words for syscall above, plus our prologue */
setc 0(%rsp) /* stash returned carry flag */
movq -16(%rbp), %r11 /* r11 = VexGuestAMD64State * */
movq %rax, OFFSET_amd64_RAX(%r11) /* save back to RAX */
movq %rdx, OFFSET_amd64_RDX(%r11) /* save back to RDX */
/* save carry flag to VEX */
xorq %rax, %rax
movb 0(%rsp), %al
movq %rax, %rdi /* arg1 = new flag */
movq %r11, %rsi /* arg2 = vex state */
addq $24, %rsp /* remove syscall parameters */
call LibVEX_GuestAMD64_put_rflag_c
4: /* Re-block signals. If eip is in [4,5), then the syscall
is complete and we needn't worry about it. */
movq $__NR_sigprocmask, %rax // syscall #
movq $VKI_SIG_SETMASK, %rdi // how
movq -32(%rbp), %rsi // postmask
movl (%rsi), %esi
syscall
jb 7f /* sigprocmask failed */
/* don't bother oldmask */

27. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
5: /* now safe from signals */
xorq %rax,%rax
movq -8(%rbp), %rdi
movq -16(%rbp), %rsi
movq -24(%rbp), %rdx
movq -32(%rbp), %rcx
movq -40(%rbp), %r8
movq %rbp, %rsp
popq %rbp
ret
7: /* failure: return 0x8000 | error code */
orq $0x8000, %rax
movq -8(%rbp), %rdi
movq -16(%rbp), %rsi
movq -24(%rbp), %rdx
movq -32(%rbp), %rcx
movq -40(%rbp), %r8
movq %rbp, %rsp
popq %rbp
ret

28. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: System call: Valgrind: read(2)
PRE(sys_read)
{
*flags |= SfMayBlock;
PRINT("sys_read ( %ld, %#lx, %llu )", ARG1, ARG2, (ULong)ARG3);
PRE_REG_READ3(ssize_t, "read",
unsigned int, fd, char *, buf, vki_size_t, count);
if (!ML_(fd_allowed)(ARG1, "read", tid, False))
SET_STATUS_Failure( VKI_EBADF );
else
PRE_MEM_WRITE( "read(buf)", ARG2, ARG3 );
}
POST(sys_read)
{
vg_assert(SUCCESS);
POST_MEM_WRITE( ARG2, RES );
}

29. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: execve(2)
▶ Executing a debug target program (client) under Valgrind
== execve(2) a program under kernel
▶ Initial process ``image'' == memory + registers +
per-process resources (``struct process'' / ``struct proc'')
▶ Memory
▶ Text, data, bss: ELF Program Header PT_LOAD entries
▶ Stack: Arguments, environments, ``aux'' information (for
dynamic linker)
▶ Registers
▶ Parameters to start code (crt or ld.so)
▶ Per-process resources
▶ Timer, ... (what else?)

30. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: execve(2): On *BSD
▶ Kernel parses ELF Program Header
▶ Kernel maps text, data, bss, and stack
▶ Kernel prepares stack context
▶ Kernel prepares register context (``struct trapframe'')
▶ Rest done by userland (start code (``crt'') or dynamic
linker (``ld.so''))

31. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
0x400000
text + data + bss
(PT_LOAD entries)
Stack
Address Space
Machine State
(struct trapframe)
Registers
Flags Co-processor
Per-Process State
(struct process/proc)
char **argv
char **env
auxinfo
arg / env strings
argc
RSP

32. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: execve(2): On Valgrind
▶ ``initimg''
▶ Basically simulates ``execve(2)''
▶ Memory and registers
▶ ELF Program Header parsing
▶ Address space mapped by Valgrind (not kernel)

33. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
VG_(ii_create_image)()
trap vector
Valgrind Text
Valgrind Stack
Kernel Text
syscall
subq $8, $rsp
andq $~15, $rsp
:
Client Text
Client Stack
Client
Machine State
argc, argv[], env[]
auxinfo
argv/env strings
ps_strings
setup_client_stack()
VG_(ii_finalise_image)()

34. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: Signal
▶ From Wikipedia: https://en.wikipedia.org/wiki/Unix_signal
Signals are a limited form of inter-process
communication used in Unix, Unix-like, and other
POSIX-compliant operating systems.
▶ Signal action: calling registered functions when signal
delivered (kernel calling userland function!)
▶ Signal trampoline: small code fragment to execute signal
handler and/or return to kernel
▶ BSD signal code: Ancient code (no one wants to touch)
Signal code is INTERESTING!!!!!

35. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: Signal: Sync signal: On *BSD
Signal Handler
Text
Stack
mov %rcx, (%rax)
Signal Handler
Text
Stack
mov %rcx, (%rax)
Signal Trampoline
Signal Handler
Text
Stack
mov %rcx, (%rax)
Signal Trampoline
Internal
Signal Stack Data
Signal Handler
Text
Stack
mov %rcx, (%rax)
Signal Trampoline
Signal Handler
Text
Stack
mov %rcx, (%rax)
Signal Trampoline
Signal Handler
Text
Stack
mov %rcx, (%rax)
Kernel creates
signal context
and calls
signal trampoline
Kernel restores
original context
Signal trampoline
reads stack data
and dispatches
handler
Handler finishes
and returns back
to trampoline
Trampoline calls
sigreturn()
to exits
signal context
Internal
Signal Stack Data
Internal
Signal Stack Data
Internal
Signal Stack Data
User code
Kernel sendsig Kernel sigreturn
User codeUser handlerUser trampoline User trampoline

36. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: Signal: Trampoline: FreeBSD
sys/amd64/amd64/sigtramp.S:
NON_GPROF_ENTRY(sigcode)
call *SIGF_HANDLER(%rsp) /* call signal handler */
lea SIGF_UC(%rsp),%rdi /* get ucontext_t */
pushq $0 /* junk to fake return addr. */
movq $SYS_sigreturn,%rax
syscall /* enter kernel with args */
0: hlt /* trap priviliged instruction */
jmp 0b
ALIGN_TEXT
esigcode:
.data
.globl szsigcode
szsigcode:
.long esigcode-sigcode

37. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: Signal: Trampoline: OpenBSD
sys/arch/amd64/amd64/locore.S:
NENTRY(sigcode)
call *%rax
movq %rsp,%rdi
pushq %rdi /* fake return address */
movq $SYS_sigreturn,%rax
syscall
movq $SYS_exit,%rax
syscall
.globl _C_LABEL(esigcode)
_C_LABEL(esigcode):

38. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: Signal: Trampoline: NetBSD
lib/libc/arch/x86_64/sys/__sigtramp2.S:
NENTRY(__sigtramp_siginfo_2)
movq %r15,%rdi
movq $SYS_setcontext, %rax
syscall
movq $-1,%rdi /* if we return here, something is wrong */
movq $SYS_exit, %rax
syscall
END(__sigtramp_siginfo_2)

39. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: Signal: On Valgrind
▶ Strategy
▶ Async: Block and poll (sigwaitinfo())
▶ Sync: ``Long-jump'' (HARD WORK)
▶ Trampoline
▶ Provide from user
▶ ``Fake'' return syscall

40. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
Signal Frame
Kernel creates
signal context
and calls
signal trampoline
Client signal handler
do task
Signal trampoline
reads stack data
and dispatches
handler
Handler creates
client signal context
and jump to
scheduler
Client trampoline
calls
client signal handler
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State V Machine State
Signal Frame
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
Signal Frame
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
Signal Frame
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
Signal Frame
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
Signal Frame
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
Signal Frame
sigreturn()
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
sigreturn()
C Signal Handler
C Text
V Stack
mov %rcx, (%rax)
V Signal Handler
V Text
C Stack
C Signal Trampoline
V Translated Code
Signal Trampoline
V Machine State
sigreturn()
Kernel sendsig
Valgrind signal handler
Client signal handler
Valgrind sendsig Valgrind sigreturn
Valgrind trampoline
Clinet trampolineClient trampoline
Client signal handler
returns to
client trampoline
Client trampoline
calls "fake"
sigreturn syscall
"Fake" sigreturn
restores
client context
Back to scheduler
to continue
restored
client context
Valgrind code
Client code
Valgrind code
Client code
longjmp

41. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Internal: Signal: Valgrind: Trampoline
.global VG_(amd64_freebsd_SUBST_FOR_sigreturn)
VG_(amd64_freebsd_SUBST_FOR_sigreturn):
/* This is a very specific sequence which GDB uses to
recognize signal handler frames. */
movq $__NR_fake_sigreturn, %rax
movq %rsp, %rdi
addq $40,%rdi
syscall
ud2
.global VG_(trampoline_stuff_end)
VG_(trampoline_stuff_end):

42. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Issues: procfs vs. sysctl
▶ procfs is heavily used
▶ To retrieve memory mapping information, opened file
descriptors, ...
▶ *BSD is basically moving away from that
▶ Need to extend sysctl (KERN_PROC_VMMAP)

43. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Issues: Versioning
▶ Syscall and other ABIs (execve + start code, signal, ...)
are hardcoded
▶ Can't handle multiple OS versions

44. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Issues: ioctl
▶ Valgrind needs to know all syscall arguments memory
read/write
▶ ``Broken'' (== impossible to become perfect) by design for
corner cases (user-defined ioctl in kernel modules)

45. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Issues: LD_PRELOAD
▶ OpenBSD prohibits LD_PRELOAD for suid'ed programs

46. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Summary
▶ External project forked from valgrind-freebsd on
BitBucket
▶ Basically ported to OpenBSD/amd64 and NetBSD/amd64
▶ Supported static and dynamic executables
▶ Supported only basic system calls
▶ Need a few changes against base
▶ Already found one (potential) bug in OpenBSD's ld.so
▶ TODOs
▶ More ground work (execve(2)/fork(2), threading, ...)
▶ More system calls (ioctl(2))
▶ More architectures (i386, arm, ...)
▶ Code clean-ups (esp. around procfs)

47. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Valgrind
User
Valgrind
Developer

48. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Valgrind
User
Valgrind
Developer

49. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Future
▶ Maitainance is NOT fun (but I will do some)
▶ Consider alternatives (clang + ASAN)

50. .....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
....
.
....
.
.....
.
....
.
.....
.
....
.
....
.
Questions