1. The document discusses debugging FreeBSD kernels through various tools and techniques such as kgdb(1), ddb(4), ktrace(1), and kdump(1). 2. Common issues like kernel crashes and hangs can be debugged using tools that examine CPU registers, step through code, and analyze kernel traces. 3. Effective debugging requires understanding kernel data structures and configuration options for enabling debugging features.

2.  FreeBSD Kernel Debugging
 Port Development

3. gdb Kdgb
DTrace Valgrind ktrace ktr
Kernel
Userland
Space

4. Kernel
Debugging Tracking

5.  The process that takes a control of execution
of another process and make step over of
code to view the process execution, show
relation between consecutive processes.
 Debugging mainly used to detect error or
bugs.

6. Common
Strategies
Kernel Crash
Kernel Hang

7. ● dumpon(8) and savecore(8)
– /etc/rc.conf
● [syrinx@tweety]/(76)% cat /etc/rc.conf | grep dump
● dumpdev="/dev/ad4s4b"
● dumpdir="/var/crash/"
● Remote systems
● options DDB_UNATTENDED
● Virtual machines - Pros and cons
– Qemu
– VirtualBox

8. ● Kernel configuration
makeoptions DEBUG=-g #memguard(9) and redzone(9)
options DEBUG_MEMGUARD
#ktrace(1) options DEBUG_REDZONE
options KTRACE
#ktr(4)
options KDB options KTR
options KDB_TRACE options ALQ
options DDB options ALQ_KTR
options GBD options KTR_ENTRIES=4096
options KTR_COMPILE=KTR_INET
options INVARIANTS options KTR_MASK=KTR_INET
options INVARIANT_SUPPORT options KTR_CPUMASK=0x3
options WITNESS
options WITNESS_SKIPSPIN
options LOCK_PROFILING

9. ● If kernel dies, display driver dies too
● Do not panic the system while fsck (8) is running
● [syrinx@tweety]/(78)% cat /etc/rc.conf | grep fsck
● fsck_y_enable="YES"
● background_fsck="NO"
● Core dumps may contain sensitive information
● # boot -s
● # dumpon /dev/ad4s4b
● # mount -a -art ufs
● # /command-to-crash-the-kernel

10. ddb(8) kgmon(8)
kgdb(1) pmcstat(1)
ktr(4) hwpmc(4)
ktr(9) addr2line(1)
ktrdump(8) kldstat(8)
ktrace(1) fstat(1)
kdump(1) iostat(8)
dmesg(8) ipcs(1)
objdump(1) ipmi(9)
size(1) netstat(1)
savecore(8) ps(1)
dumpon(8) vmstat(8)
textdump(8) redzone(9)
hexdump(1) memguard(9)
truss(1)

11. ● Kernel crashes usually require straighforward
approach
● kgdb(1) is an extention of gdb(1) that
understands FreeBSD kernel corefiles
● Symbol resolving
● Scripting support
● On-line debugging via /dev/mem
● Remote debugging
– via serial line
– via firewire

12. [syrinx@tweety]/home/syrinx(92)% sudo kgdb /boot/kernel.old/kernel /var/crash/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
Unread portion of the kernel message buffer:
rum0: could not multi read MAC register: USB_ERR_NOT_CONFIGURED
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x0
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc069bf24
stack pointer = 0x28:0xe6b40c44
frame pointer = 0x28:0xe6b40c60
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (rum0 taskq)
panic: from debugger

13. ● NULL Pointer Dereference
● Resource leak
● Bad test condition, resulting in code path never executed
● Use before test (NULL/-1)
● Use after free
● Buffer overflow
● Unsafe use of returned value
● Uninitialized value read
● Type and allocation size mismatch

14. ● ddb(4) – interactive kernel ● Inspecting locks with ddb
debugger ● db> show lock [lockaddr]
● Poor symbol resolving ● db> show turnstile [lockaddr]
compared to kgdb(1)
● # mutexes
● Extensible – new commands ● db> show lockchain
added at compile time -
● # six locks, sleepqueues
● Sources at src/sys/ddb
● db> show sleepchain
● Scripting support since
FreeBSD 7.1 – ddb(8)
● More info in manual page.
● Enter DDB
● #sysctl debug.kdb.enter=1

15.  Examination of CPU registers, variables, etc.
 Stepping over the instructions code to view
the process execution.

17.  ps shows selected fields from the process
structures. With an understanding of the
structures, it can give a good idea of what's
going on.
 top is like a repetitive ps: it shows the most
active processes at regular intervals.
 vmstat shows a number of parameters,
including virtual memory. It can also be set
up to run at regular intervals.

18.  iostat is similar to vmstat, and it duplicates
some fields, but it concentrates more on I/O
activity.
 netstat show network information. It can also
be set up to show transfer rates for specific
interfaces.
 systat is a curses-based program which
displays a large number of parameters,
including most of the parameters displayed
by vmstat, iostat and netstat.

26. [syrinx@tweety]/home/syrinx(71)% sysctl debug.ktr
debug.ktr.alq_enable: 0
debug.ktr.alq_file: /tmp/ktr.out
debug.ktr.alq_depth: 1024
debug.ktr.alq_failed: 0
debug.ktr.alq_cnt: 0
debug.ktr.alq_max: 0
debug.ktr.clear: 0
debug.ktr.version: 2
debug.ktr.entries: 1024
debug.ktr.compile: 270540806
debug.ktr.mask: 270540806
debug.ktr.cpumask: 3
● KTR buffer accessible in DDB
db> show ktr [v]
● read the contents of a ktr(4) file
[syrinx@tweety]/(105)% sudo ktrdump -t -o /tmp/ktr.out

27. ● Kernel process tracing
● [syrinx@tweety]/home/syrinx(89)% ktrace -a -tcy -f /home/syrinx/krtace.out
ifconfig
● kdump(1) displays the kernel trace data
● [syrinx@tweety]/home/syrinx(91)% kdump -f ~/krtace.out
...
7490 ifconfig CALL write(0x1,0x28203000,0x38)
7490 ifconfig GIO fd 1 wrote 56 bytes
options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
7490 ifconfig RET write 56/0x38
7490 ifconfig CALL ioctl(0x3,SIOCGIFMAC,0xbfbfdfac)
7490 ifconfig RET ioctl -1 errno 22 Invalid argument
7490 ifconfig CALL ioctl(0x3,SIOCGIFMEDIA,0xbfbfdfa0)
...

28.  ktrace -d -i -p 0
 kdump –f ktrace.out >>myfile.txt

32. 1. Writing the Makefile.
2. Writing the description files.
3. Creating the checksum file
4. Testing the port
5. Checking your port with portlint
6. Submitting the port.

33. # New ports collection makefile for: oneko
# Date created: 5 December 1994
# Whom: asami
#
# $FreeBSD$
#
PORTNAME= oneko
PORTVERSION= 1.1b
CATEGORIES= games
MASTER_SITES= ftp://ftp.cs.columbia.edu/archives/X11R5/contrib/
MAINTAINER= asami@FreeBSD.org
COMMENT= A cat chasing a mouse all over the screen
MAN1= oneko.1
MANCOMPRESSED= yes
USE_IMAKE= yes
.include <bsd.port.mk>

34.  pkg-descr
 pkg-plist
 Refer to the pkg_create(1) manual page for
details on the packing list.

35.  This is a longer description of the port. One
to a few paragraphs concisely explaining
what the port does is sufficient.
 Example
This is a port of oneko, in which a cat chases a poor mouse all over
the screen.
:
(etc.)
WWW: http://www.oneko.org/

36.  This file lists all the files installed by the port.
 Also called the “packing list” because the package is
generated by packing the files listed here.
 The pathnames are relative to the installation prefix
(usually /usr/local or /usr/X11R6).
 If you are using the MANn variables (as you should be), do
not list any manpages here. If the port creates directories
during installation, make sure to add @dirrm lines to
remove them when the package is deleted.
 Example:
bin/oneko
lib/X11/app-defaults/Oneko
lib/X11/oneko/cat1.xpm
lib/X11/oneko/cat2.xpm
lib/X11/oneko/mouse.xpm
@dirrm lib/X11/oneko

37.  Type make makesum. The ports make rules
will automatically generate the file distinfo.

38.  pkg-plist does not contain anything not
installed by your port
 pkg-plist contains everything that is installed
by your port
 Your port can be installed multiple times
using the reinstall target
 Your port cleans up after itself upon deinstall

39. 1. make install
2. make package
3. make deinstall
4. pkg_add package-name
5. make deinstall
6. make reinstall
7. make package

40.  http://www.FreeBSD.org/doc/en_US.ISO8859
-1/articles/contributing/contrib-
how.html#CONTRIB-GENERAL
 http://www.FreeBSD.org/cgi/query-pr-
summary.cgi?category=ports