Successfully reported this slideshow.
Your SlideShare is downloading. ×

Troubleshooting Linux Kernel Modules And Device Drivers

Upcoming SlideShare
Linux Troubleshooting
Linux Troubleshooting
Loading in …3

Check these out next

1 of 35 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (20)


Similar to Troubleshooting Linux Kernel Modules And Device Drivers (20)

Recently uploaded (20)


Troubleshooting Linux Kernel Modules And Device Drivers

  1. 1. Troubleshooting Linux Kernel Modules and Device Drivers Mike Anderson Chief Scientist The PTR Group, Inc. [email_address] Source:
  2. 2. What We’ll Talk About <ul><li>How do errors show up in the kernel? </li></ul><ul><li>Watching kernel/user-space interaction via strace </li></ul><ul><li>Debugging with printk </li></ul><ul><li>Using the /proc file system </li></ul><ul><li>Using the kgdb debugger </li></ul><ul><li>Debugging with hardware ala LEDs or a JTAG unit </li></ul>
  3. 3. Challenges of Kernel Debugging <ul><li>There are many features that can make kernel debugging especially difficult </li></ul><ul><ul><li>Optimizing compilers can rearrange code </li></ul></ul><ul><ul><ul><li>Instruction pointer seems to jump around </li></ul></ul></ul><ul><ul><li>The use of the MMU can obfuscate addresses </li></ul></ul><ul><ul><ul><li>Physical vs. virtual addresses </li></ul></ul></ul><ul><ul><li>Startup code is particularly difficult to debug because of its closeness to the “metal” </li></ul></ul><ul><ul><li>No equivalent for the user-space gdbserver for drivers </li></ul></ul><ul><li>Early kernel debugging may require hardware assistance </li></ul>
  4. 4. Device Drivers/Kernel Modules <ul><li>Assuming that your kernel is otherwise working, most of the problems that you’ll encounter are related to device drivers </li></ul><ul><ul><li>Drivers can either be statically linked or dynamically loaded to the kernel </li></ul></ul><ul><li>A dynamically loaded driver takes the form of a kernel module </li></ul><ul><ul><li>Can be dynamically loaded and unloaded at kernel run time </li></ul></ul><ul><ul><ul><li>Frequently handled by daemons such as udev </li></ul></ul></ul>
  5. 5. Example Module <ul><li>#include <linux/module.h> </li></ul><ul><li>#include <linux/init.h> </li></ul><ul><li>#include <linux/kernel.h> </li></ul><ul><li>MODULE_LICENSE(&quot;GPL and additional rights&quot;); </li></ul><ul><li>MODULE_AUTHOR( “” ); </li></ul><ul><li>MODULE_DESCRIPTION( “My first driver!” ); </li></ul><ul><li>int __init mymodule_init_module(void) { </li></ul><ul><li>printk(KERN_DEBUG “mymodule_init_module() called, &quot;); </li></ul><ul><li>return 0; </li></ul><ul><li>} </li></ul><ul><li>void __exit mymodule_cleanup_module(void) { </li></ul><ul><li>printk(KERN_DEBUG “mymodule_cleanup_module() called&quot;); </li></ul><ul><li>} </li></ul><ul><li>module_init(mymodule_init_module); </li></ul><ul><li>module_exit(mymodule_cleanup_module); </li></ul>
  6. 6. When Things go Wrong… <ul><li>Problems in device drivers typically manifest themselves in one of three ways </li></ul><ul><ul><li>Kernel panic </li></ul></ul><ul><ul><ul><li>Fatal to the system </li></ul></ul></ul><ul><ul><li>Kernel oops </li></ul></ul><ul><ul><ul><li>Near fatal to the system </li></ul></ul></ul><ul><ul><li>Hardware just doesn’t work correctly </li></ul></ul><ul><ul><ul><li>Could be fatal to you! </li></ul></ul></ul>Source:
  7. 7. Kernel Panic <ul><li>When the Linux kernel determines that a fatal error has occurred, and no recovery is possible, it “panics” </li></ul><ul><ul><li>Frequently, an exception in an interrupt context </li></ul></ul><ul><li>Panic outputs a message to the console </li></ul><ul><ul><li>The output will help you find the source of the bug </li></ul></ul><ul><li>Typically results in a system reboot on an embedded Linux target </li></ul><ul><ul><li>Or, blinking keyboard LEDs on some desktop versions of Linux </li></ul></ul>Source:
  8. 8. Sample Panic Output <ul><li>EIP: 0060: [<c02b2516>] Not tainted VLI </li></ul><ul><li>EFLAGS: 00010286 (2.6.18) </li></ul><ul><li>EIP is at nf_queue+0x16/0x240 </li></ul><ul><li>eax: 00000000 ebx: 00000001 ecx: 00000000 edx: 00000002 </li></ul><ul><li>esi: dece0920 edi: c03a3aa8 ebp: c0277e70 esp: cf165e78 </li></ul><ul><li>ds: 007b es: 007b ss: 0068 </li></ul><ul><li>Process sh (pid:5278, threadinfo=cf164000 task=dc5b4ab0) </li></ul><ul><li>Stack: cf165f00 dae29000 00000000 c0277e70 00000001 cf165f00 c03a3aa8 c0277e70 </li></ul><ul><li>c02b1f7c cf165f00 dece0920 00000002 00000001 dae29000 00000000 c0277e70 </li></ul><ul><li>00000000 dece0920 00000000 da641b40 da54bcde cc318abc c0277c10 00000002 </li></ul><ul><li>Call Trace: </li></ul><ul><li>[<c0277e70>] ip_local_deliver_finish+0x0/0x230 </li></ul><ul><li>[<c0277e70>] ip_local_deliver_finish+0x0/0x230 </li></ul><ul><li>… deleted for space … </li></ul><ul><li>[<c012caa6>] do_softirq+0x26/0x30 </li></ul><ul><li>[<c010528e>] do_IRQ+0x1e/0x30 </li></ul><ul><li>[<c0103ada>] common_interrupt+0x1a/0x20 </li></ul><ul><li>Code: c0 75 ec e9 bd e5 e6 ff 8d b6 00 00 00 00 8d bc 27 00 00 00 00 55 57 56 53 </li></ul><ul><li>83 ec 10 8b 54 24 2c 8b 74 24 28 8b 04 9b c0 42 3a c0 <8b> 38 85 ff 0f 84 96 01 </li></ul><ul><li>00 00 86 44 24 2c 8b 7c 24 2c c7 44 24 </li></ul><ul><li><0>Kernel panic - not syncing: Fatal exception in interrupt </li></ul>
  9. 9. Kernel oops <ul><li>An oops message is displayed when a recoverable error has occurred in kernel space: </li></ul><ul><ul><li>Access to bad address, e.g., through a NULL pointer </li></ul></ul><ul><ul><li>Illegal or invalid instruction </li></ul></ul><ul><ul><li>Etc… </li></ul></ul><ul><li>The calling user process is killed </li></ul><ul><ul><li>The system should be considered unstable at this point </li></ul></ul><ul><li>The oops message displays: </li></ul><ul><ul><li>The state of the processor at the time of the fault, including registers and address of faulting instruction </li></ul></ul><ul><ul><li>function call stack traceback </li></ul></ul><ul><li>The addresses are replaced with symbols if the kallsyms kernel configuration option is selected at kernel compile time </li></ul>
  10. 10. Example Oops <ul><li>Modules linked in: arc4 ecb crypto_blkcipher ieee80211_crypt_wep vfat fat ipt_MASQUERADE iptable_nat nf_nat bridge bnep rfcomm l2cap bluetooth ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi sunrpc ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables cpufreq_ondemand acpi_cpufreq fuse loop dm_multipath ipv6 ppdev parport_pc parport video 8139cp output ac battery firewire_ohci fglrx(P) firewire_core crc_itu_t 8139too mii ipw2200 ieee80211 ieee80211_crypt snd_intel8x0m asus_laptop snd_intel8x0 button snd_seq_dummy snd_ac97_codec ac97_bus snd_seq_oss snd_seq_midi_event joydev snd_seq pcspkr serio_raw snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm usb_storage iTCO_wdt snd_timer snd iTCO_vendor_support soundcore snd_page_alloc sr_mod cdrom sg dm_snapshot dm_zero dm_mirror dm_mod pata_acpi ata_generic ata_piix libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode] </li></ul><ul><li>Pid: 5488, comm: Xorg Tainted: P ( #1) </li></ul><ul><li>EIP: 0060:[<c04d038d>] EFLAGS: 00213246 CPU: 0 </li></ul><ul><li>EIP is at task_has_capability+0x48/0x76 </li></ul><ul><li>EAX: 00000030 EBX: f6852030 ECX: f6a8af28 EDX: 00000000 </li></ul><ul><li>ESI: f36e50e0 EDI: f354fec8 EBP: f354fed4 ESP: f354fe84 </li></ul><ul><li>DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 </li></ul><ul><li>Process Xorg (pid: 5488, ti=f354f000 task=f35f0e90 task.ti=f354f000) </li></ul><ul><li>Stack: c06d48b5 f6852030 f35f0e90 00000003 f35f0e90 f6852030 00000000 00000000 </li></ul><ul><li>00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 </li></ul><ul><li>00000000 f6852030 f35f0e90 f66f8000 f354fee4 c04d03da f35f0e90 f8d52580 </li></ul>
  11. 11. Example Oops part 2 <ul><li>Call Trace: </li></ul><ul><li>[<c04d03da>] ? selinux_capable+0x1f/0x23 </li></ul><ul><li>[<c04cbfe9>] ? security_capable+0xc/0xe </li></ul><ul><li>[<c042de97>] ? __capable+0xb/0x22 </li></ul><ul><li>[<f8c2d560>] ? firegl_version+0x0/0x1b0 [fglrx] </li></ul><ul><li>[<c042debe>] ? capable+0x10/0x12 </li></ul><ul><li>[<f8c2d427>] ? firegl_ioctl+0xe7/0x220 [fglrx] </li></ul><ul><li>[<c062aa49>] ? mutex_lock+0x1d/0x2d </li></ul><ul><li>[<c04d0c45>] ? file_has_perm+0x7c/0x85 </li></ul><ul><li>[<f8c24685>] ? ip_firegl_ioctl+0xe/0x10 [fglrx] </li></ul><ul><li>[<c048d2c8>] ? vfs_ioctl+0x50/0x69 </li></ul><ul><li>[<c048d51a>] ? do_vfs_ioctl+0x239/0x24c </li></ul><ul><li>[<c04d0ddb>] ? selinux_file_ioctl+0xa8/0xab </li></ul><ul><li>[<c048d56d>] ? sys_ioctl+0x40/0x5b </li></ul><ul><li>[<c0405bf2>] ? syscall_call+0x7/0xb </li></ul><ul><li>====================== </li></ul><ul><li>======================Code: 00 89 d0 f3 ab 8b 4d b8 89 d8 b2 04 c1 f8 05 c6 45 bc 03 89 5d c4 89 4d c0 74 16 48 b2 45 74 11 53 68 b5 48 6d c0 e8 11 f3 15 00 <0f> 0b 59 5b eb fe 8b 46 08 83 e3 1f 0f b7 f2 8d 55 bc 88 d9 52 </li></ul><ul><li>EIP: [<c04d038d>] task_has_capability+0x48/0x76 SS:ESP 0068:f354fe84 </li></ul><ul><li>---[ end trace 9451402cb10d9c54 ]--- </li></ul>
  12. 12. Enabling Symbolic Kernel Output
  13. 13. Module Debugging Techniques <ul><li>Examine the interaction with the kernel via strace </li></ul><ul><li>The next line of defense is printk </li></ul><ul><ul><li>There may be additional output you’re not seeing </li></ul></ul><ul><li>Next, we can try adding /proc file system entries </li></ul><ul><ul><li>Instrument the driver for debugging </li></ul></ul><ul><li>Enable source debugging via kgdb </li></ul><ul><li>Using hardware debuggers and “blinky lights” </li></ul>
  14. 14. Using strace to Watch System Calls <ul><li>When debugging what appears to be a kernel-space error, it can be helpful to watch the system calls that are made from user-space </li></ul><ul><ul><li>See what events lead to the error </li></ul></ul><ul><li>strace displays all system calls made by a program </li></ul><ul><ul><li>Can display timestamp information per system call as well </li></ul></ul>
  15. 15. Using strace to Watch System Calls #2 <ul><li>strace displays each system call’s arguments and return values </li></ul><ul><ul><li>string arguments are printed – very helpful! </li></ul></ul><ul><ul><li>errno values displayed symbolically </li></ul></ul><ul><li>The program being traced runs normally </li></ul><ul><ul><li>Not under control of a debugger </li></ul></ul><ul><ul><ul><li>No need to specially compile the user application </li></ul></ul></ul><ul><li>You can attach to a running program </li></ul><ul><ul><li>And trace forked applications as well… </li></ul></ul>
  16. 16. Example strace Output <ul><li>/ # strace ls /dev/labdev </li></ul><ul><li>execve(&quot;/bin/ls&quot;, [&quot;ls&quot;, &quot;/dev/labdev&quot;], [/* 8 vars */]) = 0 </li></ul><ul><li>fcntl64(0, F_GETFD) = 0 </li></ul><ul><li>fcntl64(1, F_GETFD) = 0 </li></ul><ul><li>fcntl64(2, F_GETFD) = 0 </li></ul><ul><li>geteuid() = 0 </li></ul><ul><li>getuid() = 0 </li></ul><ul><li>getegid() = 0 </li></ul><ul><li>getgid() = 0 </li></ul><ul><li>brk(0) = 0x1028ad68 </li></ul><ul><li>brk(0x1028bd68) = 0x1028bd68 </li></ul><ul><li>brk(0x1028c000) = 0x1028c000 </li></ul><ul><li>ioctl(1, TIOCGWINSZ or TIOCGWINSZ, {ws_row=0, ws_col=0, ws_xpixel=0, ws_ypixel=0 </li></ul><ul><li>ioctl(1, TCGETS or TCGETS, {B9600 opost isig icanon echo ...}) = 0 </li></ul><ul><li>ioctl(1, TCGETS or TCGETS, {B9600 opost isig icanon echo ...}) = 0 </li></ul><ul><li>lstat(&quot;/dev/labdev&quot;, {st_mode=S_IFCHR|0644, st_rdev=makedev(254, 0), ...}) = 0 </li></ul><ul><li>open(&quot;/etc/localtime&quot;, O_RDONLY) = -1 ENOENT (No such file or directory) </li></ul><ul><li>lstat(&quot;/dev/labdev&quot;, {st_mode=S_IFCHR|0644, st_rdev=makedev(254, 0), ...}) = 0 </li></ul><ul><li>fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(4, 64), ...}) = 0 </li></ul><ul><li>ioctl(1, TCGETS or TCGETS, {B9600 opost isig icanon echo ...}) = 0 </li></ul><ul><li>mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x300 </li></ul><ul><li>write(1, &quot;3[1;35m/dev/labdev3[0m&quot;, 23/dev/labdev </li></ul><ul><li>) = 23 </li></ul><ul><li>munmap(0x30000000, 4096) = 0 </li></ul><ul><li>exit(0) = ? </li></ul>
  17. 17. Debugging with printk <ul><li>printk debugging is the debug method preferred by Linus </li></ul><ul><ul><li>At least, according to his email traffic… </li></ul></ul><ul><li>Insert messages to be displayed at points of interest in kernel-space code </li></ul><ul><ul><li>E.g., printk(KERN_INFO “my_x= %d”, my_x); </li></ul></ul><ul><ul><li>Printk works like printf does in user-space except that printk can only print integers, strings & addresses </li></ul></ul><ul><li>printk can also be called from within ISRs </li></ul><ul><li>Messages can have “importance” settings that allow filtering </li></ul><ul><ul><li>Importance is set by prepending a 3-character string to the output message: “< n >” </li></ul></ul>
  18. 18. Debugging with printk #2 <ul><li>The messages are placed in a circular buffer that can be retrieved post mortem if needed </li></ul><ul><li>The “importance” string that is prepended to the printk message can be found in include/linux/kernel.h : </li></ul><ul><li>#define KERN_EMERG &quot;<0>&quot; /* system is unusable */ #define KERN_ALERT &quot;<1>&quot; /* action to be taken immediately */ #define KERN_CRIT &quot;<2>&quot; /* critical conditions */ #define KERN_ERR &quot;<3>&quot; /* error conditions */ #define KERN_WARNING &quot;<4>&quot; /* warning conditions */ #define KERN_NOTICE &quot;<5>&quot; /* normal but significant condition */ #define KERN_INFO &quot;<6>&quot; /* informational */ #define KERN_DEBUG &quot;<7>&quot; /* debug-level messages */ </li></ul>
  19. 19. Debugging with printk #3 <ul><li>Some level of control for printk output can be found in /proc/sys/kernel/printk </li></ul><ul><li>Let’s look at the following output: </li></ul><ul><ul><ul><li># cat /proc/sys/kernel/printk </li></ul></ul></ul><ul><ul><ul><li>7 4 1 7 </li></ul></ul></ul><ul><ul><li>This indicates: </li></ul></ul><ul><ul><ul><li>The console_loglevel is 7, so messages with importance of 0..6 will currently go to the console </li></ul></ul></ul><ul><ul><ul><li>The default message log level is 4, so messages that do not specify an importance are treated as level 4 </li></ul></ul></ul><ul><ul><ul><li>The minimum console log level is 1, so console_loglevel cannot be set to any value less than 1 </li></ul></ul></ul><ul><ul><ul><li>The default console log level is 7, so console_loglevel starts out set to 7 </li></ul></ul></ul>
  20. 20. Debugging with printk #4 <ul><li>You may control the console_loglevel by writing to /proc/sys/kernel/printk </li></ul><ul><li>To enable all printk messages with importance levels 0..7: </li></ul><ul><ul><ul><li># echo 8 > /proc/sys/kernel/printk </li></ul></ul></ul><ul><li>If the kernel command line contains the word “debug”, the console_loglevel starts with a value of 10 </li></ul>
  21. 21. Using the /proc File System <ul><li>Use /proc entries for driver instrumentation </li></ul><ul><li>It is possible to register “write” functions that allow us to dynamically modify values in the kernel (or drivers, or modules) </li></ul><ul><ul><li>Write values to the /proc file system entry </li></ul></ul><ul><li>Read /proc entries allow us to retrieve information from a running kernel entity </li></ul><ul><ul><li>Information is provided “live” </li></ul></ul><ul><li>Look at the driver source, there may already be /proc entries that can help you </li></ul>
  22. 22. Techniques for Source Debugging <ul><li>The two primary ways to provide source debugging in the Linux kernel are based on either kgdb or on the use of a hardware JTAG probe </li></ul><ul><li>Unfortunately, kgdb is not a standard feature of the kernel a/o </li></ul><ul><ul><li>You’ll have to patch your kernel to enable it </li></ul></ul><ul><li>Either technique will require the use of a kernel image that is compiled with debugging symbols </li></ul><ul><ul><li>Unless you like debugging in assembly language </li></ul></ul>
  23. 23. Compiling the Kernel with Debug Info <ul><li>This will increase the size of the debug kernel image by about 30% </li></ul><ul><li>However, you don’t need to load the debug version of the kernel </li></ul><ul><ul><li>Load the non-debug version to the target, but use the debug version for the debugger/JTAG probe </li></ul></ul><ul><li>Save off the vmlinux and file because these are used by the debugger or by you to find key addresses </li></ul><ul><ul><li>The (b)zImage can be loaded on the target as normal </li></ul></ul>
  24. 24. Enabling Debugging in the Kernel
  25. 25. Kernel gdb (kgdb) <ul><li>If you are using a stock kernel, kgdb is not included </li></ul><ul><ul><li>Linus doesn’t believe in a source debugger in the kernel </li></ul></ul><ul><ul><li>Many commercial Linux vendors do include it in their distributions though </li></ul></ul><ul><li>kgdb can be downloaded from: or and look for kgdb </li></ul><ul><ul><li>You’ll need to patch the kernel </li></ul></ul><ul><li>A new kgdb light is in the works for 2.6.26 </li></ul><ul><ul><li>Kgdb over the system console </li></ul></ul>
  26. 26. Enable kgdb in the Kernel
  27. 27. Kgdb Light in 2.6.26-rc8 <ul><li>Uses system console for I/O </li></ul>
  28. 28. kgdb Lash up <ul><li>kgdb supports debugging via the serial port </li></ul><ul><ul><li>The gdb debugger is running on a second machine using the vmlinux you compiled with debugging symbols </li></ul></ul><ul><ul><li>You attach to the system being debugged using gdb’s “target remote” command </li></ul></ul>Host Target Network Ethernet RS-232
  29. 29. Hardware-assisted Debugging <ul><li>There are a number of devices that can help with debugging </li></ul><ul><ul><li>LEDs, JTAGs, logic analyzers, oscilloscopes, bus analyzers and more </li></ul></ul><ul><li>These can range from a few cents to implement to several 10s of thousands of dollars </li></ul><ul><ul><li>You typically get what you pay for </li></ul></ul>
  30. 30. Debugging with LEDs <ul><li>Very simple: </li></ul><ul><ul><li>Blink on/off in various code sections under debug </li></ul></ul><ul><ul><li>Blink in sequences </li></ul></ul><ul><ul><li>Can display multiple-bit codes if multiple LEDs are available </li></ul></ul><ul><li>Very fast, little impact on run-time performance </li></ul><ul><ul><li>Adding LED debug code will likely not “make the problem go away” </li></ul></ul><ul><li>These may be the only option for debugging early x86 code </li></ul>
  31. 31. Debugging with LEDs, Caveats <ul><li>The LED(s) must be free for use </li></ul><ul><ul><li>Not tied in hardware to a network PHY or to displaying power status, for example </li></ul></ul><ul><li>LED are not very verbose </li></ul><ul><ul><li>You must decipher what the blinking means </li></ul></ul><ul><ul><li>Can be difficult to determine where you are in the code </li></ul></ul><ul><li>You can also attach an oscilloscope to the GPIO pins found on many processors for more information </li></ul>
  32. 32. Hardware Debuggers <ul><li>In the past, in-circuit emulators (ICE) where the debugger of choice </li></ul><ul><ul><li>You pulled the CPU, plugged the ICE in and plugged the CPU into the ICE </li></ul></ul><ul><ul><li>But, these where $80K+ each </li></ul></ul><ul><li>Logic analyzers are also good to have </li></ul><ul><ul><li>But, they are $35K+ for an empty mainframe </li></ul></ul><ul><ul><li>PC-based versions can be had for < $1K </li></ul></ul><ul><li>IEEE 1149.1 (JTAG) has become the debugger da jour </li></ul><ul><ul><li>JTAG uses a boundary-scan protocol </li></ul></ul><ul><ul><li>These range from $70 to $20K depending on model and features </li></ul></ul><ul><ul><li>At a minimum, a JTAG is really a “must-have” for firmware and board bring up </li></ul></ul>
  33. 33. Debugging with a JTAG Probe <ul><li>Debugging with a JTAG unit is much less involved than using kgdb </li></ul><ul><ul><li>Compile the kernel with debugging enabled </li></ul></ul><ul><ul><li>No need to patch the kernel for kgdb </li></ul></ul><ul><ul><li>Assumes your platform supports a JTAG interface </li></ul></ul><ul><li>You connect to the JTAG unit using whatever technique your JTAG probe requires </li></ul><ul><ul><li>Your JTAG GUI is dependent on the vendor </li></ul></ul><ul><ul><li>For those JTAG units that are gdb-aware, use the appropriate target remote commands </li></ul></ul>
  34. 34. Example JTAG Usage <ul><li>Connect the JTAG to the target and the host </li></ul><ul><li>Start the host application to control the JTAG </li></ul><ul><ul><li>Reset the target and load the register configuration settings to the JTAG unit </li></ul></ul><ul><li>Load code and enjoy! </li></ul><ul><ul><li>Useful for debugging drivers as well as bringing up new firmware and BSPs </li></ul></ul>
  35. 35. Summary <ul><li>“ Real developers” use printk – or at least Linus does  </li></ul><ul><li>Tools like strace allow you see the flow of execution </li></ul><ul><li>The /proc filesystem gives you a window into the kernel/drivers </li></ul><ul><li>Kgdb uses familiar gdb technology but at the kernel level </li></ul><ul><li>LEDs are a fast and easy way to get info out of the machine as well </li></ul><ul><li>Hardware JTAG debug tools may be available </li></ul><ul><ul><li>These can be invaluable if you can get one </li></ul></ul>