Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1


Published on

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1

  1. 1. Troubleshooting Linux Kernel Modules and Device Drivers Mike Anderson Chief Scientist The PTR Group, Inc. [email_address] Source:
  2. 2. What We’ll Talk About <ul><li>How do errors show up in the kernel? </li></ul><ul><li>Watching kernel/user-space interaction via strace </li></ul><ul><li>Debugging with printk </li></ul><ul><li>Using the /proc file system </li></ul><ul><li>Using the kgdb debugger </li></ul><ul><li>Debugging with hardware ala LEDs or a JTAG unit </li></ul>
  3. 3. Challenges of Kernel Debugging <ul><li>There are many features that can make kernel debugging especially difficult </li></ul><ul><ul><li>Optimizing compilers can rearrange code </li></ul></ul><ul><ul><ul><li>Instruction pointer seems to jump around </li></ul></ul></ul><ul><ul><li>The use of the MMU can obfuscate addresses </li></ul></ul><ul><ul><ul><li>Physical vs. virtual addresses </li></ul></ul></ul><ul><ul><li>Startup code is particularly difficult to debug because of its closeness to the “metal” </li></ul></ul><ul><ul><li>No equivalent for the user-space gdbserver for drivers </li></ul></ul><ul><li>Early kernel debugging may require hardware assistance </li></ul>
  4. 4. Device Drivers/Kernel Modules <ul><li>Assuming that your kernel is otherwise working, most of the problems that you’ll encounter are related to device drivers </li></ul><ul><ul><li>Drivers can either be statically linked or dynamically loaded to the kernel </li></ul></ul><ul><li>A dynamically loaded driver takes the form of a kernel module </li></ul><ul><ul><li>Can be dynamically loaded and unloaded at kernel run time </li></ul></ul><ul><ul><ul><li>Frequently handled by daemons such as udev </li></ul></ul></ul>
  5. 5. Example Module <ul><li>#include <linux/module.h> </li></ul><ul><li>#include <linux/init.h> </li></ul><ul><li>#include <linux/kernel.h> </li></ul><ul><li>MODULE_LICENSE(&quot;GPL and additional rights&quot;); </li></ul><ul><li>MODULE_AUTHOR( “” ); </li></ul><ul><li>MODULE_DESCRIPTION( “My first driver!” ); </li></ul><ul><li>int __init mymodule_init_module(void) { </li></ul><ul><li>printk(KERN_DEBUG “mymodule_init_module() called, &quot;); </li></ul><ul><li>return 0; </li></ul><ul><li>} </li></ul><ul><li>void __exit mymodule_cleanup_module(void) { </li></ul><ul><li>printk(KERN_DEBUG “mymodule_cleanup_module() calledn&quot;); </li></ul><ul><li>} </li></ul><ul><li>module_init(mymodule_init_module); </li></ul><ul><li>module_exit(mymodule_cleanup_module); </li></ul>
  6. 6. When Things go Wrong… <ul><li>Problems in device drivers typically manifest themselves in one of three ways </li></ul><ul><ul><li>Kernel panic </li></ul></ul><ul><ul><ul><li>Fatal to the system </li></ul></ul></ul><ul><ul><li>Kernel oops </li></ul></ul><ul><ul><ul><li>Near fatal to the system </li></ul></ul></ul><ul><ul><li>Hardware just doesn’t work correctly </li></ul></ul><ul><ul><ul><li>Could be fatal to you! </li></ul></ul></ul>Source:
  7. 7. Kernel Panic <ul><li>When the Linux kernel determines that a fatal error has occurred, and no recovery is possible, it “panics” </li></ul><ul><ul><li>Frequently, an exception in an interrupt context </li></ul></ul><ul><li>Panic outputs a message to the console </li></ul><ul><ul><li>The output will help you find the source of the bug </li></ul></ul><ul><li>Typically results in a system reboot on an embedded Linux target </li></ul><ul><ul><li>Or, blinking keyboard LEDs on some desktop versions of Linux </li></ul></ul>Source:
  8. 8. Sample Panic Output <ul><li>EIP: 0060: [<c02b2516>] Not tainted VLI </li></ul><ul><li>EFLAGS: 00010286 (2.6.18) </li></ul><ul><li>EIP is at nf_queue+0x16/0x240 </li></ul><ul><li>eax: 00000000 ebx: 00000001 ecx: 00000000 edx: 00000002 </li></ul><ul><li>esi: dece0920 edi: c03a3aa8 ebp: c0277e70 esp: cf165e78 </li></ul><ul><li>ds: 007b es: 007b ss: 0068 </li></ul><ul><li>Process sh (pid:5278, threadinfo=cf164000 task=dc5b4ab0) </li></ul><ul><li>Stack: cf165f00 dae29000 00000000 c0277e70 00000001 cf165f00 c03a3aa8 c0277e70 </li></ul><ul><li>c02b1f7c cf165f00 dece0920 00000002 00000001 dae29000 00000000 c0277e70 </li></ul><ul><li>00000000 dece0920 00000000 da641b40 da54bcde cc318abc c0277c10 00000002 </li></ul><ul><li>Call Trace: </li></ul><ul><li>[<c0277e70>] ip_local_deliver_finish+0x0/0x230 </li></ul><ul><li>[<c0277e70>] ip_local_deliver_finish+0x0/0x230 </li></ul><ul><li>… deleted for space … </li></ul><ul><li>[<c012caa6>] do_softirq+0x26/0x30 </li></ul><ul><li>[<c010528e>] do_IRQ+0x1e/0x30 </li></ul><ul><li>[<c0103ada>] common_interrupt+0x1a/0x20 </li></ul><ul><li>Code: c0 75 ec e9 bd e5 e6 ff 8d b6 00 00 00 00 8d bc 27 00 00 00 00 55 57 56 53 </li></ul><ul><li>83 ec 10 8b 54 24 2c 8b 74 24 28 8b 04 9b c0 42 3a c0 <8b> 38 85 ff 0f 84 96 01 </li></ul><ul><li>00 00 86 44 24 2c 8b 7c 24 2c c7 44 24 </li></ul><ul><li><0>Kernel panic - not syncing: Fatal exception in interrupt </li></ul>
  9. 9. Kernel oops <ul><li>An oops message is displayed when a recoverable error has occurred in kernel space: </li></ul><ul><ul><li>Access to bad address, e.g., through a NULL pointer </li></ul></ul><ul><ul><li>Illegal or invalid instruction </li></ul></ul><ul><ul><li>Etc… </li></ul></ul><ul><li>The calling user process is killed </li></ul><ul><ul><li>The system should be considered unstable at this point </li></ul></ul><ul><li>The oops message displays: </li></ul><ul><ul><li>The state of the processor at the time of the fault, including registers and address of faulting instruction </li></ul></ul><ul><ul><li>function call stack traceback </li></ul></ul><ul><li>The addresses are replaced with symbols if the kallsyms kernel configuration option is selected at kernel compile time </li></ul>
  10. 10. Example Oops <ul><li>Modules linked in: arc4 ecb crypto_blkcipher ieee80211_crypt_wep vfat fat ipt_MASQUERADE iptable_nat nf_nat bridge bnep rfcomm l2cap bluetooth ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi sunrpc ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables cpufreq_ondemand acpi_cpufreq fuse loop dm_multipath ipv6 ppdev parport_pc parport video 8139cp output ac battery firewire_ohci fglrx(P) firewire_core crc_itu_t 8139too mii ipw2200 ieee80211 ieee80211_crypt snd_intel8x0m asus_laptop snd_intel8x0 button snd_seq_dummy snd_ac97_codec ac97_bus snd_seq_oss snd_seq_midi_event joydev snd_seq pcspkr serio_raw snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm usb_storage iTCO_wdt snd_timer snd iTCO_vendor_support soundcore snd_page_alloc sr_mod cdrom sg dm_snapshot dm_zero dm_mirror dm_mod pata_acpi ata_generic ata_piix libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode] </li></ul><ul><li>Pid: 5488, comm: Xorg Tainted: P ( #1) </li></ul><ul><li>EIP: 0060:[<c04d038d>] EFLAGS: 00213246 CPU: 0 </li></ul><ul><li>EIP is at task_has_capability+0x48/0x76 </li></ul><ul><li>EAX: 00000030 EBX: f6852030 ECX: f6a8af28 EDX: 00000000 </li></ul><ul><li>ESI: f36e50e0 EDI: f354fec8 EBP: f354fed4 ESP: f354fe84 </li></ul><ul><li>DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 </li></ul><ul><li>Process Xorg (pid: 5488, ti=f354f000 task=f35f0e90 task.ti=f354f000) </li></ul><ul><li>Stack: c06d48b5 f6852030 f35f0e90 00000003 f35f0e90 f6852030 00000000 00000000 </li></ul><ul><li>00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 </li></ul><ul><li>00000000 f6852030 f35f0e90 f66f8000 f354fee4 c04d03da f35f0e90 f8d52580 </li></ul>
  11. 11. Example Oops part 2 <ul><li>Call Trace: </li></ul><ul><li>[<c04d03da>] ? selinux_capable+0x1f/0x23 </li></ul><ul><li>[<c04cbfe9>] ? security_capable+0xc/0xe </li></ul><ul><li>[<c042de97>] ? __capable+0xb/0x22 </li></ul><ul><li>[<f8c2d560>] ? firegl_version+0x0/0x1b0 [fglrx] </li></ul><ul><li>[<c042debe>] ? capable+0x10/0x12 </li></ul><ul><li>[<f8c2d427>] ? firegl_ioctl+0xe7/0x220 [fglrx] </li></ul><ul><li>[<c062aa49>] ? mutex_lock+0x1d/0x2d </li></ul><ul><li>[<c04d0c45>] ? file_has_perm+0x7c/0x85 </li></ul><ul><li>[<f8c24685>] ? ip_firegl_ioctl+0xe/0x10 [fglrx] </li></ul><ul><li>[<c048d2c8>] ? vfs_ioctl+0x50/0x69 </li></ul><ul><li>[<c048d51a>] ? do_vfs_ioctl+0x239/0x24c </li></ul><ul><li>[<c04d0ddb>] ? selinux_file_ioctl+0xa8/0xab </li></ul><ul><li>[<c048d56d>] ? sys_ioctl+0x40/0x5b </li></ul><ul><li>[<c0405bf2>] ? syscall_call+0x7/0xb </li></ul><ul><li>====================== </li></ul><ul><li>======================Code: 00 89 d0 f3 ab 8b 4d b8 89 d8 b2 04 c1 f8 05 c6 45 bc 03 89 5d c4 89 4d c0 74 16 48 b2 45 74 11 53 68 b5 48 6d c0 e8 11 f3 15 00 <0f> 0b 59 5b eb fe 8b 46 08 83 e3 1f 0f b7 f2 8d 55 bc 88 d9 52 </li></ul><ul><li>EIP: [<c04d038d>] task_has_capability+0x48/0x76 SS:ESP 0068:f354fe84 </li></ul><ul><li>---[ end trace 9451402cb10d9c54 ]--- </li></ul>
  12. 12. Enabling Symbolic Kernel Output
  13. 13. Module Debugging Techniques <ul><li>Examine the interaction with the kernel via strace </li></ul><ul><li>The next line of defense is printk </li></ul><ul><ul><li>There may be additional output you’re not seeing </li></ul></ul><ul><li>Next, we can try adding /proc file system entries </li></ul><ul><ul><li>Instrument the driver for debugging </li></ul></ul><ul><li>Enable source debugging via kgdb </li></ul><ul><li>Using hardware debuggers and “blinky lights” </li></ul>
  14. 14. Using strace to Watch System Calls <ul><li>When debugging what appears to be a kernel-space error, it can be helpful to watch the system calls that are made from user-space </li></ul><ul><ul><li>See what events lead to the error </li></ul></ul><ul><li>strace displays all system calls made by a program </li></ul><ul><ul><li>Can display timestamp information per system call as well </li></ul></ul>
  15. 15. Using strace to Watch System Calls #2 <ul><li>strace displays each system call’s arguments and return values </li></ul><ul><ul><li>string arguments are printed – very helpful! </li></ul></ul><ul><ul><li>errno values displayed symbolically </li></ul></ul><ul><li>The program being traced runs normally </li></ul><ul><ul><li>Not under control of a debugger </li></ul></ul><ul><ul><ul><li>No need to specially compile the user application </li></ul></ul></ul><ul><li>You can attach to a running program </li></ul><ul><ul><li>And trace forked applications as well… </li></ul></ul>
  16. 16. Example strace Output <ul><li>/ # strace ls /dev/labdev </li></ul><ul><li>execve(&quot;/bin/ls&quot;, [&quot;ls&quot;, &quot;/dev/labdev&quot;], [/* 8 vars */]) = 0 </li></ul><ul><li>fcntl64(0, F_GETFD) = 0 </li></ul><ul><li>fcntl64(1, F_GETFD) = 0 </li></ul><ul><li>fcntl64(2, F_GETFD) = 0 </li></ul><ul><li>geteuid() = 0 </li></ul><ul><li>getuid() = 0 </li></ul><ul><li>getegid() = 0 </li></ul><ul><li>getgid() = 0 </li></ul><ul><li>brk(0) = 0x1028ad68 </li></ul><ul><li>brk(0x1028bd68) = 0x1028bd68 </li></ul><ul><li>brk(0x1028c000) = 0x1028c000 </li></ul><ul><li>ioctl(1, TIOCGWINSZ or TIOCGWINSZ, {ws_row=0, ws_col=0, ws_xpixel=0, ws_ypixel=0 </li></ul><ul><li>ioctl(1, TCGETS or TCGETS, {B9600 opost isig icanon echo ...}) = 0 </li></ul><ul><li>ioctl(1, TCGETS or TCGETS, {B9600 opost isig icanon echo ...}) = 0 </li></ul><ul><li>lstat(&quot;/dev/labdev&quot;, {st_mode=S_IFCHR|0644, st_rdev=makedev(254, 0), ...}) = 0 </li></ul><ul><li>open(&quot;/etc/localtime&quot;, O_RDONLY) = -1 ENOENT (No such file or directory) </li></ul><ul><li>lstat(&quot;/dev/labdev&quot;, {st_mode=S_IFCHR|0644, st_rdev=makedev(254, 0), ...}) = 0 </li></ul><ul><li>fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(4, 64), ...}) = 0 </li></ul><ul><li>ioctl(1, TCGETS or TCGETS, {B9600 opost isig icanon echo ...}) = 0 </li></ul><ul><li>mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x300 </li></ul><ul><li>write(1, &quot;33[1;35m/dev/labdev33[0mn&quot;, 23/dev/labdev </li></ul><ul><li>) = 23 </li></ul><ul><li>munmap(0x30000000, 4096) = 0 </li></ul><ul><li>exit(0) = ? </li></ul>
  17. 17. Debugging with printk <ul><li>printk debugging is the debug method preferred by Linus </li></ul><ul><ul><li>At least, according to his email traffic… </li></ul></ul><ul><li>Insert messages to be displayed at points of interest in kernel-space code </li></ul><ul><ul><li>E.g., printk(KERN_INFO “my_x= %dn”, my_x); </li></ul></ul><ul><ul><li>Printk works like printf does in user-space except that printk can only print integers, strings & addresses </li></ul></ul><ul><li>printk can also be called from within ISRs </li></ul><ul><li>Messages can have “importance” settings that allow filtering </li></ul><ul><ul><li>Importance is set by prepending a 3-character string to the output message: “< n >” </li></ul></ul>
  18. 18. Debugging with printk #2 <ul><li>The messages are placed in a circular buffer that can be retrieved post mortem if needed </li></ul><ul><li>The “importance” string that is prepended to the printk message can be found in include/linux/kernel.h : </li></ul><ul><li>#define KERN_EMERG &quot;<0>&quot; /* system is unusable */ #define KERN_ALERT &quot;<1>&quot; /* action to be taken immediately */ #define KERN_CRIT &quot;<2>&quot; /* critical conditions */ #define KERN_ERR &quot;<3>&quot; /* error conditions */ #define KERN_WARNING &quot;<4>&quot; /* warning conditions */ #define KERN_NOTICE &quot;<5>&quot; /* normal but significant condition */ #define KERN_INFO &quot;<6>&quot; /* informational */ #define KERN_DEBUG &quot;<7>&quot; /* debug-level messages */ </li></ul>
  19. 19. Debugging with printk #3 <ul><li>Some level of control for printk output can be found in /proc/sys/kernel/printk </li></ul><ul><li>Let’s look at the following output: </li></ul><ul><ul><ul><li># cat /proc/sys/kernel/printk </li></ul></ul></ul><ul><ul><ul><li>7 4 1 7 </li></ul></ul></ul><ul><ul><li>This indicates: </li></ul></ul><ul><ul><ul><li>The console_loglevel is 7, so messages with importance of 0..6 will currently go to the console </li></ul></ul></ul><ul><ul><ul><li>The default message log level is 4, so messages that do not specify an importance are treated as level 4 </li></ul></ul></ul><ul><ul><ul><li>The minimum console log level is 1, so console_loglevel cannot be set to any value less than 1 </li></ul></ul></ul><ul><ul><ul><li>The default console log level is 7, so console_loglevel starts out set to 7 </li></ul></ul></ul>
  20. 20. Debugging with printk #4 <ul><li>You may control the console_loglevel by writing to /proc/sys/kernel/printk </li></ul><ul><li>To enable all printk messages with importance levels 0..7: </li></ul><ul><ul><ul><li># echo 8 > /proc/sys/kernel/printk </li></ul></ul></ul><ul><li>If the kernel command line contains the word “debug”, the console_loglevel starts with a value of 10 </li></ul>
  21. 21. Using the /proc File System <ul><li>Use /proc entries for driver instrumentation </li></ul><ul><li>It is possible to register “write” functions that allow us to dynamically modify values in the kernel (or drivers, or modules) </li></ul><ul><ul><li>Write values to the /proc file system entry </li></ul></ul><ul><li>Read /proc entries allow us to retrieve information from a running kernel entity </li></ul><ul><ul><li>Information is provided “live” </li></ul></ul><ul><li>Look at the driver source, there may already be /proc entries that can help you </li></ul>
  22. 22. Techniques for Source Debugging <ul><li>The two primary ways to provide source debugging in the Linux kernel are based on either kgdb or on the use of a hardware JTAG probe </li></ul><ul><li>Unfortunately, kgdb is not a standard feature of the kernel a/o </li></ul><ul><ul><li>You’ll have to patch your kernel to enable it </li></ul></ul><ul><li>Either technique will require the use of a kernel image that is compiled with debugging symbols </li></ul><ul><ul><li>Unless you like debugging in assembly language </li></ul></ul>
  23. 23. Compiling the Kernel with Debug Info <ul><li>This will increase the size of the debug kernel image by about 30% </li></ul><ul><li>However, you don’t need to load the debug version of the kernel </li></ul><ul><ul><li>Load the non-debug version to the target, but use the debug version for the debugger/JTAG probe </li></ul></ul><ul><li>Save off the vmlinux and file because these are used by the debugger or by you to find key addresses </li></ul><ul><ul><li>The (b)zImage can be loaded on the target as normal </li></ul></ul>
  24. 24. Enabling Debugging in the Kernel
  25. 25. Kernel gdb (kgdb) <ul><li>If you are using a stock kernel, kgdb is not included </li></ul><ul><ul><li>Linus doesn’t believe in a source debugger in the kernel </li></ul></ul><ul><ul><li>Many commercial Linux vendors do include it in their distributions though </li></ul></ul><ul><li>kgdb can be downloaded from: or and look for kgdb </li></ul><ul><ul><li>You’ll need to patch the kernel </li></ul></ul><ul><li>A new kgdb light is in the works for 2.6.26 </li></ul><ul><ul><li>Kgdb over the system console </li></ul></ul>
  26. 26. Enable kgdb in the Kernel
  27. 27. Kgdb Light in 2.6.26-rc8 <ul><li>Uses system console for I/O </li></ul>
  28. 28. kgdb Lash up <ul><li>kgdb supports debugging via the serial port </li></ul><ul><ul><li>The gdb debugger is running on a second machine using the vmlinux you compiled with debugging symbols </li></ul></ul><ul><ul><li>You attach to the system being debugged using gdb’s “target remote” command </li></ul></ul>Host Target Network Ethernet RS-232
  29. 29. Hardware-assisted Debugging <ul><li>There are a number of devices that can help with debugging </li></ul><ul><ul><li>LEDs, JTAGs, logic analyzers, oscilloscopes, bus analyzers and more </li></ul></ul><ul><li>These can range from a few cents to implement to several 10s of thousands of dollars </li></ul><ul><ul><li>You typically get what you pay for </li></ul></ul>
  30. 30. Debugging with LEDs <ul><li>Very simple: </li></ul><ul><ul><li>Blink on/off in various code sections under debug </li></ul></ul><ul><ul><li>Blink in sequences </li></ul></ul><ul><ul><li>Can display multiple-bit codes if multiple LEDs are available </li></ul></ul><ul><li>Very fast, little impact on run-time performance </li></ul><ul><ul><li>Adding LED debug code will likely not “make the problem go away” </li></ul></ul><ul><li>These may be the only option for debugging early x86 code </li></ul>
  31. 31. Debugging with LEDs, Caveats <ul><li>The LED(s) must be free for use </li></ul><ul><ul><li>Not tied in hardware to a network PHY or to displaying power status, for example </li></ul></ul><ul><li>LED are not very verbose </li></ul><ul><ul><li>You must decipher what the blinking means </li></ul></ul><ul><ul><li>Can be difficult to determine where you are in the code </li></ul></ul><ul><li>You can also attach an oscilloscope to the GPIO pins found on many processors for more information </li></ul>
  32. 32. Hardware Debuggers <ul><li>In the past, in-circuit emulators (ICE) where the debugger of choice </li></ul><ul><ul><li>You pulled the CPU, plugged the ICE in and plugged the CPU into the ICE </li></ul></ul><ul><ul><li>But, these where $80K+ each </li></ul></ul><ul><li>Logic analyzers are also good to have </li></ul><ul><ul><li>But, they are $35K+ for an empty mainframe </li></ul></ul><ul><ul><li>PC-based versions can be had for < $1K </li></ul></ul><ul><li>IEEE 1149.1 (JTAG) has become the debugger da jour </li></ul><ul><ul><li>JTAG uses a boundary-scan protocol </li></ul></ul><ul><ul><li>These range from $70 to $20K depending on model and features </li></ul></ul><ul><ul><li>At a minimum, a JTAG is really a “must-have” for firmware and board bring up </li></ul></ul>
  33. 33. Debugging with a JTAG Probe <ul><li>Debugging with a JTAG unit is much less involved than using kgdb </li></ul><ul><ul><li>Compile the kernel with debugging enabled </li></ul></ul><ul><ul><li>No need to patch the kernel for kgdb </li></ul></ul><ul><ul><li>Assumes your platform supports a JTAG interface </li></ul></ul><ul><li>You connect to the JTAG unit using whatever technique your JTAG probe requires </li></ul><ul><ul><li>Your JTAG GUI is dependent on the vendor </li></ul></ul><ul><ul><li>For those JTAG units that are gdb-aware, use the appropriate target remote commands </li></ul></ul>
  34. 34. Example JTAG Usage <ul><li>Connect the JTAG to the target and the host </li></ul><ul><li>Start the host application to control the JTAG </li></ul><ul><ul><li>Reset the target and load the register configuration settings to the JTAG unit </li></ul></ul><ul><li>Load code and enjoy! </li></ul><ul><ul><li>Useful for debugging drivers as well as bringing up new firmware and BSPs </li></ul></ul>
  35. 35. Summary <ul><li>“ Real developers” use printk – or at least Linus does  </li></ul><ul><li>Tools like strace allow you see the flow of execution </li></ul><ul><li>The /proc filesystem gives you a window into the kernel/drivers </li></ul><ul><li>Kgdb uses familiar gdb technology but at the kernel level </li></ul><ul><li>LEDs are a fast and easy way to get info out of the machine as well </li></ul><ul><li>Hardware JTAG debug tools may be available </li></ul><ul><ul><li>These can be invaluable if you can get one </li></ul></ul>