SlideShare a Scribd company logo
GDB Linux Kernel Awareness
Peter Griffin
Linaro Member Services
ENGINEERS AND DEVICES
WORKING TOGETHER
“Engineers and Devices Working
Together”
… with free & open source tools!
ENGINEERS
AND DEVICES
WORKING
TOGETHER
Overview
● Rationale for GDB Linux Kernel Awareness
● Targets for debugging Linux with GDB
● Upstream GDB to debug the Linux kernel
● What is the ST landing team doing?
● What is “Linux awareness”?
● Approach Taken
● Progress so far
● Next steps
● Alternative approaches
● Demo
● Q&A
ENGINEERS AND DEVICES
WORKING TOGETHER
● Many of us need to debug the Linux kernel
● Proprietary tools like Lauterbach Trace32 and ARM DS-5 are $$$
● Open source debuggers like GDB lack ‘kernel awareness’ features found in
proprietary tools.
ST LT Mission
To provide a better debugging experience when using upstream GDB to debug
Linux kernel via JTAG / OpenOCD, or QEMU.
Rationale for GDB Linux Kernel Awareness
ENGINEERS AND DEVICES
WORKING TOGETHER
Ways to Debug Linux Kernel with GDB
● GDB client using the GDB remote protocol
a. Connection to a KGDB stub in a running kernel
b. Connect to a QEMU stub running a virtual kernel environment
c. Connect to a GDB remote compliant JTAG probe, such as OpenOCD
● GDB session on host
a. Core Dump file
b. UML Kernel
ENGINEERS AND DEVICES
WORKING TOGETHER
● Debug stub in the kernel compliant with GDB remote protocol
○ Enable with CONFIG_KGDB
+ Already supported on many platforms
+ All kernel threads enumerated in GDB (via GDB remote packets)
- Requires cooperation between debugger and kernel stub
- Less suitable for serious crashes
- Isn’t enabled on production systems
- Requires enough support for serial or ethernet
Targets: KGDB with GDB
Host
GDB
Target
Linux
KGDB
GDB remote
serial/ethernet
ENGINEERS AND DEVICES
WORKING TOGETHER
Advantages
+ QEMU is open source and has GDB remote stub
+ No “real” hardware required
+ Good for testing generic kernel code, on many architectures
+ Good environment for developing GDB Linux awareness extensions
Disadvantages
- If your bug is SoC or board related it is unlikely to be useful
.
Targets: QEMU and GDB
Dev Host
GDB client
Linux
GDB stub
Qemu
QEMU and GDB (Example)
> qemu-system-arm -M vexpress-a9 -cpu cortex-a9 -m 256M -nographic -kernel ./zImage -append 'console=ttyAMA0,115200
rw nfsroot=10.0.2.2:/opt/debian/wheezy-armel-rootfs,v3 ip=dhcp' -dtb ./vexpress-v2p-ca9.dtb -s
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.3.0-rc3-00008-g6e751b6-dirty (griffinp@X1-Carbon) (gcc version 4.9.3 20141031
(prerelease) (Linaro GCC 2014.11) ) #317 SMP Tue Sep 29 13:20:36 BST 2015
[ 0.000000] CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: V2P-CA9
[..]
[ 98.288845] VFS: Unable to mount root fs via NFS, trying floppy.
[ 98.290767] VFS: Cannot open root device "(null)" or unknown-block(2,0): error -6
[ 98.291132] Please append a correct "root=" boot option; here are the available partitions:
[ 98.291681] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[ 98.292215] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-00008-g6e751b6-dirty #317
arm-linux-gnueabi-gdb vmlinux
(gdb) target remote :1234
Remote debugging using :1234
__loop_delay () at ../arch/arm/lib/delay-loop.S:47
47 subs r0, r0, #1
(gdb) info threads
Id Target Id Frame
* 1 Thread 1 (CPU#0 [running]) __loop_delay () at ../arch/arm/lib/delay-loop.S:47
(gdb) bt
#0 __loop_delay () at ../arch/arm/lib/delay-loop.S:47
#1 0xc02cbbe0 in panic (fmt=0xc0c340a8 "VFS: Unable to mount root fs on %s") at ../kernel/panic.c:201
#2 0xc0dad19c in mount_block_root (name=0xc0c34148 "/dev/root", flags=32768) at ../init/do_mounts.c:421
#3 0xc0dad354 in mount_root () at ../init/do_mounts.c:541
#4 0xc0dad4b4 in prepare_namespace () at ../init/do_mounts.c:600
#5 0xc0dace6c in kernel_init_freeable () at ../init/main.c:1026
#6 0xc09ca2f4 in kernel_init (unused=<optimised out>) at ../init/main.c:936
+ OpenOCD is open source
+ Supports GDB remote protocol
+ Supports many ARM/MIPS CPUs
+ Supports many FTDI based cheap
JTAG probes
Targets: JTAG - External Debug
Host
Target System
GDB Client
Telnet
OpenOCD
telnet
gdbserver
JTAG JTAG
http://openocd.org
http://elinux.org/JTAG
ENGINEERS AND DEVICES
WORKING TOGETHER
● Allows external access to dedicated SoC debug hardware
● Flow control via on-chip hardware breakpoints and watchpoints
○ Access to on-chip performance monitors
○ Control reset and initialization of target
● Download code directly into RAM or flash
● Manipulate MMU and caches whilst target is halted
● Allows debugging when other techniques can’t be used e.g.
○ Ability to load code on bare board & debug from reset vector
○ Debug during late and early parts of system suspend & resume
○ Debug IRQ handlers
○ Inspect very broken / corrupt systems e.g. when KGDB stub no longer functional.
Targets: JTAG - Why External Debugging?
GDB Remote Compliant JTAG - e.g. OpenOCD
Compile it
> git clone ssh://git@git.linaro.org:/people/peter.griffin/openocd-code
> cd openocd-code
> git submodule update --init --recursive
> autoreconf -iv
> export PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig/
> ./configure --enable-ftdi
> make
> make install
Run it
./openocd -f ./tcl/interface/ftdi/flyswatter2.cfg -f ./tcl/target/hi6220.cfg
Open On-Chip Debugger 0.9.0-dev-00241-gd356c86-dirty (2015-07-10-08:20)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.sourceforge.net/doc/doxygen/bugs.html
adapter speed: 10 kHz
Info : session transport was not selected, defaulting to JTAG
jtag_ntrst_delay: 100
trst_and_srst combined srst_gates_jtag trst_push_pull srst_open_drain
connect_deassert_srst
hi6220.cpu
Info : clock speed 10 kHz
Info : JTAG tap: hi6220.dap tap/device found: 0x5ba00477 (mfg: 0x23b,
part: 0xba00, ver: 0x5)
Info : hi6220.cpu: hardware has 6 breakpoints, 4 watchpoints
Example telnet interface
telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
> halt
number of cache level 2
cache l2 present :not supported
hi6220.cpu cluster 0 core 0 multi core
target state: halted
target halted in ARM64 state due to debug-request,
current mode: EL2H
cpsr: 0x800003c9 pc: 0x3ef7e908
MMU: disabled, D-Cache: disabled, I-Cache: disabled
Example gdb interface trace
> aarch64-linux-gnu-gdb
(gdb) target remote :3333
(gdb) add-symbol-file build-hikey/u-boot 0x3ef47000
(gdb) hbreak do_version
Hardware assisted breakpoint 1 at 0x3ef4cc80: file
../common/cmd_version.c, line 19.
(gdb) c
Continuing.
Breakpoint 1, do_version (cmdtp=0x3ef8c0c0, flag=0,
argc=1, argv=0x3e746850) at ../common/cmd_version.c:19
19 {
(gdb) bt
#0 do_version (cmdtp=0x3ef8c0c0, flag=0, argc=1,
argv=0x3e746850) at ../common/cmd_version.c:19
#1 0x000000003ef604a8 in cmd_call (argv=0x3e746850,
argc=1,
ENGINEERS AND DEVICES
WORKING TOGETHER
KGDB JTAG
QEMU
Good Better
Core Dumps
ENGINEERS AND DEVICES
WORKING TOGETHER
● CONFIG_PROC_KCORE
○ sudo gdb vmlinux /proc/kcore
○ Virtual ELF core file of live kernel
○ No modifications can be made
● CONFIG_PROC_VMCORE
○ /proc/vmcore
○ Used in conjunction with kexec, kdump
and the crash utility from RedHat
○ py-crash, and libkdumpfile support
coming to GDB from SUSE
Targets: Core Dumps
https://en.wikipedia.org/wiki/Kdump_(Linux) @V4711
https://www.suse.com/documentation/sles-12/book_sle_tuning/data/part_tuning_dumps.html
ENGINEERS AND DEVICES
WORKING TOGETHER
So you have upstream GDB debugging the Linux
Kernel via JTAG, or QEMU…
but compared to $$$ commercial tools or ST GDB
it can be disappointing...
Upstream GDB to Debug the Linux Kernel
ENGINEERS AND DEVICES
WORKING TOGETHER
● (gdb) info threads - Shows thread running on each physical CPU
- No visibility of other sleeping threads
- No visibility of user processes
arm-linux-gnueabi-gdb vmlinux
(gdb) target remote :1234
Remote debugging using :1234
__loop_delay () at ../arch/arm/lib/delay-loop.S:47
47 subs r0, r0, #1
(gdb) info threads
Id Target Id Frame
* 1 Thread 1 (CPU#0 [running]) __loop_delay () at
../arch/arm/lib/delay-loop.S:47
Eeek...only
1 thread!
Upstream GDB to Debug the Linux Kernel
ENGINEERS AND DEVICES
WORKING TOGETHER
● (gdb) bt - Works for CPU executing in kernel space. No unwinding for user tasks.
Upstream GDB to Debug the Linux Kernel
Example ‘bt’ executing in kernel space
arm-linux-gnueabi-gdb vmlinux
(gdb) target remote :1234
(gdb) bt
#0 __loop_delay () at
../arch/arm/lib/delay-loop.S:47
#1 0xc02cbbe0 in panic (fmt=0xc0c340a8
"VFS: Unable to mount root fs on %s") at
../kernel/panic.c:201
#2 0xc0dad19c in mount_block_root
(name=0xc0c34148 "/dev/root", flags=32768)
at ../init/do_mounts.c:421
#3 0xc0dad354 in mount_root () at
../init/do_mounts.c:541
#4 0xc0dad4b4 in prepare_namespace () at
../init/do_mounts.c:600
Example ‘bt’ executing in userspace
Program received signal SIGINT,
Interrupt.
0xb6e4978c in ?? ()
(gdb) bt
#0 0xb6e4978c in ?? ()
#1 0xb6e4a408 in ?? ()
Backtrace stopped: previous frame
identical to this frame (corrupt
stack?)
ENGINEERS AND DEVICES
WORKING TOGETHER
up / down / frame <num> - same as bt, will show physical CPU, and only
work if executing in kernel space.
thread <num> - switch physical CPUs
This could be better, more like $$$ proprietary JTAG tools.
Upstream GDB to Debug the Linux Kernel
ENGINEERS AND DEVICES
WORKING TOGETHER
● Provide the debugger with additional knowledge of the underlying O/S to
enable a better debugging experience.
○ Where is the Task List?
○ Where is in the Kernel Log Buffer?
○ What modules are loaded? Where?
● We split Linux Awareness into three areas
1. Task Awareness
○ Ability to report all task_structs as threads in GDB
○ Provides selectable GDB threads with GDB context commands (info threads,
thread <num>, frame <num>, up, down )
2. Loadable Module Support
○ Hooks for automatic symbol resolution when modules are inserted
3. OS Helper Commands
○ Interacting with the debugger to extract state from the O/S
What is Linux Awareness?
ENGINEERS AND DEVICES
WORKING TOGETHER
● Working to extend Linux Kernel Awareness in upstream GDB.
○ With support from the Linaro Toolchain Working Group.
● This is based on the C Linux Kernel Debugger (LKD) GDB plugin by ST; the aim
now is to extract and rework the features that are generally useful, and
upstream them.
● The team has engaged with the GDB and Linux kernel communities to develop
an approach.
● ST and the Linaro ST Landing Team are publicising this work to other Linaro
members who might have an interest in it.
What is the ST Landing Team Doing?
ENGINEERS AND DEVICES
WORKING TOGETHER
1129 lkd-arm.c ARM arch specific (virt to phys)
674 lkd.h
491 lkd-irqs.c
4953 lkd-main.c linux-aware target_ops, OS helpers
2456 lkd-modules.c modules target_so_ops
79 lkd-modules.h
1666 lkd-process.c kthread awareness
77 lkd-process.h
220 lkd-proxy.c
2076 lkd-sh4.c SH4 specific
13874 total
Overview of ST LKD C Patchset
ENGINEERS AND DEVICES
WORKING TOGETHER
● We have 3 options
1. Scripting extension to GDB (Python / Guile)
2. Awareness in GDB stub
3. C extension to GDB
● Each of these approaches has advantages and disadvantages
Where to Put the “Awareness”?
GDB
Client
GDB stub
KGDB /
OpenOCD
Python plugin
Guile plugin
Am I OS
aware?
...or am I
OS
aware?
or maybe
I’m OS
aware?
gdbremote
ENGINEERS AND DEVICES
WORKING TOGETHER
● You can extend GDB using Python. See:
https://sourceware.org/gdb/onlinedocs/gdb/Python.html#Python
● What is exposed via Python interface has been steadily improving in GDB
○ Add custom commands
○ Implement pretty printers
○ Frame filters, frame decorators and much more.
● The major advantage of implementing “Linux awareness” in Python is: the code
can live in the kernel source tree.
+ Extensions should evolve with the kernel source
+ No cross-dependencies between kernel & GDB
Extending GDB with Python
ENGINEERS AND DEVICES
WORKING TOGETHER
Jan Kiszka from Siemens has already implemented some Python Linux awareness
extensions and established this route.
● See Docs @ Documentation/gdb-kernel-debugging.txt
● See Python @ scripts/gdb/*
● Enable CONFIG_GDB_SCRIPTS
● Provides a series of custom Linux commands and functions
● Can be easily tested using QEMU or OpenOCD
Python GDB Linux Awareness “Extensions”
ENGINEERS AND DEVICES
WORKING TOGETHER
Linux GDB Python Extensions in v4.6
(gdb) apropos lx
function lx_current -- Return current task
function lx_module -- Find module by name and return the module variable
function lx_per_cpu -- Return per-cpu variable
function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
function lx_thread_info -- Calculate Linux thread_info from task variable
lx-cmdline -- Report the Linux Commandline used in the current kernel
lx-dmesg -- Print Linux kernel log buffer
lx-list-check -- Verify a list consistency
lx-lsmod -- List currently loaded modules
lx-ps -- Dump Linux tasks
lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
lx-version -- Report the Linux Version of the current kernel
ENGINEERS AND DEVICES
WORKING TOGETHER
● Migrating LKD C-based OS helper commands into Python.
○ C code then retired from LKD, reducing amount of ‘out of tree’ LKD code.
○ Python code when merged will evolve with the kernel source tree.
● Some commands such as dmesg already existed upstream
● First ST LT Python OS helper commands merged into Linux kernel.
○ lx-cmdline and lx-version in v4.6.
○ lx-mount, lx-iomem, lx-ioports in v4.7.
lx-cmdline -- Report the Linux Commandline used in the current kernel
lx-version -- Report the Linux Version of the current kernel
lx-iomem -- Identify the IO memory resource locations defined by the kernel
lx-ioports -- Identify the IO port resource locations defined by the kernel
ST LT Involvement: OS Helpers
ENGINEERS AND DEVICES
WORKING TOGETHER
(gdb) lx-cmdline
root=/dev/nfs nfsroot=10.0.2.2:/opt/debian/wheezy-armel-rootfs,tcp,v3 rw ip=dhcp
mem=1024M raid=noautodetect rootwait console=ttyAMA0,38400n8 devtmpfs.mount=0
(gdb) lx-version
Linux version 4.7.0-rc3-00003-g5d518fc-dirty
(griffinp@griffinp-ThinkPad-X1-Carbon-2nd) (gcc version 4.9.3 20141031 (prerelease)
(Linaro GCC 2014.11) ) #4 SMP Thu Jul 14 12:54:03 BST 2016
Example Command Output
ENGINEERS AND DEVICES
WORKING TOGETHER
Example Output
(gdb) lx-iomem
10000008-1000000b : dat
10000048-1000004b : dat
1000004c-1000004f : dat
100000a0-100000ab : vexpress-syscfg
10004000-10004fff : /smb@04000000/motherboard/iofpga@7,00000000/aaci@04000
10005000-10005fff : /smb@04000000/motherboard/iofpga@7,00000000/mmci@05000
10005000-10005fff : /smb@04000000/motherboard/iofpga@7,00000000/mmci@05000
10006000-10006fff : /smb@04000000/motherboard/iofpga@7,00000000/kmi@06000
10006000-10006fff : kmi-pl050
10007000-10007fff : /smb@04000000/motherboard/iofpga@7,00000000/kmi@07000
10007000-10007fff : kmi-pl050
10009000-10009fff : /smb@04000000/motherboard/iofpga@7,00000000/uart@09000
10009000-10009fff : /smb@04000000/motherboard/iofpga@7,00000000/uart@09000
1000a000-1000afff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0a000
1000a000-1000afff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0a000
1000b000-1000bfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0b000
1000b000-1000bfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0b000
1000c000-1000cfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0c000
1000c000-1000cfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0c000
10011000-10011fff : /smb@04000000/motherboard/iofpga@7,00000000/timer@11000
10012000-10012fff : /smb@04000000/motherboard/iofpga@7,00000000/timer@12000
10017000-10017fff : /smb@04000000/motherboard/iofpga@7,00000000/rtc@17000
10017000-10017fff : rtc-pl031
1001f000-1001ffff : /smb@04000000/motherboard/iofpga@7,00000000/clcd@1f000
1001f000-1001ffff : clcd-pl11x
[..]
ENGINEERS AND DEVICES
WORKING TOGETHER
lx-fdtdump -- Output Flattened Device Tree header and dump FDT blob to a file
(gdb) lx-fdtdump
fdt_magic: 0xD00DFEED
fdt_totalsize: 0xC108
off_dt_struct: 0x38
off_dt_strings: 0x3804
off_mem_rsvmap: 0x28
version: 17
last_comp_version: 16
Dumped fdt to fdtdump.dtb
Allows further post-processing on development machine with standard DT tools.
e.g. fdtdump fdtdump.dtb | less
Allows diagnoses of what bootloader may have changed in DT to cause board not to
boot.
lx-fdtdump (Not Upstreamed Yet)
ENGINEERS AND DEVICES
WORKING TOGETHER
● Working to extract a minimal C-based kthread patchset for GDB.
● Minimal Linux kthread RFC posted
https://sourceware.org/ml/gdb-patches/2016-02/msg00826.html
● Not much feedback from GDB community.
Now preparing a refactored C implementation of linux-kthread for upstreaming.
Based on feedback from Yao Qi at Connect BKK16
(GDB maintainer & ARM TCWG assignee to Linaro).
Fixing style issues, re-factoring architecture specifics etc.
Patches work well with nommu Linux, OpenOCD and GDB.
Aim to submit refactored patches to GDB community by end of Q4 2016.
Kernel Thread Awareness: ST LT Approach
ENGINEERS AND DEVICES
WORKING TOGETHER
~/binutils-gdb/gdb$ git diff 4f80d1d --stat
gdb/Makefile.in | 4 +-
gdb/arm-linux-tdep.c | 98 ++++++
gdb/configure.tgt | 6 +-
gdb/doc/observer.texi | 5 +
gdb/linux-kthread.c | 1548
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++
gdb/linux-kthread.h | 215 ++++++++++++++
gdb/remote.c | 5 +
gdb/thread.c | 2 +-
8 files changed, 1878 insertions(+), 5 deletions(-)
Overview linux-kthread Refactored Patchset
ENGINEERS AND DEVICES
WORKING TOGETHER
JTAGUSB
GDB Linux Kernel Awareness: Components
GDB
Python
plugins
Kernel
crash
dump file
Python plugin
source in Linux
kernel source tree
(e.g. lx-dmesg)
OpenOCD
GDB Stub
QEMU
GDB Stub
On-chip Debug
USB-
JTAG
Dongle
gdb remote
serial protocol
Target
Board
Linux
Kernel
Binary
Linux
Kernel
Binary
linux-kthread
ENGINEERS AND DEVICES
WORKING TOGETHER
● Refactored minimal kthread patchset
● OpenOCD controlling stm32 target
● STM32F429 Discovery Board running v4.7-rc3 nommu Linux
Demonstrating:
● Correct enumeration of kernel threads
○ New threads enumerated by GDB
○ Old threads pruned from GDB thread list
● Python extensions merged for v4.7
○ lx-cmdline, lx-mounts, lx-iomem
Demo!
ENGINEERS AND DEVICES
WORKING TOGETHER
Virtual Memory and MMU control
● User can halt CPU at anytime and MMU could be configured for userspace
mapping.
○ Must be able to detect this, before attempting to read any kernel memory.
○ Updating kernel thread list in GDB requires us to to switch MMU to kernel mapping to read kernel
memory and parse task_struct list.
● Prototyping with QEMU to return ARM system coprocessor registers over GDB
remote using target XML.
○ Currently returns TTBR0/1, MIDR amd IDPFR0/1 registers.
○ Using target XML allows GDB stub to return appropriate registers needed for full system debug.
○ Using registers, allows access via register cache, and also from python scripts (unlike for
example custom GDB remote packet).
○ Support for target XML currently not implemented in OpenOCD.
GDB knowledge of CPU core
● To determine whether virtual memory manipulation is required.
Current Kthread Challenges and Considerations
ENGINEERS AND DEVICES
WORKING TOGETHER
(gdb) info registers
r0 0x1 1
[..]
sp 0xc1201fa00xc1201fa0 <init_thread_union+8096>
lr 0xc03087e8 -1070561304
pc 0xc031b0e8 0xc031b0e8 <cpu_v7_do_idle+8>
cpsr 0x60000093 1610612883
ttbr0_el1 0x0 0
ttbr1_el1 0x6020406a 1612726378
midr 0x410fc090 1091551376
id_pfr0 0x1031 4145
id_pfr1 0x11 17
CP Registers Example
ENGINEERS AND DEVICES
WORKING TOGETHER
● Complete and upstream C implementation of linux-kthread.
● Automated regression testing (LAVA, CI).
● Implement and upstream remaining LKD OS helper commands
○ lx-pgtable, pgmap, process_info (all remaining commands rely on virtual memory support).
● Review and test upstream Python implementation of loadable module support
versus LKD C implementation.
Known Bugs
● GDB APIs can behave differently between GDB remote stubs (e.g. QEMU versus
OpenOCD stub)
● Bug in GDB Python shim layer https://bugs.linaro.org/show_bug.cgi?id=2489
Next ST LT Steps
ENGINEERS AND DEVICES
WORKING TOGETHER
Other Potential Extensions
● Alternatively could implement Linux kthreads in Python
○ Extend GDB by means of Python scripts. Python again lives in kernel source.
○ Need to extend GDB’s Python API.
■ GDB thread object, target object, register object, symbol object need to be exposed via Python bindings to
get working.
■ Infrastructure could be re-used for other OS’s or RTOSs
○ Some prototyping and patches by ST LT to do this.
● Implement kthread for ARMv8 and X86 architectures.
○ Can be tested using QEMU, but needs also relies on aarch64 support in OpenOCD.
● Extend to support examination of kernel crash dump files.
○ Some GDB developers (in SUSE) working on this.
○ Could investigate how to combine this work with ours.
ENGINEERS AND DEVICES
WORKING TOGETHER
● ST’s LKD GDB plugin implements this feature.
○ Automatically pull and resolve userspace symbols for userspace processes from target root
filesystem.
gdb) bt
#0 context_switch (cookie=..., next=<optimised out>, prev=<optimised out>, rq=<optimised out>) at ../kernel/sched/core.c:2867
[..]
#6 0xc042bd30 in do_select (n=<optimised out>, fds=<optimised out>, end_time=<optimised out>) at ../fs/select.c:533
#7 0xc042bee8 in core_sys_select (n=11, inp=<unavailable>, outp=<unavailable>, exp=0x0, end_time=0xee075f70) at
../fs/select.c:603
#8 0xc042c280 in SYSC_select (tvp=<optimised out>, exp=<optimised out>, outp=<optimised out>, inp=<optimised out>,
n=<optimised out>) at ../fs/select.c:644
#9 SyS_select (n=11, inp=-1098552816, outp=0, exp=<unavailable>, tvp=-1098553000) at ../fs/select.c:626
#10 ret_fast_syscall () at ../arch/arm/kernel/entry-common.S:37 Kernel / User boundary
#11 0xb6f19a80 in elf_machine_rel (reloc=0x86, version=<optimised out>, reloc_addr_arg=0x3091969, sym=0x9db0,
map=0xbe856960) at ../ports/sysdeps/arm/dl-machine.h:475
#12 elf_dynamic_do_rel (lazy=0, relsize=<optimised out>, reladdr=<optimised out>, map=0xbe856960) at do-rel.h:120
#13 _dl_start (arg=0xbe8ab000) at rtld.c:547
#14 0xbe8ab000 in ?? ()
#15 0xbe8ab000 in ?? ()
Future Extensions: Backtrace to Userspace!
ENGINEERS AND DEVICES
WORKING TOGETHER
Q+A?
ENGINEERS
AND DEVICES
WORKING
TOGETHER
Thread Awareness in GDB Stub
ENGINEERS AND DEVICES
WORKING TOGETHER
● An example of putting “thread awareness” in the GDB stub
● OpenOCD already supports various rtos task awareness (eCos, ThreadX, FreeRTOS,
ChibiOS, embKernel, mqx and Linux)!
○ See openocd-code/src/rtos/linux.c
○ Disabled by default
● Enabled by adding “-rtos linux” to the target command in OpenOCD config file
○ target create $_TARGETNAME_2 cortex_a -chain-position
$_CHIPNAME.dap -dbgbase $_DAP_DBG2 -coreid 1 -rtos linux
● This has been tested with Tincantools’ Flyswatter JTAG and a U8500 (armv7) snowball
board.
● Required some hacking to get it working, but in the end I could successfully enumerate a
full thread list in GDB!
OpenOCD -rtos Linux Task Awareness
ENGINEERS AND DEVICES
WORKING TOGETHER
Advantages
+ No GDB changes required (threads returned via gdbremote packets).
+ All kernel threads can be enumerated in GDB (once working)
+ Can be used in conjunction with python extensions (e.g. lx-dmesg etc)
+ OpenOCD already has mmu virt to phys translation code for many CPUs
Disadvantages
- Awareness in GDB means thread awareness re-implementing for each stub: Qemu, OpenOCD...
- No good for debugging kernel dumps
- Tight coupling of code parsing kernel data structures in OpenOCD
- No debug information available in OpenOCD.
- No way (I know of) to find field offsets via gdbremote protocol.
- Currently this means task_struct, thread_info and mm_struct field offsets require OpenOCD to
be recompiled for the kernel you wish to debug :(
Improvements
○ gdbremote protocol could be extended
○ At a minimum task structure field offsets configurable at runtime, instead of requiring recompile!
OpenOCD -rtos Task Awareness
ENGINEERS AND DEVICES
WORKING TOGETHER
/* gdb script to update the header file according to kernel version and build option
before executing function awareness kernel symbol must be loaded : symbol vmlinux
define awareness
set logging off
set logging file linux_header.h
set logging on
printf "#define QAT %pn",&((struct task_struct *)(0))->stack
set $a=&((struct list_head *)(0))->next
set $a=(int)$a+(int)&((struct task_struct *)(0))->tasks
printf "#define NEXT %pn",$a
printf "#define COMM %pn",&((struct task_struct *)(0))->comm
printf "#define MEM %pn",&((struct task_struct *)(0))->mm
printf "#define ONCPU %pn",&((struct task_struct *)(0))->on_cpu
printf "#define PID %pn",&((struct task_struct *)(0))->pid
*/
OpenOCD linux-header.h
#define QAT 0x4
#define NEXT 0x1b0
#define COMM 0x2d4
#define MEM 0x1cc
#define ONCPU 0x18
#define PID 0x1f4
#define CPU_CONT 0x1c
#define PREEMPT 0x4
#define MM_CTX 0x160
Thank You
#LAS16
For further information: www.linaro.org
LAS16 keynotes and videos on: connect.linaro.org

More Related Content

What's hot

Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Brendan Gregg
 
DPDK & Layer 4 Packet Processing
DPDK & Layer 4 Packet ProcessingDPDK & Layer 4 Packet Processing
DPDK & Layer 4 Packet Processing
Michelle Holley
 
Linux Initialization Process (1)
Linux Initialization Process (1)Linux Initialization Process (1)
Linux Initialization Process (1)
shimosawa
 
DPDK KNI interface
DPDK KNI interfaceDPDK KNI interface
DPDK KNI interface
Denys Haryachyy
 
Memory Bandwidth QoS
Memory Bandwidth QoSMemory Bandwidth QoS
Memory Bandwidth QoS
Rohit Jnagal
 
Ceph Day Beijing - SPDK for Ceph
Ceph Day Beijing - SPDK for CephCeph Day Beijing - SPDK for Ceph
Ceph Day Beijing - SPDK for Ceph
Danielle Womboldt
 
Dpdk performance
Dpdk performanceDpdk performance
Dpdk performance
Stephen Hemminger
 
Linux Memory Management
Linux Memory ManagementLinux Memory Management
Linux Memory Management
Ni Zo-Ma
 
Advanced Namespaces and cgroups
Advanced Namespaces and cgroupsAdvanced Namespaces and cgroups
Advanced Namespaces and cgroups
Kernel TLV
 
Linux Kernel Crashdump
Linux Kernel CrashdumpLinux Kernel Crashdump
Linux Kernel Crashdump
Marian Marinov
 
Linux Kernel - Virtual File System
Linux Kernel - Virtual File SystemLinux Kernel - Virtual File System
Linux Kernel - Virtual File System
Adrian Huang
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device Drivers
Satpal Parmar
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
SUSE Labs Taipei
 
The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecture
hugo lu
 
Kernel Module Programming
Kernel Module ProgrammingKernel Module Programming
Kernel Module Programming
Saurabh Bangad
 
Understanding of linux kernel memory model
Understanding of linux kernel memory modelUnderstanding of linux kernel memory model
Understanding of linux kernel memory model
SeongJae Park
 
Meet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracingMeet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracing
Viller Hsiao
 
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
Adrian Huang
 
Receive side scaling (RSS) with eBPF in QEMU and virtio-net
Receive side scaling (RSS) with eBPF in QEMU and virtio-netReceive side scaling (RSS) with eBPF in QEMU and virtio-net
Receive side scaling (RSS) with eBPF in QEMU and virtio-net
Yan Vugenfirer
 
semaphore & mutex.pdf
semaphore & mutex.pdfsemaphore & mutex.pdf
semaphore & mutex.pdf
Adrian Huang
 

What's hot (20)

Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
 
DPDK & Layer 4 Packet Processing
DPDK & Layer 4 Packet ProcessingDPDK & Layer 4 Packet Processing
DPDK & Layer 4 Packet Processing
 
Linux Initialization Process (1)
Linux Initialization Process (1)Linux Initialization Process (1)
Linux Initialization Process (1)
 
DPDK KNI interface
DPDK KNI interfaceDPDK KNI interface
DPDK KNI interface
 
Memory Bandwidth QoS
Memory Bandwidth QoSMemory Bandwidth QoS
Memory Bandwidth QoS
 
Ceph Day Beijing - SPDK for Ceph
Ceph Day Beijing - SPDK for CephCeph Day Beijing - SPDK for Ceph
Ceph Day Beijing - SPDK for Ceph
 
Dpdk performance
Dpdk performanceDpdk performance
Dpdk performance
 
Linux Memory Management
Linux Memory ManagementLinux Memory Management
Linux Memory Management
 
Advanced Namespaces and cgroups
Advanced Namespaces and cgroupsAdvanced Namespaces and cgroups
Advanced Namespaces and cgroups
 
Linux Kernel Crashdump
Linux Kernel CrashdumpLinux Kernel Crashdump
Linux Kernel Crashdump
 
Linux Kernel - Virtual File System
Linux Kernel - Virtual File SystemLinux Kernel - Virtual File System
Linux Kernel - Virtual File System
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device Drivers
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
 
The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecture
 
Kernel Module Programming
Kernel Module ProgrammingKernel Module Programming
Kernel Module Programming
 
Understanding of linux kernel memory model
Understanding of linux kernel memory modelUnderstanding of linux kernel memory model
Understanding of linux kernel memory model
 
Meet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracingMeet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracing
 
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
 
Receive side scaling (RSS) with eBPF in QEMU and virtio-net
Receive side scaling (RSS) with eBPF in QEMU and virtio-netReceive side scaling (RSS) with eBPF in QEMU and virtio-net
Receive side scaling (RSS) with eBPF in QEMU and virtio-net
 
semaphore & mutex.pdf
semaphore & mutex.pdfsemaphore & mutex.pdf
semaphore & mutex.pdf
 

Viewers also liked

LAS16-407: Internet of Tiny Linux (IoTL): the sequel.
LAS16-407: Internet of Tiny Linux (IoTL): the sequel.LAS16-407: Internet of Tiny Linux (IoTL): the sequel.
LAS16-407: Internet of Tiny Linux (IoTL): the sequel.
Linaro
 
LAS16-500: The Rise and Fall of Assembler and the VGIC from Hell
LAS16-500: The Rise and Fall of Assembler and the VGIC from HellLAS16-500: The Rise and Fall of Assembler and the VGIC from Hell
LAS16-500: The Rise and Fall of Assembler and the VGIC from Hell
Linaro
 
LAS16-203: Platform security architecture for embedded devices
LAS16-203: Platform security architecture for embedded devicesLAS16-203: Platform security architecture for embedded devices
LAS16-203: Platform security architecture for embedded devices
Linaro
 
LAS16-402: ARM Trusted Firmware – from Enterprise to Embedded
LAS16-402: ARM Trusted Firmware – from Enterprise to EmbeddedLAS16-402: ARM Trusted Firmware – from Enterprise to Embedded
LAS16-402: ARM Trusted Firmware – from Enterprise to Embedded
Linaro
 
From printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debuggingFrom printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debugging
The Linux Foundation
 
Advanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTngAdvanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTng
marckhouzam
 
Debugger Principle Overview & GDB Tricks
Debugger Principle Overview & GDB TricksDebugger Principle Overview & GDB Tricks
Debugger Principle Overview & GDB Tricks
dutor
 
LAS16-101: Efficient kernel backporting
LAS16-101: Efficient kernel backportingLAS16-101: Efficient kernel backporting
LAS16-101: Efficient kernel backporting
Linaro
 
LAS16-507: LXC support in LAVA
LAS16-507: LXC support in LAVALAS16-507: LXC support in LAVA
LAS16-507: LXC support in LAVA
Linaro
 
LAS16-106: GNU Toolchain Development Lifecycle
LAS16-106: GNU Toolchain Development LifecycleLAS16-106: GNU Toolchain Development Lifecycle
LAS16-106: GNU Toolchain Development Lifecycle
Linaro
 
Advanced Debugging with GDB
Advanced Debugging with GDBAdvanced Debugging with GDB
Advanced Debugging with GDB
David Khosid
 
LAS16-501: Introduction to LLVM - Projects, Components, Integration, Internals
LAS16-501: Introduction to LLVM - Projects, Components, Integration, InternalsLAS16-501: Introduction to LLVM - Projects, Components, Integration, Internals
LAS16-501: Introduction to LLVM - Projects, Components, Integration, Internals
Linaro
 
Q2.12: Debugging with GDB
Q2.12: Debugging with GDBQ2.12: Debugging with GDB
Q2.12: Debugging with GDB
Linaro
 
kubernetes, pourquoi et comment
kubernetes, pourquoi et commentkubernetes, pourquoi et comment
kubernetes, pourquoi et comment
Jean-Baptiste Claramonte
 
GDB Rocks!
GDB Rocks!GDB Rocks!
GDB Rocks!
Kent Chen
 
LAS16-TR04: Using tracing to tune and optimize EAS (English)
LAS16-TR04: Using tracing to tune and optimize EAS (English)LAS16-TR04: Using tracing to tune and optimize EAS (English)
LAS16-TR04: Using tracing to tune and optimize EAS (English)
Linaro
 
BKK16-304 The State of GDB on AArch64
BKK16-304 The State of GDB on AArch64BKK16-304 The State of GDB on AArch64
BKK16-304 The State of GDB on AArch64
Linaro
 
LAS16-307: Benchmarking Schedutil in Android
LAS16-307: Benchmarking Schedutil in AndroidLAS16-307: Benchmarking Schedutil in Android
LAS16-307: Benchmarking Schedutil in Android
Linaro
 
LAS16-406: Android Widevine on OP-TEE
LAS16-406: Android Widevine on OP-TEELAS16-406: Android Widevine on OP-TEE
LAS16-406: Android Widevine on OP-TEE
Linaro
 
Socionext ARMv8 server SoC chipset demo
Socionext ARMv8 server SoC chipset demoSocionext ARMv8 server SoC chipset demo
Socionext ARMv8 server SoC chipset demo
Linaro
 

Viewers also liked (20)

LAS16-407: Internet of Tiny Linux (IoTL): the sequel.
LAS16-407: Internet of Tiny Linux (IoTL): the sequel.LAS16-407: Internet of Tiny Linux (IoTL): the sequel.
LAS16-407: Internet of Tiny Linux (IoTL): the sequel.
 
LAS16-500: The Rise and Fall of Assembler and the VGIC from Hell
LAS16-500: The Rise and Fall of Assembler and the VGIC from HellLAS16-500: The Rise and Fall of Assembler and the VGIC from Hell
LAS16-500: The Rise and Fall of Assembler and the VGIC from Hell
 
LAS16-203: Platform security architecture for embedded devices
LAS16-203: Platform security architecture for embedded devicesLAS16-203: Platform security architecture for embedded devices
LAS16-203: Platform security architecture for embedded devices
 
LAS16-402: ARM Trusted Firmware – from Enterprise to Embedded
LAS16-402: ARM Trusted Firmware – from Enterprise to EmbeddedLAS16-402: ARM Trusted Firmware – from Enterprise to Embedded
LAS16-402: ARM Trusted Firmware – from Enterprise to Embedded
 
From printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debuggingFrom printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debugging
 
Advanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTngAdvanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTng
 
Debugger Principle Overview & GDB Tricks
Debugger Principle Overview & GDB TricksDebugger Principle Overview & GDB Tricks
Debugger Principle Overview & GDB Tricks
 
LAS16-101: Efficient kernel backporting
LAS16-101: Efficient kernel backportingLAS16-101: Efficient kernel backporting
LAS16-101: Efficient kernel backporting
 
LAS16-507: LXC support in LAVA
LAS16-507: LXC support in LAVALAS16-507: LXC support in LAVA
LAS16-507: LXC support in LAVA
 
LAS16-106: GNU Toolchain Development Lifecycle
LAS16-106: GNU Toolchain Development LifecycleLAS16-106: GNU Toolchain Development Lifecycle
LAS16-106: GNU Toolchain Development Lifecycle
 
Advanced Debugging with GDB
Advanced Debugging with GDBAdvanced Debugging with GDB
Advanced Debugging with GDB
 
LAS16-501: Introduction to LLVM - Projects, Components, Integration, Internals
LAS16-501: Introduction to LLVM - Projects, Components, Integration, InternalsLAS16-501: Introduction to LLVM - Projects, Components, Integration, Internals
LAS16-501: Introduction to LLVM - Projects, Components, Integration, Internals
 
Q2.12: Debugging with GDB
Q2.12: Debugging with GDBQ2.12: Debugging with GDB
Q2.12: Debugging with GDB
 
kubernetes, pourquoi et comment
kubernetes, pourquoi et commentkubernetes, pourquoi et comment
kubernetes, pourquoi et comment
 
GDB Rocks!
GDB Rocks!GDB Rocks!
GDB Rocks!
 
LAS16-TR04: Using tracing to tune and optimize EAS (English)
LAS16-TR04: Using tracing to tune and optimize EAS (English)LAS16-TR04: Using tracing to tune and optimize EAS (English)
LAS16-TR04: Using tracing to tune and optimize EAS (English)
 
BKK16-304 The State of GDB on AArch64
BKK16-304 The State of GDB on AArch64BKK16-304 The State of GDB on AArch64
BKK16-304 The State of GDB on AArch64
 
LAS16-307: Benchmarking Schedutil in Android
LAS16-307: Benchmarking Schedutil in AndroidLAS16-307: Benchmarking Schedutil in Android
LAS16-307: Benchmarking Schedutil in Android
 
LAS16-406: Android Widevine on OP-TEE
LAS16-406: Android Widevine on OP-TEELAS16-406: Android Widevine on OP-TEE
LAS16-406: Android Widevine on OP-TEE
 
Socionext ARMv8 server SoC chipset demo
Socionext ARMv8 server SoC chipset demoSocionext ARMv8 server SoC chipset demo
Socionext ARMv8 server SoC chipset demo
 

Similar to LAS16-403: GDB Linux Kernel Awareness

ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux Awareness
Peter Griffin
 
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Jian-Hong Pan
 
Share the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development BoardShare the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development Board
Jian-Hong Pan
 
Kernel development
Kernel developmentKernel development
Kernel development
Nuno Martins
 
Lecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports DevelopmentLecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports Development
Mohammed Farrag
 
Docker 활용법: dumpdocker
Docker 활용법: dumpdockerDocker 활용법: dumpdocker
Docker 활용법: dumpdocker
Jaehwa Park
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
libfetion
 
PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...
PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...
PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...
Faisal Akber
 
Linux Kernel Debugging
Linux Kernel DebuggingLinux Kernel Debugging
Linux Kernel Debugging
GlobalLogic Ukraine
 
Anatomy of ROCgdb presentation at gcc cauldron 2022
Anatomy of ROCgdb presentation at gcc cauldron 2022Anatomy of ROCgdb presentation at gcc cauldron 2022
Anatomy of ROCgdb presentation at gcc cauldron 2022
ssuser866937
 
XS Boston 2008 Debugging Xen
XS Boston 2008 Debugging XenXS Boston 2008 Debugging Xen
XS Boston 2008 Debugging Xen
The Linux Foundation
 
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
Mydbops
 
lab1-ppt.pdf
lab1-ppt.pdflab1-ppt.pdf
lab1-ppt.pdf
AbdelrahmanElewah1
 
Embedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debuggingEmbedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debugging
Anne Nicolas
 
Roll your own toy unix clone os
Roll your own toy unix clone osRoll your own toy unix clone os
Roll your own toy unix clone os
eramax
 
CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)
CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)
CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)
PROIDEA
 
Xen Debugging
Xen DebuggingXen Debugging
Xen Debugging
The Linux Foundation
 
Debugging of (C)Python applications
Debugging of (C)Python applicationsDebugging of (C)Python applications
Debugging of (C)Python applications
Roman Podoliaka
 
Java gpu computing
Java gpu computingJava gpu computing
Java gpu computing
Arjan Lamers
 
Porting Android
Porting AndroidPorting Android
Porting Android
Opersys inc.
 

Similar to LAS16-403: GDB Linux Kernel Awareness (20)

ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux Awareness
 
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021
 
Share the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development BoardShare the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development Board
 
Kernel development
Kernel developmentKernel development
Kernel development
 
Lecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports DevelopmentLecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports Development
 
Docker 활용법: dumpdocker
Docker 활용법: dumpdockerDocker 활용법: dumpdocker
Docker 활용법: dumpdocker
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
 
PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...
PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...
PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...
 
Linux Kernel Debugging
Linux Kernel DebuggingLinux Kernel Debugging
Linux Kernel Debugging
 
Anatomy of ROCgdb presentation at gcc cauldron 2022
Anatomy of ROCgdb presentation at gcc cauldron 2022Anatomy of ROCgdb presentation at gcc cauldron 2022
Anatomy of ROCgdb presentation at gcc cauldron 2022
 
XS Boston 2008 Debugging Xen
XS Boston 2008 Debugging XenXS Boston 2008 Debugging Xen
XS Boston 2008 Debugging Xen
 
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
 
lab1-ppt.pdf
lab1-ppt.pdflab1-ppt.pdf
lab1-ppt.pdf
 
Embedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debuggingEmbedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debugging
 
Roll your own toy unix clone os
Roll your own toy unix clone osRoll your own toy unix clone os
Roll your own toy unix clone os
 
CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)
CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)
CONFidence 2017: Hacking embedded with OpenWrt (Vladimir Mitiouchev)
 
Xen Debugging
Xen DebuggingXen Debugging
Xen Debugging
 
Debugging of (C)Python applications
Debugging of (C)Python applicationsDebugging of (C)Python applications
Debugging of (C)Python applications
 
Java gpu computing
Java gpu computingJava gpu computing
Java gpu computing
 
Porting Android
Porting AndroidPorting Android
Porting Android
 

More from Linaro

Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea GalloDeep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Linaro
 
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta VekariaArm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Linaro
 
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua MoraHuawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Linaro
 
Bud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qaBud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qa
Linaro
 
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
Linaro
 
HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018
Linaro
 
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
Linaro
 
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Linaro
 
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Linaro
 
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening KeynoteHKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
Linaro
 
HKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP WorkshopHKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP Workshop
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and allHKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
Linaro
 
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse HypervisorHKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
Linaro
 
HKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMUHKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMU
Linaro
 
HKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8MHKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8M
Linaro
 
HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation
Linaro
 
HKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted bootHKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted boot
Linaro
 

More from Linaro (20)

Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea GalloDeep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
 
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta VekariaArm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
 
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua MoraHuawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
 
Bud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qaBud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qa
 
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
 
HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018
 
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
 
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
 
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
 
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
 
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening KeynoteHKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
 
HKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP WorkshopHKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP Workshop
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
 
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and allHKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
 
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse HypervisorHKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
 
HKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMUHKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMU
 
HKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8MHKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8M
 
HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation
 
HKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted bootHKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted boot
 

Recently uploaded

Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
BibashShahi
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 

Recently uploaded (20)

Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
Artificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic WarfareArtificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic Warfare
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 

LAS16-403: GDB Linux Kernel Awareness

  • 1. GDB Linux Kernel Awareness Peter Griffin Linaro Member Services
  • 2. ENGINEERS AND DEVICES WORKING TOGETHER “Engineers and Devices Working Together” … with free & open source tools!
  • 3. ENGINEERS AND DEVICES WORKING TOGETHER Overview ● Rationale for GDB Linux Kernel Awareness ● Targets for debugging Linux with GDB ● Upstream GDB to debug the Linux kernel ● What is the ST landing team doing? ● What is “Linux awareness”? ● Approach Taken ● Progress so far ● Next steps ● Alternative approaches ● Demo ● Q&A
  • 4. ENGINEERS AND DEVICES WORKING TOGETHER ● Many of us need to debug the Linux kernel ● Proprietary tools like Lauterbach Trace32 and ARM DS-5 are $$$ ● Open source debuggers like GDB lack ‘kernel awareness’ features found in proprietary tools. ST LT Mission To provide a better debugging experience when using upstream GDB to debug Linux kernel via JTAG / OpenOCD, or QEMU. Rationale for GDB Linux Kernel Awareness
  • 5. ENGINEERS AND DEVICES WORKING TOGETHER Ways to Debug Linux Kernel with GDB ● GDB client using the GDB remote protocol a. Connection to a KGDB stub in a running kernel b. Connect to a QEMU stub running a virtual kernel environment c. Connect to a GDB remote compliant JTAG probe, such as OpenOCD ● GDB session on host a. Core Dump file b. UML Kernel
  • 6. ENGINEERS AND DEVICES WORKING TOGETHER ● Debug stub in the kernel compliant with GDB remote protocol ○ Enable with CONFIG_KGDB + Already supported on many platforms + All kernel threads enumerated in GDB (via GDB remote packets) - Requires cooperation between debugger and kernel stub - Less suitable for serious crashes - Isn’t enabled on production systems - Requires enough support for serial or ethernet Targets: KGDB with GDB Host GDB Target Linux KGDB GDB remote serial/ethernet
  • 7. ENGINEERS AND DEVICES WORKING TOGETHER Advantages + QEMU is open source and has GDB remote stub + No “real” hardware required + Good for testing generic kernel code, on many architectures + Good environment for developing GDB Linux awareness extensions Disadvantages - If your bug is SoC or board related it is unlikely to be useful . Targets: QEMU and GDB Dev Host GDB client Linux GDB stub Qemu
  • 8. QEMU and GDB (Example) > qemu-system-arm -M vexpress-a9 -cpu cortex-a9 -m 256M -nographic -kernel ./zImage -append 'console=ttyAMA0,115200 rw nfsroot=10.0.2.2:/opt/debian/wheezy-armel-rootfs,v3 ip=dhcp' -dtb ./vexpress-v2p-ca9.dtb -s [ 0.000000] Booting Linux on physical CPU 0x0 [ 0.000000] Linux version 4.3.0-rc3-00008-g6e751b6-dirty (griffinp@X1-Carbon) (gcc version 4.9.3 20141031 (prerelease) (Linaro GCC 2014.11) ) #317 SMP Tue Sep 29 13:20:36 BST 2015 [ 0.000000] CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] Machine model: V2P-CA9 [..] [ 98.288845] VFS: Unable to mount root fs via NFS, trying floppy. [ 98.290767] VFS: Cannot open root device "(null)" or unknown-block(2,0): error -6 [ 98.291132] Please append a correct "root=" boot option; here are the available partitions: [ 98.291681] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0) [ 98.292215] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-00008-g6e751b6-dirty #317 arm-linux-gnueabi-gdb vmlinux (gdb) target remote :1234 Remote debugging using :1234 __loop_delay () at ../arch/arm/lib/delay-loop.S:47 47 subs r0, r0, #1 (gdb) info threads Id Target Id Frame * 1 Thread 1 (CPU#0 [running]) __loop_delay () at ../arch/arm/lib/delay-loop.S:47 (gdb) bt #0 __loop_delay () at ../arch/arm/lib/delay-loop.S:47 #1 0xc02cbbe0 in panic (fmt=0xc0c340a8 "VFS: Unable to mount root fs on %s") at ../kernel/panic.c:201 #2 0xc0dad19c in mount_block_root (name=0xc0c34148 "/dev/root", flags=32768) at ../init/do_mounts.c:421 #3 0xc0dad354 in mount_root () at ../init/do_mounts.c:541 #4 0xc0dad4b4 in prepare_namespace () at ../init/do_mounts.c:600 #5 0xc0dace6c in kernel_init_freeable () at ../init/main.c:1026 #6 0xc09ca2f4 in kernel_init (unused=<optimised out>) at ../init/main.c:936
  • 9. + OpenOCD is open source + Supports GDB remote protocol + Supports many ARM/MIPS CPUs + Supports many FTDI based cheap JTAG probes Targets: JTAG - External Debug Host Target System GDB Client Telnet OpenOCD telnet gdbserver JTAG JTAG http://openocd.org http://elinux.org/JTAG
  • 10. ENGINEERS AND DEVICES WORKING TOGETHER ● Allows external access to dedicated SoC debug hardware ● Flow control via on-chip hardware breakpoints and watchpoints ○ Access to on-chip performance monitors ○ Control reset and initialization of target ● Download code directly into RAM or flash ● Manipulate MMU and caches whilst target is halted ● Allows debugging when other techniques can’t be used e.g. ○ Ability to load code on bare board & debug from reset vector ○ Debug during late and early parts of system suspend & resume ○ Debug IRQ handlers ○ Inspect very broken / corrupt systems e.g. when KGDB stub no longer functional. Targets: JTAG - Why External Debugging?
  • 11. GDB Remote Compliant JTAG - e.g. OpenOCD Compile it > git clone ssh://git@git.linaro.org:/people/peter.griffin/openocd-code > cd openocd-code > git submodule update --init --recursive > autoreconf -iv > export PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig/ > ./configure --enable-ftdi > make > make install Run it ./openocd -f ./tcl/interface/ftdi/flyswatter2.cfg -f ./tcl/target/hi6220.cfg Open On-Chip Debugger 0.9.0-dev-00241-gd356c86-dirty (2015-07-10-08:20) Licensed under GNU GPL v2 For bug reports, read http://openocd.sourceforge.net/doc/doxygen/bugs.html adapter speed: 10 kHz Info : session transport was not selected, defaulting to JTAG jtag_ntrst_delay: 100 trst_and_srst combined srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst hi6220.cpu Info : clock speed 10 kHz Info : JTAG tap: hi6220.dap tap/device found: 0x5ba00477 (mfg: 0x23b, part: 0xba00, ver: 0x5) Info : hi6220.cpu: hardware has 6 breakpoints, 4 watchpoints Example telnet interface telnet localhost 4444 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Open On-Chip Debugger > halt number of cache level 2 cache l2 present :not supported hi6220.cpu cluster 0 core 0 multi core target state: halted target halted in ARM64 state due to debug-request, current mode: EL2H cpsr: 0x800003c9 pc: 0x3ef7e908 MMU: disabled, D-Cache: disabled, I-Cache: disabled Example gdb interface trace > aarch64-linux-gnu-gdb (gdb) target remote :3333 (gdb) add-symbol-file build-hikey/u-boot 0x3ef47000 (gdb) hbreak do_version Hardware assisted breakpoint 1 at 0x3ef4cc80: file ../common/cmd_version.c, line 19. (gdb) c Continuing. Breakpoint 1, do_version (cmdtp=0x3ef8c0c0, flag=0, argc=1, argv=0x3e746850) at ../common/cmd_version.c:19 19 { (gdb) bt #0 do_version (cmdtp=0x3ef8c0c0, flag=0, argc=1, argv=0x3e746850) at ../common/cmd_version.c:19 #1 0x000000003ef604a8 in cmd_call (argv=0x3e746850, argc=1,
  • 12. ENGINEERS AND DEVICES WORKING TOGETHER KGDB JTAG QEMU Good Better Core Dumps
  • 13. ENGINEERS AND DEVICES WORKING TOGETHER ● CONFIG_PROC_KCORE ○ sudo gdb vmlinux /proc/kcore ○ Virtual ELF core file of live kernel ○ No modifications can be made ● CONFIG_PROC_VMCORE ○ /proc/vmcore ○ Used in conjunction with kexec, kdump and the crash utility from RedHat ○ py-crash, and libkdumpfile support coming to GDB from SUSE Targets: Core Dumps https://en.wikipedia.org/wiki/Kdump_(Linux) @V4711 https://www.suse.com/documentation/sles-12/book_sle_tuning/data/part_tuning_dumps.html
  • 14. ENGINEERS AND DEVICES WORKING TOGETHER So you have upstream GDB debugging the Linux Kernel via JTAG, or QEMU… but compared to $$$ commercial tools or ST GDB it can be disappointing... Upstream GDB to Debug the Linux Kernel
  • 15. ENGINEERS AND DEVICES WORKING TOGETHER ● (gdb) info threads - Shows thread running on each physical CPU - No visibility of other sleeping threads - No visibility of user processes arm-linux-gnueabi-gdb vmlinux (gdb) target remote :1234 Remote debugging using :1234 __loop_delay () at ../arch/arm/lib/delay-loop.S:47 47 subs r0, r0, #1 (gdb) info threads Id Target Id Frame * 1 Thread 1 (CPU#0 [running]) __loop_delay () at ../arch/arm/lib/delay-loop.S:47 Eeek...only 1 thread! Upstream GDB to Debug the Linux Kernel
  • 16. ENGINEERS AND DEVICES WORKING TOGETHER ● (gdb) bt - Works for CPU executing in kernel space. No unwinding for user tasks. Upstream GDB to Debug the Linux Kernel Example ‘bt’ executing in kernel space arm-linux-gnueabi-gdb vmlinux (gdb) target remote :1234 (gdb) bt #0 __loop_delay () at ../arch/arm/lib/delay-loop.S:47 #1 0xc02cbbe0 in panic (fmt=0xc0c340a8 "VFS: Unable to mount root fs on %s") at ../kernel/panic.c:201 #2 0xc0dad19c in mount_block_root (name=0xc0c34148 "/dev/root", flags=32768) at ../init/do_mounts.c:421 #3 0xc0dad354 in mount_root () at ../init/do_mounts.c:541 #4 0xc0dad4b4 in prepare_namespace () at ../init/do_mounts.c:600 Example ‘bt’ executing in userspace Program received signal SIGINT, Interrupt. 0xb6e4978c in ?? () (gdb) bt #0 0xb6e4978c in ?? () #1 0xb6e4a408 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?)
  • 17. ENGINEERS AND DEVICES WORKING TOGETHER up / down / frame <num> - same as bt, will show physical CPU, and only work if executing in kernel space. thread <num> - switch physical CPUs This could be better, more like $$$ proprietary JTAG tools. Upstream GDB to Debug the Linux Kernel
  • 18. ENGINEERS AND DEVICES WORKING TOGETHER ● Provide the debugger with additional knowledge of the underlying O/S to enable a better debugging experience. ○ Where is the Task List? ○ Where is in the Kernel Log Buffer? ○ What modules are loaded? Where? ● We split Linux Awareness into three areas 1. Task Awareness ○ Ability to report all task_structs as threads in GDB ○ Provides selectable GDB threads with GDB context commands (info threads, thread <num>, frame <num>, up, down ) 2. Loadable Module Support ○ Hooks for automatic symbol resolution when modules are inserted 3. OS Helper Commands ○ Interacting with the debugger to extract state from the O/S What is Linux Awareness?
  • 19. ENGINEERS AND DEVICES WORKING TOGETHER ● Working to extend Linux Kernel Awareness in upstream GDB. ○ With support from the Linaro Toolchain Working Group. ● This is based on the C Linux Kernel Debugger (LKD) GDB plugin by ST; the aim now is to extract and rework the features that are generally useful, and upstream them. ● The team has engaged with the GDB and Linux kernel communities to develop an approach. ● ST and the Linaro ST Landing Team are publicising this work to other Linaro members who might have an interest in it. What is the ST Landing Team Doing?
  • 20. ENGINEERS AND DEVICES WORKING TOGETHER 1129 lkd-arm.c ARM arch specific (virt to phys) 674 lkd.h 491 lkd-irqs.c 4953 lkd-main.c linux-aware target_ops, OS helpers 2456 lkd-modules.c modules target_so_ops 79 lkd-modules.h 1666 lkd-process.c kthread awareness 77 lkd-process.h 220 lkd-proxy.c 2076 lkd-sh4.c SH4 specific 13874 total Overview of ST LKD C Patchset
  • 21. ENGINEERS AND DEVICES WORKING TOGETHER ● We have 3 options 1. Scripting extension to GDB (Python / Guile) 2. Awareness in GDB stub 3. C extension to GDB ● Each of these approaches has advantages and disadvantages Where to Put the “Awareness”? GDB Client GDB stub KGDB / OpenOCD Python plugin Guile plugin Am I OS aware? ...or am I OS aware? or maybe I’m OS aware? gdbremote
  • 22. ENGINEERS AND DEVICES WORKING TOGETHER ● You can extend GDB using Python. See: https://sourceware.org/gdb/onlinedocs/gdb/Python.html#Python ● What is exposed via Python interface has been steadily improving in GDB ○ Add custom commands ○ Implement pretty printers ○ Frame filters, frame decorators and much more. ● The major advantage of implementing “Linux awareness” in Python is: the code can live in the kernel source tree. + Extensions should evolve with the kernel source + No cross-dependencies between kernel & GDB Extending GDB with Python
  • 23. ENGINEERS AND DEVICES WORKING TOGETHER Jan Kiszka from Siemens has already implemented some Python Linux awareness extensions and established this route. ● See Docs @ Documentation/gdb-kernel-debugging.txt ● See Python @ scripts/gdb/* ● Enable CONFIG_GDB_SCRIPTS ● Provides a series of custom Linux commands and functions ● Can be easily tested using QEMU or OpenOCD Python GDB Linux Awareness “Extensions”
  • 24. ENGINEERS AND DEVICES WORKING TOGETHER Linux GDB Python Extensions in v4.6 (gdb) apropos lx function lx_current -- Return current task function lx_module -- Find module by name and return the module variable function lx_per_cpu -- Return per-cpu variable function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable function lx_thread_info -- Calculate Linux thread_info from task variable lx-cmdline -- Report the Linux Commandline used in the current kernel lx-dmesg -- Print Linux kernel log buffer lx-list-check -- Verify a list consistency lx-lsmod -- List currently loaded modules lx-ps -- Dump Linux tasks lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules lx-version -- Report the Linux Version of the current kernel
  • 25. ENGINEERS AND DEVICES WORKING TOGETHER ● Migrating LKD C-based OS helper commands into Python. ○ C code then retired from LKD, reducing amount of ‘out of tree’ LKD code. ○ Python code when merged will evolve with the kernel source tree. ● Some commands such as dmesg already existed upstream ● First ST LT Python OS helper commands merged into Linux kernel. ○ lx-cmdline and lx-version in v4.6. ○ lx-mount, lx-iomem, lx-ioports in v4.7. lx-cmdline -- Report the Linux Commandline used in the current kernel lx-version -- Report the Linux Version of the current kernel lx-iomem -- Identify the IO memory resource locations defined by the kernel lx-ioports -- Identify the IO port resource locations defined by the kernel ST LT Involvement: OS Helpers
  • 26. ENGINEERS AND DEVICES WORKING TOGETHER (gdb) lx-cmdline root=/dev/nfs nfsroot=10.0.2.2:/opt/debian/wheezy-armel-rootfs,tcp,v3 rw ip=dhcp mem=1024M raid=noautodetect rootwait console=ttyAMA0,38400n8 devtmpfs.mount=0 (gdb) lx-version Linux version 4.7.0-rc3-00003-g5d518fc-dirty (griffinp@griffinp-ThinkPad-X1-Carbon-2nd) (gcc version 4.9.3 20141031 (prerelease) (Linaro GCC 2014.11) ) #4 SMP Thu Jul 14 12:54:03 BST 2016 Example Command Output
  • 27. ENGINEERS AND DEVICES WORKING TOGETHER Example Output (gdb) lx-iomem 10000008-1000000b : dat 10000048-1000004b : dat 1000004c-1000004f : dat 100000a0-100000ab : vexpress-syscfg 10004000-10004fff : /smb@04000000/motherboard/iofpga@7,00000000/aaci@04000 10005000-10005fff : /smb@04000000/motherboard/iofpga@7,00000000/mmci@05000 10005000-10005fff : /smb@04000000/motherboard/iofpga@7,00000000/mmci@05000 10006000-10006fff : /smb@04000000/motherboard/iofpga@7,00000000/kmi@06000 10006000-10006fff : kmi-pl050 10007000-10007fff : /smb@04000000/motherboard/iofpga@7,00000000/kmi@07000 10007000-10007fff : kmi-pl050 10009000-10009fff : /smb@04000000/motherboard/iofpga@7,00000000/uart@09000 10009000-10009fff : /smb@04000000/motherboard/iofpga@7,00000000/uart@09000 1000a000-1000afff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0a000 1000a000-1000afff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0a000 1000b000-1000bfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0b000 1000b000-1000bfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0b000 1000c000-1000cfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0c000 1000c000-1000cfff : /smb@04000000/motherboard/iofpga@7,00000000/uart@0c000 10011000-10011fff : /smb@04000000/motherboard/iofpga@7,00000000/timer@11000 10012000-10012fff : /smb@04000000/motherboard/iofpga@7,00000000/timer@12000 10017000-10017fff : /smb@04000000/motherboard/iofpga@7,00000000/rtc@17000 10017000-10017fff : rtc-pl031 1001f000-1001ffff : /smb@04000000/motherboard/iofpga@7,00000000/clcd@1f000 1001f000-1001ffff : clcd-pl11x [..]
  • 28. ENGINEERS AND DEVICES WORKING TOGETHER lx-fdtdump -- Output Flattened Device Tree header and dump FDT blob to a file (gdb) lx-fdtdump fdt_magic: 0xD00DFEED fdt_totalsize: 0xC108 off_dt_struct: 0x38 off_dt_strings: 0x3804 off_mem_rsvmap: 0x28 version: 17 last_comp_version: 16 Dumped fdt to fdtdump.dtb Allows further post-processing on development machine with standard DT tools. e.g. fdtdump fdtdump.dtb | less Allows diagnoses of what bootloader may have changed in DT to cause board not to boot. lx-fdtdump (Not Upstreamed Yet)
  • 29. ENGINEERS AND DEVICES WORKING TOGETHER ● Working to extract a minimal C-based kthread patchset for GDB. ● Minimal Linux kthread RFC posted https://sourceware.org/ml/gdb-patches/2016-02/msg00826.html ● Not much feedback from GDB community. Now preparing a refactored C implementation of linux-kthread for upstreaming. Based on feedback from Yao Qi at Connect BKK16 (GDB maintainer & ARM TCWG assignee to Linaro). Fixing style issues, re-factoring architecture specifics etc. Patches work well with nommu Linux, OpenOCD and GDB. Aim to submit refactored patches to GDB community by end of Q4 2016. Kernel Thread Awareness: ST LT Approach
  • 30. ENGINEERS AND DEVICES WORKING TOGETHER ~/binutils-gdb/gdb$ git diff 4f80d1d --stat gdb/Makefile.in | 4 +- gdb/arm-linux-tdep.c | 98 ++++++ gdb/configure.tgt | 6 +- gdb/doc/observer.texi | 5 + gdb/linux-kthread.c | 1548 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++ gdb/linux-kthread.h | 215 ++++++++++++++ gdb/remote.c | 5 + gdb/thread.c | 2 +- 8 files changed, 1878 insertions(+), 5 deletions(-) Overview linux-kthread Refactored Patchset
  • 31. ENGINEERS AND DEVICES WORKING TOGETHER JTAGUSB GDB Linux Kernel Awareness: Components GDB Python plugins Kernel crash dump file Python plugin source in Linux kernel source tree (e.g. lx-dmesg) OpenOCD GDB Stub QEMU GDB Stub On-chip Debug USB- JTAG Dongle gdb remote serial protocol Target Board Linux Kernel Binary Linux Kernel Binary linux-kthread
  • 32. ENGINEERS AND DEVICES WORKING TOGETHER ● Refactored minimal kthread patchset ● OpenOCD controlling stm32 target ● STM32F429 Discovery Board running v4.7-rc3 nommu Linux Demonstrating: ● Correct enumeration of kernel threads ○ New threads enumerated by GDB ○ Old threads pruned from GDB thread list ● Python extensions merged for v4.7 ○ lx-cmdline, lx-mounts, lx-iomem Demo!
  • 33. ENGINEERS AND DEVICES WORKING TOGETHER Virtual Memory and MMU control ● User can halt CPU at anytime and MMU could be configured for userspace mapping. ○ Must be able to detect this, before attempting to read any kernel memory. ○ Updating kernel thread list in GDB requires us to to switch MMU to kernel mapping to read kernel memory and parse task_struct list. ● Prototyping with QEMU to return ARM system coprocessor registers over GDB remote using target XML. ○ Currently returns TTBR0/1, MIDR amd IDPFR0/1 registers. ○ Using target XML allows GDB stub to return appropriate registers needed for full system debug. ○ Using registers, allows access via register cache, and also from python scripts (unlike for example custom GDB remote packet). ○ Support for target XML currently not implemented in OpenOCD. GDB knowledge of CPU core ● To determine whether virtual memory manipulation is required. Current Kthread Challenges and Considerations
  • 34. ENGINEERS AND DEVICES WORKING TOGETHER (gdb) info registers r0 0x1 1 [..] sp 0xc1201fa00xc1201fa0 <init_thread_union+8096> lr 0xc03087e8 -1070561304 pc 0xc031b0e8 0xc031b0e8 <cpu_v7_do_idle+8> cpsr 0x60000093 1610612883 ttbr0_el1 0x0 0 ttbr1_el1 0x6020406a 1612726378 midr 0x410fc090 1091551376 id_pfr0 0x1031 4145 id_pfr1 0x11 17 CP Registers Example
  • 35. ENGINEERS AND DEVICES WORKING TOGETHER ● Complete and upstream C implementation of linux-kthread. ● Automated regression testing (LAVA, CI). ● Implement and upstream remaining LKD OS helper commands ○ lx-pgtable, pgmap, process_info (all remaining commands rely on virtual memory support). ● Review and test upstream Python implementation of loadable module support versus LKD C implementation. Known Bugs ● GDB APIs can behave differently between GDB remote stubs (e.g. QEMU versus OpenOCD stub) ● Bug in GDB Python shim layer https://bugs.linaro.org/show_bug.cgi?id=2489 Next ST LT Steps
  • 36. ENGINEERS AND DEVICES WORKING TOGETHER Other Potential Extensions ● Alternatively could implement Linux kthreads in Python ○ Extend GDB by means of Python scripts. Python again lives in kernel source. ○ Need to extend GDB’s Python API. ■ GDB thread object, target object, register object, symbol object need to be exposed via Python bindings to get working. ■ Infrastructure could be re-used for other OS’s or RTOSs ○ Some prototyping and patches by ST LT to do this. ● Implement kthread for ARMv8 and X86 architectures. ○ Can be tested using QEMU, but needs also relies on aarch64 support in OpenOCD. ● Extend to support examination of kernel crash dump files. ○ Some GDB developers (in SUSE) working on this. ○ Could investigate how to combine this work with ours.
  • 37. ENGINEERS AND DEVICES WORKING TOGETHER ● ST’s LKD GDB plugin implements this feature. ○ Automatically pull and resolve userspace symbols for userspace processes from target root filesystem. gdb) bt #0 context_switch (cookie=..., next=<optimised out>, prev=<optimised out>, rq=<optimised out>) at ../kernel/sched/core.c:2867 [..] #6 0xc042bd30 in do_select (n=<optimised out>, fds=<optimised out>, end_time=<optimised out>) at ../fs/select.c:533 #7 0xc042bee8 in core_sys_select (n=11, inp=<unavailable>, outp=<unavailable>, exp=0x0, end_time=0xee075f70) at ../fs/select.c:603 #8 0xc042c280 in SYSC_select (tvp=<optimised out>, exp=<optimised out>, outp=<optimised out>, inp=<optimised out>, n=<optimised out>) at ../fs/select.c:644 #9 SyS_select (n=11, inp=-1098552816, outp=0, exp=<unavailable>, tvp=-1098553000) at ../fs/select.c:626 #10 ret_fast_syscall () at ../arch/arm/kernel/entry-common.S:37 Kernel / User boundary #11 0xb6f19a80 in elf_machine_rel (reloc=0x86, version=<optimised out>, reloc_addr_arg=0x3091969, sym=0x9db0, map=0xbe856960) at ../ports/sysdeps/arm/dl-machine.h:475 #12 elf_dynamic_do_rel (lazy=0, relsize=<optimised out>, reladdr=<optimised out>, map=0xbe856960) at do-rel.h:120 #13 _dl_start (arg=0xbe8ab000) at rtld.c:547 #14 0xbe8ab000 in ?? () #15 0xbe8ab000 in ?? () Future Extensions: Backtrace to Userspace!
  • 40. ENGINEERS AND DEVICES WORKING TOGETHER ● An example of putting “thread awareness” in the GDB stub ● OpenOCD already supports various rtos task awareness (eCos, ThreadX, FreeRTOS, ChibiOS, embKernel, mqx and Linux)! ○ See openocd-code/src/rtos/linux.c ○ Disabled by default ● Enabled by adding “-rtos linux” to the target command in OpenOCD config file ○ target create $_TARGETNAME_2 cortex_a -chain-position $_CHIPNAME.dap -dbgbase $_DAP_DBG2 -coreid 1 -rtos linux ● This has been tested with Tincantools’ Flyswatter JTAG and a U8500 (armv7) snowball board. ● Required some hacking to get it working, but in the end I could successfully enumerate a full thread list in GDB! OpenOCD -rtos Linux Task Awareness
  • 41. ENGINEERS AND DEVICES WORKING TOGETHER Advantages + No GDB changes required (threads returned via gdbremote packets). + All kernel threads can be enumerated in GDB (once working) + Can be used in conjunction with python extensions (e.g. lx-dmesg etc) + OpenOCD already has mmu virt to phys translation code for many CPUs Disadvantages - Awareness in GDB means thread awareness re-implementing for each stub: Qemu, OpenOCD... - No good for debugging kernel dumps - Tight coupling of code parsing kernel data structures in OpenOCD - No debug information available in OpenOCD. - No way (I know of) to find field offsets via gdbremote protocol. - Currently this means task_struct, thread_info and mm_struct field offsets require OpenOCD to be recompiled for the kernel you wish to debug :( Improvements ○ gdbremote protocol could be extended ○ At a minimum task structure field offsets configurable at runtime, instead of requiring recompile! OpenOCD -rtos Task Awareness
  • 42. ENGINEERS AND DEVICES WORKING TOGETHER /* gdb script to update the header file according to kernel version and build option before executing function awareness kernel symbol must be loaded : symbol vmlinux define awareness set logging off set logging file linux_header.h set logging on printf "#define QAT %pn",&((struct task_struct *)(0))->stack set $a=&((struct list_head *)(0))->next set $a=(int)$a+(int)&((struct task_struct *)(0))->tasks printf "#define NEXT %pn",$a printf "#define COMM %pn",&((struct task_struct *)(0))->comm printf "#define MEM %pn",&((struct task_struct *)(0))->mm printf "#define ONCPU %pn",&((struct task_struct *)(0))->on_cpu printf "#define PID %pn",&((struct task_struct *)(0))->pid */ OpenOCD linux-header.h #define QAT 0x4 #define NEXT 0x1b0 #define COMM 0x2d4 #define MEM 0x1cc #define ONCPU 0x18 #define PID 0x1f4 #define CPU_CONT 0x1c #define PREEMPT 0x4 #define MM_CTX 0x160
  • 43. Thank You #LAS16 For further information: www.linaro.org LAS16 keynotes and videos on: connect.linaro.org