SlideShare a Scribd company logo
Let's trace Linux Kernel
with KGDB
潘建宏 Jian-Hong Pan (StarNight)
@ COSCUP 2021
Who am I
潘建宏 / Jian-Hong Pan (StarNight)
Endless OS Foundation
You can find me at
● http://www.slideshare.net/chienhungpan/
● GitHub: starnight
● Email:
jhp [AT] endlessos.org
chienhung.pan [AT] gmail.com
Outline
● How to debug Linux kernel
● KGDB
○ The Linux kernel build config
● KGDB with VM (QEMU)
● KGDB with real system (Raspberry Pi 4B)
○ How does Raspberry Pi 4B boot kernel
○ Raspberry Pi 4B’s JTAG
○ OpenOCD & the JTAG adapter
○ Add U-Boot to catch up kernel boot
○ Real debug
● Reference
How to Debug Linux kernel???
● Read the kernel message log, then grep the key words in kernel
source codes.
● Add some more debug messages with printk like functions in
interesting parts.
● Kprobes, ftrace, eBPF ....
● Export the signals to GPIO pins!!!???
Basic idea is “trace the code path to understand why it goes here, there.”
Can kernel be debugged with GDB like the user space programs?
KGDB
The kernel debugger kgdb, hypervisors like QEMU or JTAG-based
hardware interfaces allow to debug the Linux kernel and its modules
during runtime using gdb.
- from Debugging kernel and modules via gdb
Host Target Machine
Debug Line
The debug line could be professional like
JTAG, or simple one like serial.
Build Linux Kernel on Host
Build the kernel with following configuration (at least)
● # CONFIG_RANDOMIZE_BASE is not set
● CONFIG_DEBUG_INFO=y
● # CONFIG_DEBUG_INFO_REDUCED is not set
● CONFIG_GDB_SCRIPTS=y
● CONFIG_FRAME_POINTER=y
Start with Virtual Environment - QEMU
Host Target Machine
(QEMU VM)
Debug Line
built kernel
Install
Install the Built Kernel into the Target Machine (VM)
Have a QEMU guest VM with an OS (for example, Debian) as the target
machine
1. Start the VM with 9p shared folder (on Host)
$ qemu-system-x86_64 -m 2048 -smp 2 qemu-images/debian.qcow2 -enable-kvm -virtfs
local,path=linux-stable/,mount_tag=host0,security_model=mapped-xattr,id=host_share
2. Create a folder for mounting the shared folder from host (on guest
VM)
$ mkdir linux-stable && sudo mount -t 9p -o trans=virtio host0 linux-stable/
3. Installed the built kernel into the guest VM (on guest VM)
$ cd linux-stable && sudo make install modules_install
Debug the Kernel on Target Machine (VM)
Host Target Machine
(QEMU VM)
Debug Line
built kernel gdb connection
Enable the gdb stub of QEMU/KVM
● Start the guest VM with enabled gdb stub (on Host)
$ qemu-system-x86_64 -m 2048 -smp 2 qemu-images/debian.qcow2 -s -S
● Must disable kvm
● Manual of qemu-system-x86_64:
-s
...
freeze CPU at startup (use 'c' to start execution)
-gdb dev accept gdb connection on 'dev'. (QEMU defaults to
starting the guest without waiting for gdb to connect;
use -S too if you want it to not start execution.)
-S shorthand for -gdb tcp::1234
Use GDB on Host
1. Start gdb in the linux project folder
$ cd /path/to/linux-stable && gdb vmlinux
2. Attach to the booted geust VM
(gdb) target remote localhost:1234
3. Use gdb: set break points, continue, debug …
# Add a break point for test
(gdb) break drm_core_init
Breakpoint 1 at 0xffffffff82a7ebf0: file drivers/gpu/drm/drm_drv.c, line 1116.
# Continue the kernel
(gdb) continue
Continuing.
Thread 1 hit Breakpoint 1, drm_core_init () at drivers/gpu/drm/drm_drv.c:1116
1116 drm_connector_ida_init();
(gdb) backtrace
#0 drm_core_init () at drivers/gpu/drm/drm_drv.c:1116
#1 0xffffffff81000d61 in do_one_initcall (fn=0xffffffff82a7ebf0 <drm_core_init>) at init/main.c:1205
Let’s debug in the Real World!
Raspberry Pi as the target machine
Debug the Kernel on Target Machine (Raspberry Pi)
Target Machine
(Raspberry Pi)
Debug Line
JTAG
Host
built kernel
OpenOCD
How Debian Boots on Raspberry Pi 4B
● Download Debain images for Raspberry Pi 4B and install it by
following RaspberryPiImages
● The image has 2 partitions. First one is the boot partition.
● Here are the files in the boot partition
$ ls
bcm2711-rpi-4-b.dtb cmdline.txt fixup_cd.dat start4db.elf start_x.elf
bcm2837-rpi-3-a-plus.dtb config.txt fixup.dat start4.elf sysconf.txt
bcm2837-rpi-3-b.dtb fixup4cd.dat fixup_db.dat start4x.elf vmlinuz-5.10.0-7-arm64
bcm2837-rpi-3-b-plus.dtb fixup4.dat fixup_x.dat start_cd.elf
bcm2837-rpi-cm3-io3.dtb fixup4db.dat initrd.img-5.10.0-7-arm64 start_db.elf
bootcode.bin fixup4x.dat start4cd.elf start.elf
Device Boot Start End Sectors Size Id Type
/dev/sda1 8192 614399 606208 296M c W95 FAT32 (LBA)
/dev/sda2 614400 124735487 124121088 59.2G 83 Linux
Debian’s config.txt & cmdline.txt
$ cat config.txt
# Switch the CPU from ARMv7 into ARMv8 (aarch64) mode
arm_64bit=1
enable_uart=1
upstream_kernel=1
kernel=vmlinuz-5.10.0-7-arm64
# For details on the initramfs directive, see
# https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=10532
initramfs initrd.img-5.10.0-7-arm64
$ cat cmdline.txt
console=tty0 console=ttyS1,115200 root=/dev/mmcblk1p2 rw fsck.repair=yes net.ifnames=0 rootwait
Enable Raspberry Pi 4B’s JTAG
● The chip is BCM2711
● Raspberry Pi 4B maps JTAG pins to GPIO
○ Setting enable_jtag_gpio=1 selects Alt4 mode for GPIO pins 22-27, and
sets up some internal SoC connections, thus enabling the JTAG interface
for the ARM CPU.
Pin # Function
GPIO22 ARM_TRST
GPIO23 ARM_RTCK
GPIO24 ARM_TDO
GPIO25 ARM_TCK
GPIO26 ARM_TDI
GPIO27 ARM_TMS
Reference: Raspberry Pi Documentation's GPIO control in config.txt
Table 94. GPIO Pins Alternative
Function Assignment of
BCM2711 ARM Peripherals
JTAG interface
Open On-Chip Debugger (OpenOCD)
● It does so with the assistance of a debug adapter, which is a small
hardware module which helps provide the right kind of electrical
signaling to the target being debugged. These are required since
the debug host (on which OpenOCD runs) won’t usually have native
support for such signaling, or the connector needed to hook up to
the target.
● A JTAG Adapter supports JTAG signaling, and is used to
communicate with JTAG (IEEE 1149.1) compliant TAPs on your target
board. A TAP is a “Test Access Port”, a module which processes
special instructions and data. TAPs are daisy-chained within and
between chips and boards. JTAG supports debugging and boundary
scan operations.
● List of Debug Adapter Hardware, or search the list of tcl/interface in
OpenOCD repository directly
Reference: OpenOCD User's Guid - About
Try to Buy a JTAG Adatper
● I do not have much money :( $$$$$
● It is really hard to buy and get the electronic parts, due to the
pandemic.
● I have no idea why all of the JTAG adapters come from overseas.
● The shipment becomes extremely expensive and much longer
waiting time.
So, I can only suspend this study. Sad … T^T
Until ...
Thanks to Mat lend me a
FT2232H-56Q Mini Module
as a JTAG interface!!!
Hacking Thursday! Hurray!!!
JTAG interface
Table 3.13 FT2232H Pin
Configurations for 56-pin
VQFN package of FT2232H
Dual High Speed USB to
Multipurpose UART/FIFO IC
Datasheet
JTAG interface
Table 3.1 FT2232H-56Q Mini
Module Connection – CN2 of
FT2232H-56Q Mini Module
Datasheet
Prepare JTAG Interface cfg for OpenOCD
# Refer to tcl/interface/ftdi/minimodule.cfg
#
# FTDI MiniModule
#
# http://www.ftdichip.com/Support/Documents/DataSheets/Modules/DS_FT2232H_Mini_Module.pdf
#
adapter driver ftdi
#ftdi_device_desc "FT2232H-56Q MiniModule"
ftdi_vid_pid 0x0403 0x6010
# Every pin set as high impedance except TCK, TDI, TDO and TMS
ftdi_layout_init 0x0008 0x000b
transport select jtag
# nTRST defined on pin CN2-13 of the MiniModule (pin ADBUS5 [AD5] on the FT2232H chip)
# This choice is arbitrary. Use other GPIO pin if desired.
ftdi_layout_signal nTRST -data 0x0020 -oe 0x0020
Execute OpenOCD with the cfgs
$ openocd -f minimodule.cfg -c "set USE_SMP 1" -f bcm2711.cfg -c "reset_config trst_only"
Open On-Chip Debugger 0.11.0
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
1
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 1000 kHz
Info : JTAG tap: bcm2711.cpu tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver: 0x4)
Info : bcm2711.cpu0: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu1: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu2: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu3: hardware has 6 breakpoints, 4 watchpoints
Info : starting gdb server for bcm2711.cpu0 on 3333
Info : Listening on port 3333 for gdb connections
Note: the patch tcl/board: Add Raspberry Pi 4 model B board
But the debug connection,
including gdb and OpenOCD is too
late to catch up the kernel boot.
And, Raspberry Pi does not expose
JTAG System Reset (SRST) pin.
The RUN/GLOBAL_EN pin on
Raspberry Pi boards might be
implemented as the
System Reset (SRST) pin.
Re: openocd, jtag, fyi
But, OpenOCD still cannot catch up
and halt target system on time.
How about intercept boot with
U-Boot ...
3th stage
Kernel boots
Boot Linux kernel via U-Boot on Raspberry Pi 4B
Raspberry Pi OS’ firmwares
3th stage
U-Boot loads kernel, initramfs and
device tree, then boots kernel with
boot script
1st stage
Boot from GPU with firmware in
ROM loads start4x.elf ... in FAT
2rd stage
Start4.elf loads DT,
kernel …
and starts CPU
U-Boot
Build U-Boot
1. Build U-Boot & install
$ make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- rpi_4_defconfig
$ make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-
$ cp u-boot.bin <RPi image’s root partition path>/boot/kernel8.img
2. Prepare the boot script for U-Boot
3. Prepare the boot script and following configuration file for U-Boot
4. Get RPi 4B’s device tree blob from Raspberry Pi OS for U-Boot
5. Modify config.txt to make boot firmware load and execute the kernel8.img
Prepare U-Boot script
$ cat boot.src
echo "Load u-Boot environment from uEnv.txt"
load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} uEnv.txt
env import -t ${scriptaddr} ${filesize}
echo "Load device tree from "${fdtdir}/${fdtfile}
load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${fdtdir}/${fdtfile}
echo "Load kernel from "${kernel_image}
load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${kernel_image}
echo "Load RAM disk from "${ramdisk_image}
load ${devtype} ${devnum}:${distro_bootpart} ${ramdisk_addr_r} ${ramdisk_image}
echo "Boot ..."
setenv bootargs ${bootargs}
booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}
$ mkimage -T script -C none -d boot.src <Path to image’s boot partition>/boot.scr
And, the uEnv.txt for U-Boot script
$ cat uEnv.txt
kernel_image=vmlinux-5.12.10
ramdisk_image=initrd.img-5.12.10
fdtdir=dtbs/5.12.10
bootargs=console=tty0 console=ttyS1,115200 root=/dev/mmcblk1p2 rw
fsck.repair=yes net.ifnames=0 rootwait
Modify RPi’s config.txt for Executing kernel8.img
$ cat config.txt
arm_64bit=1
enable_uart=1
enable_jtag_gpio=1
The New Status of the Boot Partition
$ ls
bcm2711-rpi-4-b.dtb fixup4db.dat start4x.elf
bcm2837-rpi-3-a-plus.dtb fixup4x.dat start_cd.elf
bcm2837-rpi-3-b.dtb fixup_cd.dat start_db.elf
bcm2837-rpi-3-b-plus.dtb fixup.dat start.elf
bcm2837-rpi-cm3-io3.dtb fixup_db.dat start_x.elf
bootcode.bin fixup_x.dat sysconf.txt
boot.scr initrd.img-5.10.0-7-arm64 System.map-5.12.10
cmdline.txt initrd.img-5.12.10 uEnv.txt
config.txt kernel8.img vmlinux-5.12.10
dtbs start4cd.elf vmlinuz-5.10.0-7-arm64
fixup4cd.dat start4db.elf
fixup4.dat start4.elf
Press any key to stop autoboot in U-Boot
U-Boot 2021.04 (Jun 20 2021 - 17:36:34 +0800)
...
scanning bus xhci_pci for devices... 3 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
U-Boot> boot
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
724 bytes read in 13 ms (53.7 KiB/s)
…
Starting kernel ...
Press a key to make U-boot paused,
then connect the GDB via OpenOCD + JTAG.
Finally, continue boot
Launch OpenOCD to Start GDB Server with JTAG
$ openocd -f minimodule.cfg -c "set USE_SMP 1" -f bcm2711.cfg -c "reset_config trst_only"
Open On-Chip Debugger 0.11.0
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
1
Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 1000 kHz
Info : JTAG tap: bcm2711.cpu tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver:
0x4)
Info : bcm2711.cpu0: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu1: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu2: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2711.cpu3: hardware has 6 breakpoints, 4 watchpoints
Info : starting gdb server for bcm2711.cpu0 on 3333
Info : Listening on port 3333 for gdb connections
Debug with GDB via the OpenOCD + JTAG
$ aarch64-linux-gnu-gdb vmlinux
...
Reading symbols from vmlinux...
(gdb) target extended-remote localhost:3333
Remote debugging using localhost:3333
0x000000003b39133c in ?? ()
(gdb) continue
Continuing.
^C
...
Program received signal SIGINT, Interrupt.
mem_serial_in (p=<optimized out>, offset=<optimized out>)
at drivers/tty/serial/8250/8250_port.c:399
399 return readb(p->membase + offset);
(gdb) hbreak drm_core_init
Hardware assisted breakpoint 1 at 0xffff80001115775c: file drivers/gpu/drm/drm_drv.c, line 1044.
(gdb) continue
Continuing.
...
Continue U-Boot to boot the kernal image
Intercept/Interrupt the kernal boot at a proper point
hbreak in Setting Breakpoints
Debug with GDB via the OpenOCD + JTAG (cont.)
Breakpoint 1, drm_core_init () at drivers/gpu/drm/drm_drv.c:1044
1044 drm_connector_ida_init();
(gdb) backtrace
#0 drm_core_init () at drivers/gpu/drm/drm_drv.c:1044
#1 0xffff800010012e80 in do_one_initcall (
fn=0xffff80001115775c <drm_core_init>) at init/main.c:1226
#2 0xffff8000111212b4 in do_initcall_level (
command_line=0xffff000040264500 "console", level=6) at init/main.c:1299
#3 do_initcalls () at init/main.c:1315
#4 do_basic_setup () at init/main.c:1335
#5 kernel_init_freeable () at init/main.c:1537
#6 0xffff800010c5419c in kernel_init (unused=<optimized out>)
at init/main.c:1424
#7 0xffff80001001448c in ret_from_fork () at arch/arm64/kernel/entry.S:955
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
Summary
● Some key items for KGDB in Linux kernel building config
● Tried KGDB with QEMU easily
○ Learned steps to debug the Lnux kernel on QEMU guest OS
● Tried KGDB with real physical system Raspberry Pi 4B
○ The JTAG interface, the adapter and corresponding hardware feature
○ BCM2711 on Raspberry Pi 4B’s JTAG feature and configuration
○ Debugged the Linux kernel on Raspberry Pi 4B with KGDB via OpenOCD + JTAG
adapter
● If you have some money, please buy a professional JTAG adapter. It
has less barriers and saves your time.
● KGDB is not the only way to debug Linux kernel. The printk series
might be good enough for most of cases. Especially, timing related
issues.
Reference
● Debugging kernel and modules via gdb
● JTAG
● OpenOCD
● Bare Metal Raspberry Pi 3B+: JTAG
● Baremetal Raspberry Pi 4 with FT2232H
● FT2232H Dual High Speed USB to Multipurpose UART/FIFO IC datasheet
● FT2232H-56Q Mini Module Datasheet
● Raspberry Pi → Documentation → The boot folder
● Device Tree
● RPi U-Boot
● GDB → Setting Breakpoints
● JTAG Reset on Raspberry Pi forum
● A Journey to Boot Linux on Raspberry Pi
Let's trace Linux Lernel with KGDB @ COSCUP 2021

More Related Content

What's hot

Scheduling in Android
Scheduling in AndroidScheduling in Android
Scheduling in Android
Opersys inc.
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking Explained
Thomas Graf
 
Bootloaders (U-Boot)
Bootloaders (U-Boot) Bootloaders (U-Boot)
Bootloaders (U-Boot)
Omkar Rane
 
The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecturehugo lu
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
Adrien Mahieux
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
Brendan Gregg
 
The TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux KernelThe TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux Kernel
Divye Kapoor
 
U-Boot - An universal bootloader
U-Boot - An universal bootloader U-Boot - An universal bootloader
U-Boot - An universal bootloader
Emertxe Information Technologies Pvt Ltd
 
The Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast StorageThe Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast Storage
Kernel TLV
 
Linux Kernel I/O Schedulers
Linux Kernel I/O SchedulersLinux Kernel I/O Schedulers
Linux Kernel I/O Schedulers
RajKumar Rampelli
 
Interrupt Affinityについて
Interrupt AffinityについてInterrupt Affinityについて
Interrupt AffinityについてTakuya ASADA
 
Embedded Linux BSP Training (Intro)
Embedded Linux BSP Training (Intro)Embedded Linux BSP Training (Intro)
Embedded Linux BSP Training (Intro)
RuggedBoardGroup
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
Netronome
 
Embedded_Linux_Booting
Embedded_Linux_BootingEmbedded_Linux_Booting
Embedded_Linux_BootingRashila Rr
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
Hao-Ran Liu
 
Linux Kernel Crashdump
Linux Kernel CrashdumpLinux Kernel Crashdump
Linux Kernel Crashdump
Marian Marinov
 
Slab Allocator in Linux Kernel
Slab Allocator in Linux KernelSlab Allocator in Linux Kernel
Slab Allocator in Linux Kernel
Adrian Huang
 
U-Boot Porting on New Hardware
U-Boot Porting on New HardwareU-Boot Porting on New Hardware
U-Boot Porting on New Hardware
RuggedBoardGroup
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
Brendan Gregg
 
Security Monitoring with eBPF
Security Monitoring with eBPFSecurity Monitoring with eBPF
Security Monitoring with eBPF
Alex Maestretti
 

What's hot (20)

Scheduling in Android
Scheduling in AndroidScheduling in Android
Scheduling in Android
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking Explained
 
Bootloaders (U-Boot)
Bootloaders (U-Boot) Bootloaders (U-Boot)
Bootloaders (U-Boot)
 
The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecture
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
The TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux KernelThe TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux Kernel
 
U-Boot - An universal bootloader
U-Boot - An universal bootloader U-Boot - An universal bootloader
U-Boot - An universal bootloader
 
The Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast StorageThe Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast Storage
 
Linux Kernel I/O Schedulers
Linux Kernel I/O SchedulersLinux Kernel I/O Schedulers
Linux Kernel I/O Schedulers
 
Interrupt Affinityについて
Interrupt AffinityについてInterrupt Affinityについて
Interrupt Affinityについて
 
Embedded Linux BSP Training (Intro)
Embedded Linux BSP Training (Intro)Embedded Linux BSP Training (Intro)
Embedded Linux BSP Training (Intro)
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
 
Embedded_Linux_Booting
Embedded_Linux_BootingEmbedded_Linux_Booting
Embedded_Linux_Booting
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
 
Linux Kernel Crashdump
Linux Kernel CrashdumpLinux Kernel Crashdump
Linux Kernel Crashdump
 
Slab Allocator in Linux Kernel
Slab Allocator in Linux KernelSlab Allocator in Linux Kernel
Slab Allocator in Linux Kernel
 
U-Boot Porting on New Hardware
U-Boot Porting on New HardwareU-Boot Porting on New Hardware
U-Boot Porting on New Hardware
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
 
Security Monitoring with eBPF
Security Monitoring with eBPFSecurity Monitoring with eBPF
Security Monitoring with eBPF
 

Similar to Let's trace Linux Lernel with KGDB @ COSCUP 2021

ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux AwarenessPeter Griffin
 
LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness Peter Griffin
 
LAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel AwarenessLAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel Awareness
Linaro
 
Share the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development BoardShare the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development Board
Jian-Hong Pan
 
Linux Kernel Platform Development: Challenges and Insights
 Linux Kernel Platform Development: Challenges and Insights Linux Kernel Platform Development: Challenges and Insights
Linux Kernel Platform Development: Challenges and Insights
GlobalLogic Ukraine
 
Embedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debuggingEmbedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debugging
Anne Nicolas
 
Roll your own toy unix clone os
Roll your own toy unix clone osRoll your own toy unix clone os
Roll your own toy unix clone os
eramax
 
Basic Linux kernel
Basic Linux kernelBasic Linux kernel
Basic Linux kernel
Morteza Nourelahi Alamdari
 
Linux: the first second
Linux: the first secondLinux: the first second
Linux: the first second
Alison Chaiken
 
Advanced Diagnostics 2
Advanced Diagnostics 2Advanced Diagnostics 2
Advanced Diagnostics 2Aero Plane
 
CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...
CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...
CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...
PROIDEA
 
Mainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernels
Mainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernelsMainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernels
Mainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernels
Dobrica Pavlinušić
 
Linux+sensor+device-tree+shell=IoT !
Linux+sensor+device-tree+shell=IoT !Linux+sensor+device-tree+shell=IoT !
Linux+sensor+device-tree+shell=IoT !
Dobrica Pavlinušić
 
HKG15-409: ARM Hibernation enablement on SoCs - a case study
HKG15-409: ARM Hibernation enablement on SoCs - a case studyHKG15-409: ARM Hibernation enablement on SoCs - a case study
HKG15-409: ARM Hibernation enablement on SoCs - a case study
Linaro
 
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysisKernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
Anne Nicolas
 
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheapUWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
edlangley
 
Linux Kernel Debugging
Linux Kernel DebuggingLinux Kernel Debugging
Linux Kernel Debugging
GlobalLogic Ukraine
 
Ganeti - build your own cloud
Ganeti - build your own cloudGaneti - build your own cloud
Ganeti - build your own cloud
Dobrica Pavlinušić
 
Introduction to Modern U-Boot
Introduction to Modern U-BootIntroduction to Modern U-Boot
Introduction to Modern U-Boot
GlobalLogic Ukraine
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting Started
Brendan Gregg
 

Similar to Let's trace Linux Lernel with KGDB @ COSCUP 2021 (20)

ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux Awareness
 
LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness
 
LAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel AwarenessLAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel Awareness
 
Share the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development BoardShare the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development Board
 
Linux Kernel Platform Development: Challenges and Insights
 Linux Kernel Platform Development: Challenges and Insights Linux Kernel Platform Development: Challenges and Insights
Linux Kernel Platform Development: Challenges and Insights
 
Embedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debuggingEmbedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debugging
 
Roll your own toy unix clone os
Roll your own toy unix clone osRoll your own toy unix clone os
Roll your own toy unix clone os
 
Basic Linux kernel
Basic Linux kernelBasic Linux kernel
Basic Linux kernel
 
Linux: the first second
Linux: the first secondLinux: the first second
Linux: the first second
 
Advanced Diagnostics 2
Advanced Diagnostics 2Advanced Diagnostics 2
Advanced Diagnostics 2
 
CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...
CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...
CONFidence 2017: Escaping the (sand)box: The promises and pitfalls of modern ...
 
Mainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernels
Mainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernelsMainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernels
Mainline kernel on ARM Tegra20 devices that are left behind on 2.6 kernels
 
Linux+sensor+device-tree+shell=IoT !
Linux+sensor+device-tree+shell=IoT !Linux+sensor+device-tree+shell=IoT !
Linux+sensor+device-tree+shell=IoT !
 
HKG15-409: ARM Hibernation enablement on SoCs - a case study
HKG15-409: ARM Hibernation enablement on SoCs - a case studyHKG15-409: ARM Hibernation enablement on SoCs - a case study
HKG15-409: ARM Hibernation enablement on SoCs - a case study
 
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysisKernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
 
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheapUWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
 
Linux Kernel Debugging
Linux Kernel DebuggingLinux Kernel Debugging
Linux Kernel Debugging
 
Ganeti - build your own cloud
Ganeti - build your own cloudGaneti - build your own cloud
Ganeti - build your own cloud
 
Introduction to Modern U-Boot
Introduction to Modern U-BootIntroduction to Modern U-Boot
Introduction to Modern U-Boot
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting Started
 

More from Jian-Hong Pan

國稅局,我也好想用電腦報稅
國稅局,我也好想用電腦報稅國稅局,我也好想用電腦報稅
國稅局,我也好想用電腦報稅
Jian-Hong Pan
 
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
Jian-Hong Pan
 
A Journey to Boot Linux on Raspberry Pi
A Journey to Boot Linux on Raspberry PiA Journey to Boot Linux on Raspberry Pi
A Journey to Boot Linux on Raspberry Pi
Jian-Hong Pan
 
Have a Simple Modbus Server
Have a Simple Modbus ServerHave a Simple Modbus Server
Have a Simple Modbus Server
Jian-Hong Pan
 
Software Packaging for Cross OS Distribution
Software Packaging for Cross OS DistributionSoftware Packaging for Cross OS Distribution
Software Packaging for Cross OS Distribution
Jian-Hong Pan
 
Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!
Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!
Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!
Jian-Hong Pan
 
LoRaWAN class module and subsystem
LoRaWAN class module and subsystemLoRaWAN class module and subsystem
LoRaWAN class module and subsystem
Jian-Hong Pan
 
Let's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoT
Let's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoTLet's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoT
Let's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoT
Jian-Hong Pan
 
The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017
Jian-Hong Pan
 
Build a Micro HTTP Server for Embedded System
Build a Micro HTTP Server for Embedded SystemBuild a Micro HTTP Server for Embedded System
Build a Micro HTTP Server for Embedded System
Jian-Hong Pan
 
Micro HTTP Server Implemented in C @ COSCUP 2016
Micro HTTP Server Implemented in C @ COSCUP 2016Micro HTTP Server Implemented in C @ COSCUP 2016
Micro HTTP Server Implemented in C @ COSCUP 2016
Jian-Hong Pan
 
Bind Python and C @ COSCUP 2015
Bind Python and C @ COSCUP 2015Bind Python and C @ COSCUP 2015
Bind Python and C @ COSCUP 2015
Jian-Hong Pan
 
Find the bottleneck of your system
Find the bottleneck of your systemFind the bottleneck of your system
Find the bottleneck of your system
Jian-Hong Pan
 
Learn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDev
Learn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDevLearn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDev
Learn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDev
Jian-Hong Pan
 
Debug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code Meetup
Debug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code MeetupDebug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code Meetup
Debug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code Meetup
Jian-Hong Pan
 
Make Your Own Developement Board @ 2014.4.21 JuluOSDev
Make Your Own Developement Board @ 2014.4.21 JuluOSDevMake Your Own Developement Board @ 2014.4.21 JuluOSDev
Make Your Own Developement Board @ 2014.4.21 JuluOSDevJian-Hong Pan
 
The Simple Scheduler in Embedded System @ OSDC.TW 2014
The Simple Scheduler in Embedded System @ OSDC.TW 2014The Simple Scheduler in Embedded System @ OSDC.TW 2014
The Simple Scheduler in Embedded System @ OSDC.TW 2014
Jian-Hong Pan
 
Node.js 1, 2, 3
Node.js 1, 2, 3Node.js 1, 2, 3
Node.js 1, 2, 3
Jian-Hong Pan
 

More from Jian-Hong Pan (18)

國稅局,我也好想用電腦報稅
國稅局,我也好想用電腦報稅國稅局,我也好想用電腦報稅
國稅局,我也好想用電腦報稅
 
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
 
A Journey to Boot Linux on Raspberry Pi
A Journey to Boot Linux on Raspberry PiA Journey to Boot Linux on Raspberry Pi
A Journey to Boot Linux on Raspberry Pi
 
Have a Simple Modbus Server
Have a Simple Modbus ServerHave a Simple Modbus Server
Have a Simple Modbus Server
 
Software Packaging for Cross OS Distribution
Software Packaging for Cross OS DistributionSoftware Packaging for Cross OS Distribution
Software Packaging for Cross OS Distribution
 
Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!
Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!
Nasa Hackthon 2018 Light Wonder - Go! Polar Bear!
 
LoRaWAN class module and subsystem
LoRaWAN class module and subsystemLoRaWAN class module and subsystem
LoRaWAN class module and subsystem
 
Let's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoT
Let's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoTLet's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoT
Let's Have an IEEE 802.15.4 over LoRa Linux Device Driver for IoT
 
The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017The Considerations for Internet of Things @ 2017
The Considerations for Internet of Things @ 2017
 
Build a Micro HTTP Server for Embedded System
Build a Micro HTTP Server for Embedded SystemBuild a Micro HTTP Server for Embedded System
Build a Micro HTTP Server for Embedded System
 
Micro HTTP Server Implemented in C @ COSCUP 2016
Micro HTTP Server Implemented in C @ COSCUP 2016Micro HTTP Server Implemented in C @ COSCUP 2016
Micro HTTP Server Implemented in C @ COSCUP 2016
 
Bind Python and C @ COSCUP 2015
Bind Python and C @ COSCUP 2015Bind Python and C @ COSCUP 2015
Bind Python and C @ COSCUP 2015
 
Find the bottleneck of your system
Find the bottleneck of your systemFind the bottleneck of your system
Find the bottleneck of your system
 
Learn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDev
Learn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDevLearn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDev
Learn How to Develop Embedded System for ARM @ 2014.12.22 JuluOSDev
 
Debug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code Meetup
Debug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code MeetupDebug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code Meetup
Debug C/C++ Programs More Comfortably @ 2014.12.14 Trace Code Meetup
 
Make Your Own Developement Board @ 2014.4.21 JuluOSDev
Make Your Own Developement Board @ 2014.4.21 JuluOSDevMake Your Own Developement Board @ 2014.4.21 JuluOSDev
Make Your Own Developement Board @ 2014.4.21 JuluOSDev
 
The Simple Scheduler in Embedded System @ OSDC.TW 2014
The Simple Scheduler in Embedded System @ OSDC.TW 2014The Simple Scheduler in Embedded System @ OSDC.TW 2014
The Simple Scheduler in Embedded System @ OSDC.TW 2014
 
Node.js 1, 2, 3
Node.js 1, 2, 3Node.js 1, 2, 3
Node.js 1, 2, 3
 

Recently uploaded

ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
CatarinaPereira64715
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 

Recently uploaded (20)

ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 

Let's trace Linux Lernel with KGDB @ COSCUP 2021

  • 1. Let's trace Linux Kernel with KGDB 潘建宏 Jian-Hong Pan (StarNight) @ COSCUP 2021
  • 2. Who am I 潘建宏 / Jian-Hong Pan (StarNight) Endless OS Foundation You can find me at ● http://www.slideshare.net/chienhungpan/ ● GitHub: starnight ● Email: jhp [AT] endlessos.org chienhung.pan [AT] gmail.com
  • 3. Outline ● How to debug Linux kernel ● KGDB ○ The Linux kernel build config ● KGDB with VM (QEMU) ● KGDB with real system (Raspberry Pi 4B) ○ How does Raspberry Pi 4B boot kernel ○ Raspberry Pi 4B’s JTAG ○ OpenOCD & the JTAG adapter ○ Add U-Boot to catch up kernel boot ○ Real debug ● Reference
  • 4. How to Debug Linux kernel??? ● Read the kernel message log, then grep the key words in kernel source codes. ● Add some more debug messages with printk like functions in interesting parts. ● Kprobes, ftrace, eBPF .... ● Export the signals to GPIO pins!!!??? Basic idea is “trace the code path to understand why it goes here, there.” Can kernel be debugged with GDB like the user space programs?
  • 5. KGDB The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware interfaces allow to debug the Linux kernel and its modules during runtime using gdb. - from Debugging kernel and modules via gdb Host Target Machine Debug Line The debug line could be professional like JTAG, or simple one like serial.
  • 6. Build Linux Kernel on Host Build the kernel with following configuration (at least) ● # CONFIG_RANDOMIZE_BASE is not set ● CONFIG_DEBUG_INFO=y ● # CONFIG_DEBUG_INFO_REDUCED is not set ● CONFIG_GDB_SCRIPTS=y ● CONFIG_FRAME_POINTER=y
  • 7. Start with Virtual Environment - QEMU Host Target Machine (QEMU VM) Debug Line built kernel Install
  • 8. Install the Built Kernel into the Target Machine (VM) Have a QEMU guest VM with an OS (for example, Debian) as the target machine 1. Start the VM with 9p shared folder (on Host) $ qemu-system-x86_64 -m 2048 -smp 2 qemu-images/debian.qcow2 -enable-kvm -virtfs local,path=linux-stable/,mount_tag=host0,security_model=mapped-xattr,id=host_share 2. Create a folder for mounting the shared folder from host (on guest VM) $ mkdir linux-stable && sudo mount -t 9p -o trans=virtio host0 linux-stable/ 3. Installed the built kernel into the guest VM (on guest VM) $ cd linux-stable && sudo make install modules_install
  • 9. Debug the Kernel on Target Machine (VM) Host Target Machine (QEMU VM) Debug Line built kernel gdb connection
  • 10. Enable the gdb stub of QEMU/KVM ● Start the guest VM with enabled gdb stub (on Host) $ qemu-system-x86_64 -m 2048 -smp 2 qemu-images/debian.qcow2 -s -S ● Must disable kvm ● Manual of qemu-system-x86_64: -s ... freeze CPU at startup (use 'c' to start execution) -gdb dev accept gdb connection on 'dev'. (QEMU defaults to starting the guest without waiting for gdb to connect; use -S too if you want it to not start execution.) -S shorthand for -gdb tcp::1234
  • 11. Use GDB on Host 1. Start gdb in the linux project folder $ cd /path/to/linux-stable && gdb vmlinux 2. Attach to the booted geust VM (gdb) target remote localhost:1234 3. Use gdb: set break points, continue, debug … # Add a break point for test (gdb) break drm_core_init Breakpoint 1 at 0xffffffff82a7ebf0: file drivers/gpu/drm/drm_drv.c, line 1116. # Continue the kernel (gdb) continue Continuing. Thread 1 hit Breakpoint 1, drm_core_init () at drivers/gpu/drm/drm_drv.c:1116 1116 drm_connector_ida_init(); (gdb) backtrace #0 drm_core_init () at drivers/gpu/drm/drm_drv.c:1116 #1 0xffffffff81000d61 in do_one_initcall (fn=0xffffffff82a7ebf0 <drm_core_init>) at init/main.c:1205
  • 12.
  • 13. Let’s debug in the Real World! Raspberry Pi as the target machine
  • 14. Debug the Kernel on Target Machine (Raspberry Pi) Target Machine (Raspberry Pi) Debug Line JTAG Host built kernel OpenOCD
  • 15. How Debian Boots on Raspberry Pi 4B ● Download Debain images for Raspberry Pi 4B and install it by following RaspberryPiImages ● The image has 2 partitions. First one is the boot partition. ● Here are the files in the boot partition $ ls bcm2711-rpi-4-b.dtb cmdline.txt fixup_cd.dat start4db.elf start_x.elf bcm2837-rpi-3-a-plus.dtb config.txt fixup.dat start4.elf sysconf.txt bcm2837-rpi-3-b.dtb fixup4cd.dat fixup_db.dat start4x.elf vmlinuz-5.10.0-7-arm64 bcm2837-rpi-3-b-plus.dtb fixup4.dat fixup_x.dat start_cd.elf bcm2837-rpi-cm3-io3.dtb fixup4db.dat initrd.img-5.10.0-7-arm64 start_db.elf bootcode.bin fixup4x.dat start4cd.elf start.elf Device Boot Start End Sectors Size Id Type /dev/sda1 8192 614399 606208 296M c W95 FAT32 (LBA) /dev/sda2 614400 124735487 124121088 59.2G 83 Linux
  • 16. Debian’s config.txt & cmdline.txt $ cat config.txt # Switch the CPU from ARMv7 into ARMv8 (aarch64) mode arm_64bit=1 enable_uart=1 upstream_kernel=1 kernel=vmlinuz-5.10.0-7-arm64 # For details on the initramfs directive, see # https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=10532 initramfs initrd.img-5.10.0-7-arm64 $ cat cmdline.txt console=tty0 console=ttyS1,115200 root=/dev/mmcblk1p2 rw fsck.repair=yes net.ifnames=0 rootwait
  • 17. Enable Raspberry Pi 4B’s JTAG ● The chip is BCM2711 ● Raspberry Pi 4B maps JTAG pins to GPIO ○ Setting enable_jtag_gpio=1 selects Alt4 mode for GPIO pins 22-27, and sets up some internal SoC connections, thus enabling the JTAG interface for the ARM CPU. Pin # Function GPIO22 ARM_TRST GPIO23 ARM_RTCK GPIO24 ARM_TDO GPIO25 ARM_TCK GPIO26 ARM_TDI GPIO27 ARM_TMS Reference: Raspberry Pi Documentation's GPIO control in config.txt
  • 18. Table 94. GPIO Pins Alternative Function Assignment of BCM2711 ARM Peripherals JTAG interface
  • 19. Open On-Chip Debugger (OpenOCD) ● It does so with the assistance of a debug adapter, which is a small hardware module which helps provide the right kind of electrical signaling to the target being debugged. These are required since the debug host (on which OpenOCD runs) won’t usually have native support for such signaling, or the connector needed to hook up to the target. ● A JTAG Adapter supports JTAG signaling, and is used to communicate with JTAG (IEEE 1149.1) compliant TAPs on your target board. A TAP is a “Test Access Port”, a module which processes special instructions and data. TAPs are daisy-chained within and between chips and boards. JTAG supports debugging and boundary scan operations. ● List of Debug Adapter Hardware, or search the list of tcl/interface in OpenOCD repository directly Reference: OpenOCD User's Guid - About
  • 20. Try to Buy a JTAG Adatper ● I do not have much money :( $$$$$ ● It is really hard to buy and get the electronic parts, due to the pandemic. ● I have no idea why all of the JTAG adapters come from overseas. ● The shipment becomes extremely expensive and much longer waiting time. So, I can only suspend this study. Sad … T^T Until ...
  • 21. Thanks to Mat lend me a FT2232H-56Q Mini Module as a JTAG interface!!! Hacking Thursday! Hurray!!!
  • 22. JTAG interface Table 3.13 FT2232H Pin Configurations for 56-pin VQFN package of FT2232H Dual High Speed USB to Multipurpose UART/FIFO IC Datasheet
  • 23. JTAG interface Table 3.1 FT2232H-56Q Mini Module Connection – CN2 of FT2232H-56Q Mini Module Datasheet
  • 24. Prepare JTAG Interface cfg for OpenOCD # Refer to tcl/interface/ftdi/minimodule.cfg # # FTDI MiniModule # # http://www.ftdichip.com/Support/Documents/DataSheets/Modules/DS_FT2232H_Mini_Module.pdf # adapter driver ftdi #ftdi_device_desc "FT2232H-56Q MiniModule" ftdi_vid_pid 0x0403 0x6010 # Every pin set as high impedance except TCK, TDI, TDO and TMS ftdi_layout_init 0x0008 0x000b transport select jtag # nTRST defined on pin CN2-13 of the MiniModule (pin ADBUS5 [AD5] on the FT2232H chip) # This choice is arbitrary. Use other GPIO pin if desired. ftdi_layout_signal nTRST -data 0x0020 -oe 0x0020
  • 25. Execute OpenOCD with the cfgs $ openocd -f minimodule.cfg -c "set USE_SMP 1" -f bcm2711.cfg -c "reset_config trst_only" Open On-Chip Debugger 0.11.0 Licensed under GNU GPL v2 For bug reports, read http://openocd.org/doc/doxygen/bugs.html 1 Info : Listening on port 6666 for tcl connections Info : Listening on port 4444 for telnet connections Info : clock speed 1000 kHz Info : JTAG tap: bcm2711.cpu tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver: 0x4) Info : bcm2711.cpu0: hardware has 6 breakpoints, 4 watchpoints Info : bcm2711.cpu1: hardware has 6 breakpoints, 4 watchpoints Info : bcm2711.cpu2: hardware has 6 breakpoints, 4 watchpoints Info : bcm2711.cpu3: hardware has 6 breakpoints, 4 watchpoints Info : starting gdb server for bcm2711.cpu0 on 3333 Info : Listening on port 3333 for gdb connections Note: the patch tcl/board: Add Raspberry Pi 4 model B board
  • 26. But the debug connection, including gdb and OpenOCD is too late to catch up the kernel boot. And, Raspberry Pi does not expose JTAG System Reset (SRST) pin.
  • 27. The RUN/GLOBAL_EN pin on Raspberry Pi boards might be implemented as the System Reset (SRST) pin. Re: openocd, jtag, fyi But, OpenOCD still cannot catch up and halt target system on time.
  • 28. How about intercept boot with U-Boot ...
  • 29. 3th stage Kernel boots Boot Linux kernel via U-Boot on Raspberry Pi 4B Raspberry Pi OS’ firmwares 3th stage U-Boot loads kernel, initramfs and device tree, then boots kernel with boot script 1st stage Boot from GPU with firmware in ROM loads start4x.elf ... in FAT 2rd stage Start4.elf loads DT, kernel … and starts CPU U-Boot
  • 30. Build U-Boot 1. Build U-Boot & install $ make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- rpi_4_defconfig $ make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- $ cp u-boot.bin <RPi image’s root partition path>/boot/kernel8.img 2. Prepare the boot script for U-Boot 3. Prepare the boot script and following configuration file for U-Boot 4. Get RPi 4B’s device tree blob from Raspberry Pi OS for U-Boot 5. Modify config.txt to make boot firmware load and execute the kernel8.img
  • 31. Prepare U-Boot script $ cat boot.src echo "Load u-Boot environment from uEnv.txt" load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} uEnv.txt env import -t ${scriptaddr} ${filesize} echo "Load device tree from "${fdtdir}/${fdtfile} load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${fdtdir}/${fdtfile} echo "Load kernel from "${kernel_image} load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${kernel_image} echo "Load RAM disk from "${ramdisk_image} load ${devtype} ${devnum}:${distro_bootpart} ${ramdisk_addr_r} ${ramdisk_image} echo "Boot ..." setenv bootargs ${bootargs} booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r} $ mkimage -T script -C none -d boot.src <Path to image’s boot partition>/boot.scr
  • 32. And, the uEnv.txt for U-Boot script $ cat uEnv.txt kernel_image=vmlinux-5.12.10 ramdisk_image=initrd.img-5.12.10 fdtdir=dtbs/5.12.10 bootargs=console=tty0 console=ttyS1,115200 root=/dev/mmcblk1p2 rw fsck.repair=yes net.ifnames=0 rootwait
  • 33. Modify RPi’s config.txt for Executing kernel8.img $ cat config.txt arm_64bit=1 enable_uart=1 enable_jtag_gpio=1
  • 34. The New Status of the Boot Partition $ ls bcm2711-rpi-4-b.dtb fixup4db.dat start4x.elf bcm2837-rpi-3-a-plus.dtb fixup4x.dat start_cd.elf bcm2837-rpi-3-b.dtb fixup_cd.dat start_db.elf bcm2837-rpi-3-b-plus.dtb fixup.dat start.elf bcm2837-rpi-cm3-io3.dtb fixup_db.dat start_x.elf bootcode.bin fixup_x.dat sysconf.txt boot.scr initrd.img-5.10.0-7-arm64 System.map-5.12.10 cmdline.txt initrd.img-5.12.10 uEnv.txt config.txt kernel8.img vmlinux-5.12.10 dtbs start4cd.elf vmlinuz-5.10.0-7-arm64 fixup4cd.dat start4db.elf fixup4.dat start4.elf
  • 35.
  • 36. Press any key to stop autoboot in U-Boot U-Boot 2021.04 (Jun 20 2021 - 17:36:34 +0800) ... scanning bus xhci_pci for devices... 3 USB Device(s) found scanning usb for storage devices... 0 Storage Device(s) found Hit any key to stop autoboot: 0 U-Boot> boot switch to partitions #0, OK mmc0 is current device Scanning mmc 0:1... Found U-Boot script /boot.scr 724 bytes read in 13 ms (53.7 KiB/s) … Starting kernel ... Press a key to make U-boot paused, then connect the GDB via OpenOCD + JTAG. Finally, continue boot
  • 37. Launch OpenOCD to Start GDB Server with JTAG $ openocd -f minimodule.cfg -c "set USE_SMP 1" -f bcm2711.cfg -c "reset_config trst_only" Open On-Chip Debugger 0.11.0 Licensed under GNU GPL v2 For bug reports, read http://openocd.org/doc/doxygen/bugs.html 1 Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'. Info : Listening on port 6666 for tcl connections Info : Listening on port 4444 for telnet connections Info : clock speed 1000 kHz Info : JTAG tap: bcm2711.cpu tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver: 0x4) Info : bcm2711.cpu0: hardware has 6 breakpoints, 4 watchpoints Info : bcm2711.cpu1: hardware has 6 breakpoints, 4 watchpoints Info : bcm2711.cpu2: hardware has 6 breakpoints, 4 watchpoints Info : bcm2711.cpu3: hardware has 6 breakpoints, 4 watchpoints Info : starting gdb server for bcm2711.cpu0 on 3333 Info : Listening on port 3333 for gdb connections
  • 38. Debug with GDB via the OpenOCD + JTAG $ aarch64-linux-gnu-gdb vmlinux ... Reading symbols from vmlinux... (gdb) target extended-remote localhost:3333 Remote debugging using localhost:3333 0x000000003b39133c in ?? () (gdb) continue Continuing. ^C ... Program received signal SIGINT, Interrupt. mem_serial_in (p=<optimized out>, offset=<optimized out>) at drivers/tty/serial/8250/8250_port.c:399 399 return readb(p->membase + offset); (gdb) hbreak drm_core_init Hardware assisted breakpoint 1 at 0xffff80001115775c: file drivers/gpu/drm/drm_drv.c, line 1044. (gdb) continue Continuing. ... Continue U-Boot to boot the kernal image Intercept/Interrupt the kernal boot at a proper point hbreak in Setting Breakpoints
  • 39. Debug with GDB via the OpenOCD + JTAG (cont.) Breakpoint 1, drm_core_init () at drivers/gpu/drm/drm_drv.c:1044 1044 drm_connector_ida_init(); (gdb) backtrace #0 drm_core_init () at drivers/gpu/drm/drm_drv.c:1044 #1 0xffff800010012e80 in do_one_initcall ( fn=0xffff80001115775c <drm_core_init>) at init/main.c:1226 #2 0xffff8000111212b4 in do_initcall_level ( command_line=0xffff000040264500 "console", level=6) at init/main.c:1299 #3 do_initcalls () at init/main.c:1315 #4 do_basic_setup () at init/main.c:1335 #5 kernel_init_freeable () at init/main.c:1537 #6 0xffff800010c5419c in kernel_init (unused=<optimized out>) at init/main.c:1424 #7 0xffff80001001448c in ret_from_fork () at arch/arm64/kernel/entry.S:955 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)
  • 40. Summary ● Some key items for KGDB in Linux kernel building config ● Tried KGDB with QEMU easily ○ Learned steps to debug the Lnux kernel on QEMU guest OS ● Tried KGDB with real physical system Raspberry Pi 4B ○ The JTAG interface, the adapter and corresponding hardware feature ○ BCM2711 on Raspberry Pi 4B’s JTAG feature and configuration ○ Debugged the Linux kernel on Raspberry Pi 4B with KGDB via OpenOCD + JTAG adapter ● If you have some money, please buy a professional JTAG adapter. It has less barriers and saves your time. ● KGDB is not the only way to debug Linux kernel. The printk series might be good enough for most of cases. Especially, timing related issues.
  • 41. Reference ● Debugging kernel and modules via gdb ● JTAG ● OpenOCD ● Bare Metal Raspberry Pi 3B+: JTAG ● Baremetal Raspberry Pi 4 with FT2232H ● FT2232H Dual High Speed USB to Multipurpose UART/FIFO IC datasheet ● FT2232H-56Q Mini Module Datasheet ● Raspberry Pi → Documentation → The boot folder ● Device Tree ● RPi U-Boot ● GDB → Setting Breakpoints ● JTAG Reset on Raspberry Pi forum ● A Journey to Boot Linux on Raspberry Pi