Step-by-step instructions to debug using kgdb - that work, symbolic debug using Xilinx's SDK and PetaLinux Tools and much more!!!

1. Beyond printk: Efficient Zynq
UltraScale+ MPSoC Linux
Debugging and Development
Presented by Zach Pfeffer
Owner of Centennial Software Solutions
Need help? Email inquires@centennialsoftwaresolutions.com
or call +1-303-246-3297
©2018 Zach Pfeffer

2. Motivation
• Debugging with JTAG or GDB can be more efficient than printk debug
©2018 Zach Pfeffer

3. printk debug
• Inserting printk’s into code
• Recompiling the kernel
• Running the kernel
• Analyze the values printed and the order
• Repeat
©2018 Zach Pfeffer

4. GDB or JTAG Debug
• Run the kernel
• Look at everything at everypoint
More efficient
©2018 Zach Pfeffer

5. Note
• The 2017.4 SDK would not load symbols for me
• Therefore the presentation is primarly about kgdb:
• Configuration
• Set up
• Some use
This is why people still use printk debug – it always works©2018 Zach Pfeffer

6. Topics
• Tools Used
• Get PetaLinux Tools 2017.4
• Setup and Connect to kgdb
• Attempt to Load vmlinux Symbols using the SDK and System
Debugger
• Working Without Symbols
• Call to Action for Xilinx
©2018 Zach Pfeffer

7. Tools Used Tools Used
Get PetaLinux Tools 2017.4
Setup and Connect to kgdb
Attempt to Load vmlinux Symbols using the SDK and System
Debugger
Working Without Symbols
Call to Action for Xilinx
©2018 Zach Pfeffer

8. Tools Used
• PetaLinux Tools 2017.4
• Xilinx SDK 2017.4
©2018 Zach Pfeffer

9. Linux Kernel Used
https://github.com/Xilinx/linux-
xlnx/tree/b450e900fdb473a53613ad014f31eedbc80b1
c90 included in:
PetaLinux Tools 2017.4
©2018 Zach Pfeffer

10. Prerequisites
Some of the commands used vary from PetaLinux Tools
See Effective Linux development using PetaLinux Tools
for explanations
©2018 Zach Pfeffer

11. Get PetaLinux Tools 2017.4Tools Used
Get PetaLinux Tools 2017.4
Setup and Connect to kgdb
Attempt to Load vmlinux Symbols using the SDK and System
Debugger
Working Without Symbols
Call to Action for Xilinx
©2018 Zach Pfeffer

12. Walk-through was done with PetaLinux Tools 2017.4
Steps may depend on that version
Here’s how to get it
©2018 Zach Pfeffer

13. Find and Get PetaLinux Tools 2017.4
1. Go to xilinx.com
2. Click Developer Zone
3. Click
Embedded
Development
©2018 Zach Pfeffer

14. 5. Click OS &
Middleware
Get PetaLinux Tools 2017.4
4.
Scroll
Down
©2018 Zach Pfeffer

15. 7. Click PetaLinux
tools
Get PetaLinux Tools 2017.4
6.
Scroll
Down
©2018 Zach Pfeffer

16. Get PetaLinux Tools 2017.4
8. Click Download
PetaLinux
©2018 Zach Pfeffer

17. Get PetaLinux Tools 2017.4
9. Click 2017.4
©2018 Zach Pfeffer

18. Get PetaLinux Tools 2017.4
11. Click PetaLinux
2017.4 Installer
10.
Scroll
Down
Also at:
https://www.xilinx.com/member/forms/download/xef.html?filename=petalinux-v2017.4-final-installer.run
©2018 Zach Pfeffer

19. Setup and Connect to kgdbTools Used
Get PetaLinux Tools 2017.4
Setup and Connect to kgdb
Attempt to Load vmlinux Symbols using the SDK and System
Debugger
Working Without Symbols
Call to Action for Xilinx
©2018 Zach Pfeffer

20. Kgdb is a source level debugger for the Linux kernel
gdb can “break-in” to the kernel to inspect memory, variable
and look through call stack information
-from link
These are the exact instructions that worked for me
©2018 Zach Pfeffer

21. Needed Kernel Config Options
Turn on CONFIG_KGDB
Turn on CONFIG_DEBUG_INFO
Turn on CONFIG_FRAME_POINTER
Turn off CONFIG_STRICT_KERNEL_RWX (not needed in our case)
Turn on CONFIG_KGDB_SERIAL_CONSOLE
©2018 Zach Pfeffer

22. Needed Kernel Config Options
Config Option Required/Suggested Description
CONFIG_KGDB Required on Enables KGDB
CONFIG_DEBUG_INFO Suggested on Enables symbols in vmlinux
CONFIG_FRAME_POINTER Suggested on Insert code so a GDB can create
backtraces
CONFIG_STRICT_KERNEL_RWX Suggested off Cannot set some software
breakpoints if On
CONFIG_KGDB_SERIAL_CONSOLE Required on if
debugging over serial)
Enabled target connection over
serial
©2018 Zach Pfeffer

23. Run menuconfig to Set Options
bitbake virtual/kernel -c menuconfig
©2018 Zach Pfeffer

24. Turn on CONFIG_KGDB
│ Symbol: KGDB [=n] │
│ Type : boolean │
│ Prompt: KGDB: kernel debugger │
│ Location: │
│ (1) -> Kernel hacking │
│ Defined at lib/Kconfig.kgdb:5 │
│ Depends on: HAVE_ARCH_KGDB [=y] && DEBUG_KERNEL [=y] │
©2018 Zach Pfeffer

25. Turn on
CONFIG_KGDB ©2018 Zach Pfeffer

26. Turn on CONFIG_DEBUG_INFO
│ Symbol: DEBUG_INFO [=y] │
│ Type : boolean │
│ Prompt: Compile the kernel with debug info │
│ Location: │
│ -> Kernel hacking │
│ (1) -> Compile-time checks and compiler options │
│ Defined at lib/Kconfig.debug:120 │
│ Depends on: DEBUG_KERNEL [=y] && !COMPILE_TEST [=n] │
©2018 Zach Pfeffer

27. Turn on
CONFIG_DEBUG_INFO ©2018 Zach Pfeffer

28. Turn on CONFIG_FRAME_POINTER
│ Symbol: FRAME_POINTER [=y] │
│ Type : boolean │
│ Prompt: Compile the kernel with frame pointers │
│ Location: │
│ -> Kernel hacking │
│ (1) -> Compile-time checks and compiler options │
│ Defined at lib/Kconfig.debug:336 │
│ Depends on: DEBUG_KERNEL [=y] && (CRIS || M68K || FRV || UML || AVR32 │
│ Selected by: ARM64 [=y] || FAULT_INJECTION_STACKTRACE_FILTER [=n] && │
©2018 Zach Pfeffer

29. Turn on
CONFIG_FRAME_POINTER ©2018 Zach Pfeffer

30. Turn off CONFIG_STRICT_KERNEL_RWX
Kconfig not present in 2017.4 kernel
©2018 Zach Pfeffer

31. Turn on CONFIG_KGDB_SERIAL_CONSOLE
│ Symbol: KGDB_SERIAL_CONSOLE [=n] │
│ Type : tristate │
│ Prompt: KGDB: use kgdb over the serial console │
│ Location: │
│ -> Kernel hacking │
│ (4) -> KGDB: kernel debugger (KGDB [=n]) │
│ Defined at lib/Kconfig.kgdb:21 │
│ Depends on: KGDB [=n] && TTY [=y] && HW_CONSOLE [=y] │
│ Selects: CONSOLE_POLL [=n] && MAGIC_SYSRQ [=y] │
©2018 Zach Pfeffer

32. Turn on
CONFIG_KGDB_SERIAL_CONSOLE ©2018 Zach Pfeffer

33. Recompile the Kernel
bitbake virtual/kernel -c compile -C compile
©2018 Zach Pfeffer

34. Package for U-Boot
Package arch/arm/boot/Image for U-Boot with:
./build/tmp/sysroots/x86_64-linux/usr/bin/mkimage -A arm64 -T kernel -C none
-a 0x00080000 -e 00080000 -d ./build/tmp/work/plnx_aarch64-xilinx-
linux/linux-xlnx/4.9-xilinx-v2017.4+gitAUTOINC+b450e900fd-r0/linux-
plnx_aarch64-standard-build/arch/arm64/boot/Image ./images/linux/uImage
-A set architecture to arm64
-T set image to kernel
-C no compression
-a set load address to 0x80000
-e set entry point to 0x80000
-d image
U-Boot image
©2018 Zach Pfeffer

35. Update bootargs in system-user.dtsi
File @ ./project-spec/meta-user/recipes-bsp/device-
tree/files/system-user.dtsi
chosen {
bootargs = "earlycon kgdbwait";
stdout-path = "serial0:115200n8";
}; Add
kgdbwait
©2018 Zach Pfeffer

36. Rebuild the DTB
bitbake virtual/dtb -c compile -C compile
©2018 Zach Pfeffer

37. Open Minicom
minicom -o -w -C May-29th-2018.log
-o Do not initialize. Skip init
code. Can quit and restart a
session.
-w Turn line wrap on -C Open capture file.
©2018 Zach Pfeffer

38. Config minicom
1. Type Control-A Z in minicom
+-------------------------------------------------------------------+
| Minicom Command Summary |
| |
| Commands can be called by CTRL-A <key> |
| |
| Main Functions Other Functions |
| |
| Dialing directory..D run script (Go)....G | Clear Screen.......C |
| Send files.........S Receive files......R | cOnfigure Minicom..O |
| comm Parameters....P Add linefeed.......A | Suspend minicom....J |
| Capture on/off.....L Hangup.............H | eXit and reset.....X |
| send break.........F initialize Modem...M | Quit with no reset.Q |
| Terminal settings..T run Kermit.........K | Cursor key mode....I |
| lineWrap on/off....W local Echo on/off..E | Help screen........Z |
| Paste file.........Y Timestamp toggle...N | scroll Back........B |
| Add Carriage Ret...U |
| |
| Select function or press Enter for none. |
+-------------------------------------------------------------------+
2. Type O
©2018 Zach Pfeffer

39. +------------------------------------------
| A - Serial Device : /dev/ttyUSB0
| B - Lockfile Location : /var/lock
| C - Callin Program :
| D - Callout Program :
| E - Bps/Par/Bits : 115200 8N1
| F - Hardware Flow Control : No
| G - Software Flow Control : No
|
| Change which setting?
+------------------------------------------
+-----[configuration]------+
| Filenames and paths |
| File transfer protocols |
| Serial port setup |
| Modem and dialing |
| Screen and keyboard |
| Save setup as dfl |
| Save setup as.. |
| Exit |
+--------------------------+
3. Select Serial
port setup
Config minicom
5. Set
115200 8N1
4. Set
/dev/ttyUSB0
©2018 Zach Pfeffer

40. +-----[configuration]------+
| Filenames and paths |
| File transfer protocols |
| Serial port setup |
| Modem and dialing |
| Screen and keyboard |
| Save setup as dfl |
| Save setup as.. |
| Exit |
+--------------------------+
Config minicom
6. Save setup as dfl
7. Exit
©2018 Zach Pfeffer

41. Open Minicom again (with configuration)
minicom -o -w -C May-29th-2018.log
-o Do not initialize. Skip init
code. Can quit and restart a
session.
-w Turn line wrap on -C Open capture file.
©2018 Zach Pfeffer

42. L/6NZxJ2UOfZgiPyoSE4DU/D996T07aeMBOWSYFBomD7diKkutWo
LKmB8I0dpLGPPmF1uW2qi+r
cVnQEmHbkzqk7OL7t8vupaqTgdTSFlM0AhmZMVMWic9AmWE4bsJ0
TDppO7GK9Md7xad6J1hLJdp
7U/ul root@mtd_board
Fingerprint: md5
f1:f5:98:89:58:f1:76:8b:b1:3a:96:54:ce:59:4a:96
dropbear.
Starting tcf-agent: OK
PetaLinux 2017.4 mtd_board /dev/ttyPS0
mtd_board login: root
Password: …and log in
Login: root
Password: root
Boot Unit
©2018 Zach Pfeffer

43. echo "ttyPS0,115200" > /sys/module/kgdboc/parameters/kgdboc
[ 48.875933] KGDB: Registered I/O driver kgdboc
[ 48.880335] KGDB: Waiting for connection from remote gdb...
At the command line echo…
Make sure you see these lines
In Minicom:
©2018 Zach Pfeffer

44. Close Minicom
1. Press Control-A then x
+----------------------+
| Leave Minicom? |
| Yes No |
+----------------------+
2.
Select
Yes ©2018 Zach Pfeffer

45. Get vmlinux
build/tmp/work/plnx_aarch64-xilinx-linux/linux-xlnx/4.9-xilinx-
v2017.4+gitAUTOINC+b450e900fd-r0/linux-plnx_aarch64-standard-
build/arch/arm64/boot/vmlinux
Where Linux kernel artifacts are put
find . -name "vmlinux"
Run find from your PetaLinux Project directory
Look for the vmlinux in arch/arm64/boot
©2018 Zach Pfeffer

46. Get gdb
/home/pfefferz/tools/opt/pkg/petalinux/tools/linux-i386/aarch64-
linux-gnu/bin/aarch64-linux-gnu-gdb
This is a standalone exe
find $PETALINUX -name "aarch64-linux-gnu-gdb"
Run find from where PetaLinux Tools was installed directory
Look for the aarch64-linux-gnu-gdb in bin/
©2018 Zach Pfeffer

47. Launch gdb with vmlinux & specify serial port
speed
PATH=/home/pfefferz/tools/opt/pkg/petalinux/tools/
linux-i386/aarch64-linux-gnu/bin/:$PATH aarch64-
linux-gnu-gdb -b 115200
~/plprj4/mtd_build/build/tmp/work/plnx_aarch64-
xilinx-linux/linux-xlnx/4.9-xilinx-
v2017.4+gitAUTOINC+b450e900fd-r0/linux-
plnx_aarch64-standard-
build/arch/arm64/boot/vmlinux
Add the path to aarch64-linux-gnu-gdb gdb program Serial port speed Path to vmlinux
©2018 Zach Pfeffer

48. Connect to the target
target remote /dev/ttyUSB0
Type:
Remote debugging using /dev/ttyUSB0
arch_kgdb_breakpoint () at
/home/pfefferz/plprj4/mtd_board/build/tmp/work-
shared/plnx_aarch64/kernel-
source/arch/arm64/include/asm/kgdb.h:32
32 asm ("brk %0" : : "I"
(KGDB_COMPILED_DBG_BRK_IMM));
Output if it worked:
©2018 Zach Pfeffer

49. Debugging the Serial Link
Command Description
(gdb) set serial baud 115200 Set the baud rate in gdb
(gdb) set debug remote 1
Turn on debugging info
You’ll see a bunch of cryptic output
(gdb) show remoteflow Check if flow control is off
(gdb) show serial parity Check if parity is set
(gdb) set remotelogfile bringupserial.txt Log commands to a file
©2018 Zach Pfeffer

50. Sharing the Serial Link with agent-proxy
“agent-proxy acts as a proxy for the TARGET's serial port. It splits up the
serial port for multiplexing. One for primary console I/O, the other for
GDB session. Thus, we can work on both simultaneously. You should
run the agent-proxy on HOST machine.”
From post @ stackoverflow
©2018 Zach Pfeffer

51. Sharing the Serial Link with agent-proxy
git clone
http://git.kernel.org/pub/scm/utils/kernel/kgdb/
agent-proxy.git
cd agent-proxy ; make
./agent-proxy 5550^5551 0 /dev/ttyUSB0,115200
This will redirect:
TARGET's console to HOST:5550
TARGET's kgdb listening port to
HOST:5551©2018 Zach Pfeffer

52. Sharing the Serial Link with agent-proxy
In GDB:
target remote localhost:5551
Connect to the serial port:
telnet localhost 5550
zach@RemoteLab-01:~$ telnet localhost 5550
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
ZynqMP>
ZynqMP> boot
Type ^] then ‘quit’ to exit©2018 Zach Pfeffer

53. Sharing the Serial Link with agent-proxy
©2018 Zach Pfeffer

54. Sharing the Serial Link with agent-proxy
©2018 Zach Pfeffer

55. Tip: Fix up path to source for GDB
Use:
set substitute-path
/home/pfefferz/plprj4/mtd_board/build/tmp/
work-shared/plnx_aarch64/kernel-source
/home/zach/kernel-source
…to enable gdb to locate source
…if the path to the source changed.
Compiled directory
Where the source lives now
©2018 Zach Pfeffer

56. Adding kgdboc=ttyPS0,115200 to bootargs had no effect
Adding kgdbwait to bootargs _was_ required, but did not stop the
kernel to allow me to connect (its actual function)
I could not connect to the target from GDB via:
CTRL-A f g in Minicom or echo g > /proc/sysrq-trigger
These steps are listed in the Documentation of the kernel that
PetaLinux Tools 2017.4 ships with @ link but do not work:
Doesn’t Work for Me
©2018 Zach Pfeffer

57. Doesn’t Work for Me in GDB Session
step, Control-C (stepi and bt work)
©2018 Zach Pfeffer

58. Attempt to Load vmlinux Symbols
using the SDK and System
Debugger Tools Used
Get PetaLinux Tools 2017.4
Setup and Connect to kgdb
Attempt to Load vmlinux Symbols using the SDK and System
Debugger
Working Without Symbols
Call to Action for Xilinx
©2018 Zach Pfeffer

59. Click OK
1. Launch the SDK
2. Click OK
Start the SDK
©2018 Zach Pfeffer

60. 1
2
Click (1) and (2) to set up a Debug Configuration
Create a Debug Configuration
©2018 Zach Pfeffer

61. 2
1
Create a Debug Configuration
Select (1) click (2) to create a Debug Configuration
©2018 Zach Pfeffer

62. 1. Name it
2. Set to Att..
3. Click New
(1) Name it (2) click Attach to running target and (3)
click New
Create a Debug Configuration
©2018 Zach Pfeffer

63. 1. Name it
2. Click
3. Set Hardware Server4. localhost
5. Click
Create a New Target Connection
©2018 Zach Pfeffer

64. You should see…
Click OK
Create a New Target Connection
©2018 Zach Pfeffer

65. Click OK
Create a New Target Connection
©2018 Zach Pfeffer

66. Click Debug
Start Debugging Session
©2018 Zach Pfeffer

67. Click On Cortex-A53 #01
Click pause2
Halt the Target
©2018 Zach Pfeffer

68. No Symbols
Right click
©2018 Zach Pfeffer

69. Select Symbol Files…
©2018 Zach Pfeffer

70. Select Add…
©2018 Zach Pfeffer

71. pfefferz@plc2:~/plprj4/mtd_board$ find . -name "vmlinux"
./images/linux/vmlinux
./build/tmp/work/plnx_aarch64-xilinx-linux/linux-
xlnx/4.9-xilinx-v2017.4+gitAUTOINC+b450e900fd-r0/deploy-
linux-xlnx/vmlinux
./build/tmp/work/plnx_aarch64-xilinx-linux/linux-
xlnx/4.9-xilinx-v2017.4+gitAUTOINC+b450e900fd-r0/linux-
plnx_aarch64-standard-build/arch/arm64/boot/vmlinux
./build/tmp/work/plnx_aarch64-xilinx-linux/linux-
xlnx/4.9-xilinx-v2017.4+gitAUTOINC+b450e900fd-r0/linux-
plnx_aarch64-standard-build/vmlinux
./build/tmp/deploy/images/plnx_aarch64/vmlinux
pfefferz@plc2:~/plprj4/mtd_board$
Suggest using the one in arch/arm64/boot/vmlinux
since that gets produced directly from
bitbake virtual/kernel -C compile -c
compile
Find vmlinux in the build
©2018 Zach Pfeffer

72. 1. Use absolute path to
vmlinux
2. Click
3. Click
4. Click
5.
Click OK©2018 Zach Pfeffer

73. Click OK©2018 Zach Pfeffer

74. 1
Click
2. Arrrg no paths!
©2018 Zach Pfeffer

75. Working Without SymbolsTools Used
Get PetaLinux Tools 2017.4
Setup and Connect to kgdb
Attempt to Load vmlinux Symbols using the SDK and System
Debugger
Working Without Symbols
Call to Action for Xilinx
©2018 Zach Pfeffer

76. Working Without Symbols
• System Debugger: still good without symbols
• Tools available to help
©2018 Zach Pfeffer

77. Located in the PetaLinux Tools installation @
./tools/opt/pkg/petalinux/tools/linux-
i386/aarch64-linux-gnu/bin/
addr2line, readelf and objdump can help
©2018 Zach Pfeffer

78. ./tools/opt/pkg/petalinux/tools/linux-i386/aarch64-linux-gnu/bin/aarch64-linux-gnu-addr2line -e
/home/pfefferz/plprj4/mtd_board/build/tmp/work/plnx_aarch64-xilinx-linux/linux-xlnx/4.9-xilinx-
v2017.4+gitAUTOINC+b450e900fd-r0/linux-plnx_aarch64-standard-build/arch/arm64/boot/vmlinux 0xffffff80084a4a80
Example: map an address from the SDK to file and line numbers
/home/pfefferz/plprj4/mtd_board/build/tmp/work-
shared/plnx_aarch64/kernel-
source/arch/arm64/include/asm/atomic_ll_sc.h:292
addr2line
©2018 Zach Pfeffer

79. ./tools/opt/pkg/petalinux/tools/linux-i386/aarch64-linux-gnu/bin/aarch64-linux-gnu-objdump -d
/home/pfefferz/plprj4/mtd_board/build/tmp/work/plnx_aarch64-xilinx-linux/linux-xlnx/4.9-xilinx-
v2017.4+gitAUTOINC+b450e900fd-r0/linux-plnx_aarch64-standard-build/arch/arm64/boot/vmlinux | grep
"<start_kernel>"
Example: get the address of start_kernel
ffffff8008c307c8 <start_kernel>:
objdump
©2018 Zach Pfeffer

80. Hit start_kernel Breakpoint
4. Arrrg, no source!
1. Click to set a
breakpoint
2. Reset
Unit 3.
Breakpoint
hits
©2018 Zach Pfeffer

81. Example: get the kernel start address
./tools/opt/pkg/petalinux/tools/linux-i386/aarch64-linux-gnu/bin/aarch64-linux-gnu-readelf -S
/home/pfefferz/plprj4/mtd_board/build/tmp/work/plnx_aarch64-xilinx-linux/linux-xlnx/4.9-xilinx-
v2017.4+gitAUTOINC+b450e900fd-r0/linux-plnx_aarch64-standard-build/arch/arm64/boot/vmlinux
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .head.text PROGBITS ffffff8008080000 00010000
0000000000001000 0000000000000000 AX 0 0 4096
[ 2] .text PROGBITS ffffff8008081000 00011000
00000000008d45e0 0000000000000000 AX 0 0 2048
readelf
©2018 Zach Pfeffer

82. Call to Action for XilinxTools Used
Get PetaLinux Tools 2017.4
Setup and Connect to kgdb
Attempt to Load vmlinux Symbols using the SDK and System
Debugger
Working Without Symbols
Call to Action for Xilinx
©2018 Zach Pfeffer

83. Final Thoughts
• Yocto is _not_ a development tool
• For system engineers
• Xilinx SDK is for development
• Need all software to be built, source-level debugged and profiled via the SDK
©2018 Zach Pfeffer

84. Xilinx Call to Action for 2018.3
• The Xilinx SDK shall be able to easily build, load, source debug via the
System Debugger and benchmark the PMUFW
• The Xilinx SDK shall be able to easily build, load, source debug via the
System Debugger and benchmark ATF
• The Xilinx SDK shall be able to easily build, load, source debug via the
System Debugger and benchmark U-Boot
• The Xilinx SDK shall be able to easily build, load, source debug via the
System Debugger and benchmark the Linux Kernel
• The Xilinx SDK shall be able to easily build, load, source debug via the
System Debugger and benchmark the Xen Hypervisor
• It shall be easy in the SDK to perform the above mentioned items
concurrently with Linux application debug
©2018 Zach Pfeffer

85. Enjoyed this?
Looking for more help with PetaLinux Tools?
Check out https://www.zachpfeffer.com
©2018 Zach Pfeffer

86. References
• Info for presentation from:
Documentation/dev-tools/kgdb.rst
• Lightbulbs from
• https://www.iconspng.com/image/43308/light-bulb-on-off
©2018 Zach Pfeffer