debug generic linux process for performance.

1. DEBUG tools
Sep 2018
Vipin Varghese

2. • SIMD and intrinsic
• Cache and memory monitoring with Linux tools.
• Firmware version check and Firmware upgrade/downgrade.
• NUMA memory, PCIe and performance
• Memory monitoring
• Interrupts and system context switch cases.
• Memory maps.
• ethtool
• lspci
• Lshw
• lstopo
Linux utility

3. lstopo --pid 2 --fontsize 15 --gridsize 12 --no-
collapse

4. Network Interface
ethtool –S <ifname>
ethtool –i <ifname>
ethtool –p <ifname>
ethtool –m <ifname>
ethtool –k <ifname>
ethtool –P <ifname>
ethtool --phy-statistics <ifname>
ethtool –show-features
cat /proc/net/dev | grep <ifname>
cat /proc/interrupts | grep <ifname>

5. 1. Debug tools
2. Use of GDB and features
3. Use of LD_PRELOAD
4. Core file generate
5. Use of ltrace, strace, ptrace
6. Stack flow analysis.
7. GCC and build info
8. Perf stat |Vtunes usage
9. stack trace for all threads without GDB|PTRACE|PDUMP
Generic software debugging tools

6. GDB
call actual library functions or even functions from within the debugged program using the command call
start GDB with gdbtui or gdb -tui. Switch using 'layout src|asm|regs'
shell allows you to execute commands in the shell
print, examine and display
info file - Entry point
set disassembly-flavor intel
set print pretty
set print addr off
set print array
set print array on
set print array off
display next 5 instructions - x/5i $pc
disassemble <function name>
.gdbinit
file exe
break *0x400710
set disassembly-flavor intel
layout asm
layout regs
run argument1 argument2

7. GDB - Extra
we can use set so do the magic for us. Let's first inspect the instruction bytes:
(gdb) x/10b $pc
(gdb) set write
(gdb) set {unsigned int}$pc = 0x90909090
(gdb) set {unsigned char}($pc+4) = 0x90
(gdb) set write off
(gdb) x/10i $pc
x/6i $pc
=> 0x40911f: nop
0x409120: nop
0x409121: nop
0x409122: nop
0x409123: nop
0x409124: push rbp
set {unsigned int}0x40911f = 0x90909090
{unsigned char}0x409123 = 0x9
set $pc+=5
jump *$pc+5

8. LD_PRELOAD
set LD_PRELOAD to the path of a shared object, that file will be loaded before any other library (including the C runtime, libc.so).
To run with special library (example malloc) ‘LD_PRELOAD=/path/to/my/malloc.so /bin/ls’

9. generate core files
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 960
virtual memory (kbytes, -v) unlimited
ulimit -c unlimited
cat /proc/<process id>/limits
cat /proc/<process id>/sched

10. objdump
File header: -f
File format: -p
Section header: -h
All headers: -x
Executable sections: -d
Assembler sections: -D
Full contents: -s
Debug: -g
Symbol table: -t
Dynamic Symbol table: -T
Dynamic Relocation: -R
Function content via name: -s -j.rodata, -D --prefix-addresses
readelf --relocs

11. STRACE
strace -e trace=open,read <executable>
strace -t -e open <Executable>
strace -r -e open <exdcutable>
strace -c <executbale>
strace -i <executable>
strace -T -e read <executable>
strace -e trace=network|signal|memory <executable>
strace userspace utility for Linux helps to diagnose, debug and instructional by monitoring system calls and signal. The operation of
strace is made possible by the kernel feature known as ptrace.
Specifying a list of paths to be traced (-P /etc/ld.so.cache, for example).
Modifying return and error code of the specified syscalls, and inject signals upon their execution (since strace 4.15, -e inject=
option).
Extracting information about file descriptors (including sockets, -y option).

12. nm <executable>
t|T – The symbol is present in the .text code section
b|B – The symbol is in UN-initialized .data section
D|d – The symbol is in Initialized .data section.
nm -A ./*.o
nm -u undefined symbols
nm -n symbol
nm -S symbol wth size
nm -D dynamic symbol
A : Global absolute symbol.
a : Local absolute symbol.
B : Global bss symbol.
b : Local bss symbol.
D : Global data symbol.
d : Local data symbol.
f : Source file name symbol.
L : Global thread-local symbol (TLS).
l : Static thread-local symbol (TLS).
T : Global text symbol.
t : Local text symbol.
U : Undefined symbol.