Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Viller Hsiao <villerhsiao@gmail.com> 
Dec. 14, 2014 
Small Talk about Dynamically 
Trace Code
Rights to Copy 
copyright © 2014 Viller Hsiao 
Attribution – ShareAlike 3.0 
You are free 
● to copy, distribute, display,...
Who am I ? 
Viller Hsiao 
source url: http://goo.gl/n7NAfS 
Embeded Linux / RTOS engineer 
2007 ~ Present Realtek Corp. 
2...
An Embedded Programmer 
gcc 
Hardware 
Spec 
mmaakkeeffiillee menuconfig 
uclibc 
musl 
armcc 
newlib 
ld objdump 
kbuild ...
Topic Today 
How to trace C sources environment 
We have cross reference tools. 
ctags, cscope, LXR, source insight ... an...
Difficulties of Tracing C/C++ Sources 
● Preprocessor related 
● Too much #if/#elif/#endif 
● Too much level of macros 
● ...
Difficulties of Tracing C/C++ Sources (Cont.) 
● Runtime related 
● Some debugging and error handling stuff 
● Multi-task ...
Topic Today 
How to trace C sources environment 
Dynamically 
Trace code from runtime execution result 
(Mainly in linux k...
Tracing from real execution log 
Provides clues to trace code 
http://www.clker.com/cliparts/c/c/7/p/7/4/earth-hi.png 
htt...
Agenda Today 
● Build environment 
● Config system 
● Makefile / kbuild 
● Toolchain tips 
● Runtime trace 
● Qemu log 
● ...
menuconfig 
● Too much words in menuconfig 
● Hard to map definition to code world (CONFIG_XXX) 
12/14/14 11/38
menuconfig Tips – Search 
● Press '/' to search specific configurations 
12/14/14 12/38
menuconfig Tips – Search (Cont.) 
● Search result 
Path in menu 
Kconfig path 
Dependency 
12/14/14 13/38
menuconfig Tips – Result output 
● Linux/.config 
# 
# 
# # Automatically Automatically generated generated file; file; DO...
Tips of Tracing makefile 
● make V=1 
● make --dry-run 
12/14/14 15/38
Tips of Tracing makefile (Cont.) 
● make --debug 
12/14/14 16/38
Tips of Tracing Shell Script 
● #!/bin/sh “-x” 
#!/bin/sh -x 
FOO=something_produced_from_command 
if [ x${FOO} != x'' ]; ...
Tips of Tracing Macros/Definitions 
● gcc options: “-save-temps” or “-E” 
$ gcc -save-temps -o main main.c 
$ ls 
main mai...
Tips of Tracing Macros/Definitions (Cont.) 
12/14/14 19/38
Tips of Structure Definitions 
● Point to Real definition in the headers 
# 5 "include/linux/linkage.h" 2 
# 1 "include/li...
Kbuild Tips 
● Add compile flags for some object 
CFLAGS_CFLAGS_ffoooo..oo == --OO00 --gg --ssaavvee--tteemmpp 
● Remove c...
Qemu Tips 
● Qemu log (“-d op -D /path/to/qemu.log”) 
Setup the search rule 
12/14/14 22/38
Qemu Tips (Cont.) 
● Qemu log + regular expression 
Setup the search rule 
Example: get first user space entry point 
12/1...
Qemu Tips (Cont.) 
● Other tips 
● Qemu log + disassembly (objdump) 
● Qemu + gdb ( + qemu log) 
– Breakpoint / stack trac...
Ftrace – Overview 
● Kernel function tracer ( > 2.6.28 ?) 
● Record execution log of kernel function 
● Implemented by “-p...
Ftrace – Configs 
12/14/14 26/38
Ftrace – Steps 
[tracing]$ echo “*mtd*” > set_ftrace_filter 
[tracing]$ echo “*blk*” >> set_ftrace_filter 
[tracing]$ echo...
Ftrace – Available Tracer 
[tracing] $ cat available_tracers 
function 
function_graph 
nop 
[tracing] $ cat available_tra...
ftrace 
● function tracer 
12/14/14 29/38
ftrace 
● function tracer 
12/14/14 30/38
ftrace 
● function_graph tracer 
12/14/14 31/38
ftrace 
● function_graph tracer 
Task 
switch 
12/14/14 32/38
ftrace 
● Better front-end 
● trace-cmd 
● kernelshark 
12/14/14 33/38
Ftrace for Code Tracing 
● Ftrace provides clues of 
● Per-module view 
● Top-down view 
● Procedure excution flow 
● Task...
Ftrace – Other tracer 
● Trace event 
● ftrace + tracepoint 
● Probe event 
● ftrace + kprobe 
● Other tracer 
● Stack tra...
Conclusion 
● Trace code 
● Cross reference tool helps 
● Drill-down trace 
– Start from interface functions 
● Filesystem...
Reference 
[1] Building embedded linux demo case 
i. Sherif Mousa (Mar 26, 2014), “Building Embedded Linux” 
ii. Linaro wi...
THE END 
12/14/14 Viller Hsiao
Upcoming SlideShare
Loading in …5
×

Trace kernel code tips

3,130 views

Published on

Share some tips for tracing linux kernel source dynamically

Published in: Engineering

Trace kernel code tips

  1. 1. Viller Hsiao <villerhsiao@gmail.com> Dec. 14, 2014 Small Talk about Dynamically Trace Code
  2. 2. Rights to Copy copyright © 2014 Viller Hsiao Attribution – ShareAlike 3.0 You are free ● to copy, distribute, display, and perform the work ● to make derivative works ● to make commercial use of the work Under the following conditions Corrections, suggestions, contributions and translations are welcome! – Attribution. You must give the original author credit. – Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. For any reuse or distribution, you must make clear to others the license terms of this work. ● Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode 12/14/14 2/38
  3. 3. Who am I ? Viller Hsiao source url: http://goo.gl/n7NAfS Embeded Linux / RTOS engineer 2007 ~ Present Realtek Corp. 2003 ~ 2007 BenQ/Qisda 12/14/14 3/38
  4. 4. An Embedded Programmer gcc Hardware Spec mmaakkeeffiillee menuconfig uclibc musl armcc newlib ld objdump kbuild memory management Linker script glibc uboot source url: http://ext.pimg.tw/aabb52043/1367377582-3526768649_n.png 12/14/14 4/38
  5. 5. Topic Today How to trace C sources environment We have cross reference tools. ctags, cscope, LXR, source insight ... and so on BUT 12/14/14 5/38
  6. 6. Difficulties of Tracing C/C++ Sources ● Preprocessor related ● Too much #if/#elif/#endif ● Too much level of macros ● Configurable included files 12/14/14 6/38
  7. 7. Difficulties of Tracing C/C++ Sources (Cont.) ● Runtime related ● Some debugging and error handling stuff ● Multi-task environment ● Hard to realize HW state change Different task context 12/14/14 7/38
  8. 8. Topic Today How to trace C sources environment Dynamically Trace code from runtime execution result (Mainly in linux kernel source) 12/14/14 8/38
  9. 9. Tracing from real execution log Provides clues to trace code http://www.clker.com/cliparts/c/c/7/p/7/4/earth-hi.png http://goo.gl/4PnwFe 12/14/14 9/38
  10. 10. Agenda Today ● Build environment ● Config system ● Makefile / kbuild ● Toolchain tips ● Runtime trace ● Qemu log ● Linux ftrace 12/14/14 10/38
  11. 11. menuconfig ● Too much words in menuconfig ● Hard to map definition to code world (CONFIG_XXX) 12/14/14 11/38
  12. 12. menuconfig Tips – Search ● Press '/' to search specific configurations 12/14/14 12/38
  13. 13. menuconfig Tips – Search (Cont.) ● Search result Path in menu Kconfig path Dependency 12/14/14 13/38
  14. 14. menuconfig Tips – Result output ● Linux/.config # # # # Automatically Automatically generated generated file; file; DO DO NOT NOT EDIT. EDIT. # # Linux/Linux/arm arm 3.14.25 3.14.25 Kernel Kernel Configuration Configuration # # CONFIG_CONFIG_ARM=ARM=y y CONFIG_CONFIG_SYS_SYS_SUPPORTS_SUPPORTS_APM_APM_EMULATION=EMULATION=y y # # General setup # CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_CROSS_COMPILE="" # CONFIG_COMPILE_TEST is not set CONFIG_LOCALVERSION="" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_HAVE_KERNEL_GZIP=y CONFIG_HAVE_KERNEL_LZ4=y CONFIG_KERNEL_GZIP=y # CONFIG_KERNEL_LZMA is not set # CONFIG_KERNEL_XZ is not set # # General setup # CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_CROSS_COMPILE="" # CONFIG_COMPILE_TEST is not set CONFIG_LOCALVERSION="" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_HAVE_KERNEL_GZIP=y CONFIG_HAVE_KERNEL_LZ4=y CONFIG_KERNEL_GZIP=y # CONFIG_KERNEL_LZMA is not set # CONFIG_KERNEL_XZ is not set 12/14/14 14/38
  15. 15. Tips of Tracing makefile ● make V=1 ● make --dry-run 12/14/14 15/38
  16. 16. Tips of Tracing makefile (Cont.) ● make --debug 12/14/14 16/38
  17. 17. Tips of Tracing Shell Script ● #!/bin/sh “-x” #!/bin/sh -x FOO=something_produced_from_command if [ x${FOO} != x'' ]; then #!/bin/sh -x FOO=something_produced_from_command if [ x${FOO} != x'' ]; then echo "get foo" echo "get foo" else else echo "no foo" echo "no foo" fi fi ● Result + FOO=something_produced_from_command + '[' xsomething_produced_from_command '!=' x ']' + echo 'get foo' get foo + FOO=something_produced_from_command + '[' xsomething_produced_from_command '!=' x ']' + echo 'get foo' get foo 12/14/14 17/38
  18. 18. Tips of Tracing Macros/Definitions ● gcc options: “-save-temps” or “-E” $ gcc -save-temps -o main main.c $ ls main main.c main.i main.o main.s Makefile $ gcc -save-temps -o main main.c $ ls main main.c main.i main.o main.s Makefile ● Use astyle to beauty result $$ aassttyylele --AA88 mmaainin..ii 12/14/14 18/38
  19. 19. Tips of Tracing Macros/Definitions (Cont.) 12/14/14 19/38
  20. 20. Tips of Structure Definitions ● Point to Real definition in the headers # 5 "include/linux/linkage.h" 2 # 1 "include/linux/stringify.h" 1 # 6 "include/linux/linkage.h" 2 # 1 "include/linux/export.h" 1 # 26 "include/linux/export.h" struct kernel_symbol { unsigned long value; const char *name; }; # 5 "include/linux/linkage.h" 2 # 1 "include/linux/stringify.h" 1 # 6 "include/linux/linkage.h" 2 # 1 "include/linux/export.h" 1 # 26 "include/linux/export.h" struct kernel_symbol { unsigned long value; const char *name; }; 12/14/14 20/38
  21. 21. Kbuild Tips ● Add compile flags for some object CFLAGS_CFLAGS_ffoooo..oo == --OO00 --gg --ssaavvee--tteemmpp ● Remove compile flags for some object CCFFLLAAGGSS__RREEMMOOVVEE__ffoooo..oo == --OOss ● All subdirectory ssuubbddirir--ccccfflalaggss--yy == --ssaavvee--tteemmppss 12/14/14 21/38
  22. 22. Qemu Tips ● Qemu log (“-d op -D /path/to/qemu.log”) Setup the search rule 12/14/14 22/38
  23. 23. Qemu Tips (Cont.) ● Qemu log + regular expression Setup the search rule Example: get first user space entry point 12/14/14 23/38
  24. 24. Qemu Tips (Cont.) ● Other tips ● Qemu log + disassembly (objdump) ● Qemu + gdb ( + qemu log) – Breakpoint / stack trace – Qemu options ● -s or -gdb tcp::2345: enable gdb port ● -S: freeze CPU at startup 12/14/14 24/38
  25. 25. Ftrace – Overview ● Kernel function tracer ( > 2.6.28 ?) ● Record execution log of kernel function ● Implemented by “-pg” – Use dynamic ftrace to reduce overhead – Add “-g -O0” to export more information ● Accessing API – debugfs – /path/to/debug/tracing/ mount -t debugfs n mount -t debugfs noonnee //ssyyss//kkeerrnneel/l/ddeebbuugg 12/14/14 25/38
  26. 26. Ftrace – Configs 12/14/14 26/38
  27. 27. Ftrace – Steps [tracing]$ echo “*mtd*” > set_ftrace_filter [tracing]$ echo “*blk*” >> set_ftrace_filter [tracing]$ echo “*:mod:ext3” >> set_ftrace_filter [tracing]$ echo function > current_tracer [tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on; [tracing]$ cat trace [tracing]$ echo “*mtd*” > set_ftrace_filter [tracing]$ echo “*blk*” >> set_ftrace_filter [tracing]$ echo “*:mod:ext3” >> set_ftrace_filter [tracing]$ echo function > current_tracer [tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on; [tracing]$ cat trace 12/14/14 27/38
  28. 28. Ftrace – Available Tracer [tracing] $ cat available_tracers function function_graph nop [tracing] $ cat available_tracers function function_graph nop 12/14/14 28/38
  29. 29. ftrace ● function tracer 12/14/14 29/38
  30. 30. ftrace ● function tracer 12/14/14 30/38
  31. 31. ftrace ● function_graph tracer 12/14/14 31/38
  32. 32. ftrace ● function_graph tracer Task switch 12/14/14 32/38
  33. 33. ftrace ● Better front-end ● trace-cmd ● kernelshark 12/14/14 33/38
  34. 34. Ftrace for Code Tracing ● Ftrace provides clues of ● Per-module view ● Top-down view ● Procedure excution flow ● Tasks/Threads model relationship 12/14/14 34/38
  35. 35. Ftrace – Other tracer ● Trace event ● ftrace + tracepoint ● Probe event ● ftrace + kprobe ● Other tracer ● Stack tracer ● Irqoff latency tracer ● Syscall tracer ● And so forth … but not the topic today 12/14/14 35/38
  36. 36. Conclusion ● Trace code ● Cross reference tool helps ● Drill-down trace – Start from interface functions ● Filesystem hooks: open/read/write/ioctl ... ● probe/exit ● ISR entry ● Architecture interface – Global functions: EXPORT_SYMBOLS ● Execution log provide some hints, too. – Log from build flow – Qemu instruction trace log – Kernel ftrace 12/14/14 36/38
  37. 37. Reference [1] Building embedded linux demo case i. Sherif Mousa (Mar 26, 2014), “Building Embedded Linux” ii. Linaro wiki, “cross-NG” [2] Linux-3.14 documentation, i. “kbuild/makefile.txt” ii. “trace/ftrace.txt” [3] 池田宗廣,大岩尚宏,島本裕志,竹步晶雄,平松雅巳(Jan. 2014), “Linux kernel hacks” [4] man of i. astyle ii. qemu iii. Gnu make 12/14/14 37/38
  38. 38. THE END 12/14/14 Viller Hsiao

×