SlideShare a Scribd company logo

Kernel Debugging & Profiling

1 of 14
© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Kernel Debugging & Profiling
2© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
What to Expect?
Kernel Debugging Tools & Techniques
Kernel Profiling Tools & Mechanism
Kernel Testing Ways
3© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Debugging by Printing
printk & dmesg
int klogctl(int cmd, char *bufp, int len);
syslogd
#include <syslog.h>
void openlog(char *ident, int option, int facility);
void syslog ( int priority, char *format, …);
void closelog( void );
ident – typically short program name
option - LOG_CONS, LOG_PERROR, LOG_PID, …
facility - LOG_AUTHPRIV, LOG_CRON, LOG_DAEMON, LOG_USER, ...
priority - LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR,
LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG
klogd – bridge between kernel ring buffer and syslogd
4© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Debugging by Querying
Through Kernel Windows
/proc
Process related stuff
General system information: cpuinfo, meminfo, …
Drivers related info: devices, iomem, interrupts, ...
/sys (Utilities: sysfsutils, sysdiag)
Buses, Classes, Devices, ...
Using ioctl
Through specific drivers
Using custom commands
5© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Crash Dumping & Analysis
Oops Analysis
CONFIG_BUG, CONFIG_KALLSYMS
Call Trace
cat /proc/kallsyms
objdump -d | -S <obj | exe>
Crash Dumping (using 2 kernel images – could be same)
System Kernel: CONFIG_KEXEC, CONFIG_DEBUG_INFO
Dump-capture Kernel: CONFIG_CRASH_DUMP, CONFIG_PROC_VMCORE, Other
Architecture specific CONFIG_s
Boot System Kernel with crashkernel=size@offset, or crashkernel=range:size
Overlay Dump-capture Kernel using kexec
After Crash (Simulated using SysRq+c, Or real)
System reboots w/ Dump-capture Kernel
Collect the dump using kdump, Or simply cp /proc/vmcore <dump file>
Analyse using <gdb | crash> vmlinux <dump file>
6© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Kernel Hacking Related Options
CONFIG_DEBUG_INFO (gcc -g)
CONFIG_DEBUG_FS
CONFIG_DYNAMIC_DEBUG
CONFIG_DEBUG_STACKOVERFLOW
CONFIG_LOCKUP_DETECTOR
CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, ...
CONFIG_MAGIC_SYSRQ
CONFIG_EARLY_PRINTK
CONFIG_DEBUG_SLAB, CONFIG_DEBUG_VM, ...
CONFIG_DEBUG_STACK_USAGE
CONFIG_PANIC_ON_OOPS
CONFIG_PANIC_ON_OOPS_VALUE
Ad

Recommended

Arm device tree and linux device drivers
Arm device tree and linux device driversArm device tree and linux device drivers
Arm device tree and linux device driversHoucheng Lin
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debuggingHao-Ran Liu
 
Linux Kernel Module - For NLKB
Linux Kernel Module - For NLKBLinux Kernel Module - For NLKB
Linux Kernel Module - For NLKBshimosawa
 
U boot porting guide for SoC
U boot porting guide for SoCU boot porting guide for SoC
U boot porting guide for SoCMacpaul Lin
 
Profiling your Applications using the Linux Perf Tools
Profiling your Applications using the Linux Perf ToolsProfiling your Applications using the Linux Perf Tools
Profiling your Applications using the Linux Perf ToolsemBO_Conference
 
Understanding a kernel oops and a kernel panic
Understanding a kernel oops and a kernel panicUnderstanding a kernel oops and a kernel panic
Understanding a kernel oops and a kernel panicJoseph Lu
 

More Related Content

What's hot

Part 02 Linux Kernel Module Programming
Part 02 Linux Kernel Module ProgrammingPart 02 Linux Kernel Module Programming
Part 02 Linux Kernel Module ProgrammingTushar B Kute
 
Linux Internals - Kernel/Core
Linux Internals - Kernel/CoreLinux Internals - Kernel/Core
Linux Internals - Kernel/CoreShay Cohen
 
Jagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratchJagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratchlinuxlab_conf
 
linux device driver
linux device driverlinux device driver
linux device driverRahul Batra
 
LCA14: LCA14-306: CPUidle & CPUfreq integration with scheduler
LCA14: LCA14-306: CPUidle & CPUfreq integration with schedulerLCA14: LCA14-306: CPUidle & CPUfreq integration with scheduler
LCA14: LCA14-306: CPUidle & CPUfreq integration with schedulerLinaro
 
Debugging linux kernel tools and techniques
Debugging linux kernel tools and  techniquesDebugging linux kernel tools and  techniques
Debugging linux kernel tools and techniquesSatpal Parmar
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtAnne Nicolas
 
U-Boot presentation 2013
U-Boot presentation  2013U-Boot presentation  2013
U-Boot presentation 2013Wave Digitech
 
Introduction Linux Device Drivers
Introduction Linux Device DriversIntroduction Linux Device Drivers
Introduction Linux Device DriversNEEVEE Technologies
 
Uboot startup sequence
Uboot startup sequenceUboot startup sequence
Uboot startup sequenceHoucheng Lin
 
U Boot or Universal Bootloader
U Boot or Universal BootloaderU Boot or Universal Bootloader
U Boot or Universal BootloaderSatpal Parmar
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at NetflixBrendan Gregg
 
U-Boot Porting on New Hardware
U-Boot Porting on New HardwareU-Boot Porting on New Hardware
U-Boot Porting on New HardwareRuggedBoardGroup
 

What's hot (20)

Linux Porting
Linux PortingLinux Porting
Linux Porting
 
Toolchain
ToolchainToolchain
Toolchain
 
Linux device drivers
Linux device drivers Linux device drivers
Linux device drivers
 
Part 02 Linux Kernel Module Programming
Part 02 Linux Kernel Module ProgrammingPart 02 Linux Kernel Module Programming
Part 02 Linux Kernel Module Programming
 
Linux Internals - Kernel/Core
Linux Internals - Kernel/CoreLinux Internals - Kernel/Core
Linux Internals - Kernel/Core
 
Jagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratchJagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratch
 
linux device driver
linux device driverlinux device driver
linux device driver
 
Character drivers
Character driversCharacter drivers
Character drivers
 
LCA14: LCA14-306: CPUidle & CPUfreq integration with scheduler
LCA14: LCA14-306: CPUidle & CPUfreq integration with schedulerLCA14: LCA14-306: CPUidle & CPUfreq integration with scheduler
LCA14: LCA14-306: CPUidle & CPUfreq integration with scheduler
 
Introduction to Linux Drivers
Introduction to Linux DriversIntroduction to Linux Drivers
Introduction to Linux Drivers
 
USB Drivers
USB DriversUSB Drivers
USB Drivers
 
BeagleBone Black Bootloaders
BeagleBone Black BootloadersBeagleBone Black Bootloaders
BeagleBone Black Bootloaders
 
Debugging linux kernel tools and techniques
Debugging linux kernel tools and  techniquesDebugging linux kernel tools and  techniques
Debugging linux kernel tools and techniques
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
 
U-Boot presentation 2013
U-Boot presentation  2013U-Boot presentation  2013
U-Boot presentation 2013
 
Introduction Linux Device Drivers
Introduction Linux Device DriversIntroduction Linux Device Drivers
Introduction Linux Device Drivers
 
Uboot startup sequence
Uboot startup sequenceUboot startup sequence
Uboot startup sequence
 
U Boot or Universal Bootloader
U Boot or Universal BootloaderU Boot or Universal Bootloader
U Boot or Universal Bootloader
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
U-Boot Porting on New Hardware
U-Boot Porting on New HardwareU-Boot Porting on New Hardware
U-Boot Porting on New Hardware
 

Viewers also liked (15)

References
ReferencesReferences
References
 
gcc and friends
gcc and friendsgcc and friends
gcc and friends
 
Embedded C
Embedded CEmbedded C
Embedded C
 
Network Drivers
Network DriversNetwork Drivers
Network Drivers
 
File System Modules
File System ModulesFile System Modules
File System Modules
 
Serial Drivers
Serial DriversSerial Drivers
Serial Drivers
 
PCI Drivers
PCI DriversPCI Drivers
PCI Drivers
 
Block Drivers
Block DriversBlock Drivers
Block Drivers
 
Interrupts
InterruptsInterrupts
Interrupts
 
SPI Drivers
SPI DriversSPI Drivers
SPI Drivers
 
I2C Drivers
I2C DriversI2C Drivers
I2C Drivers
 
BeagleBoard-xM Bootloaders
BeagleBoard-xM BootloadersBeagleBoard-xM Bootloaders
BeagleBoard-xM Bootloaders
 
Platform Drivers
Platform DriversPlatform Drivers
Platform Drivers
 
Character Drivers
Character DriversCharacter Drivers
Character Drivers
 
File Systems
File SystemsFile Systems
File Systems
 

Similar to Kernel Debugging & Profiling

Android Boot Time Optimization
Android Boot Time OptimizationAndroid Boot Time Optimization
Android Boot Time OptimizationKan-Ru Chen
 
OSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner FischerOSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner FischerNETWAYS
 
HKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightHKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightLinaro
 
Linux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingLinux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingAnil Kumar Pugalia
 
Virtual platform
Virtual platformVirtual platform
Virtual platformsean chen
 
Embedded Android
Embedded AndroidEmbedded Android
Embedded Android晓东 杜
 
Mod06 new development tools
Mod06 new development toolsMod06 new development tools
Mod06 new development toolsPeter Haase
 
QEMU - Binary Translation
QEMU - Binary Translation QEMU - Binary Translation
QEMU - Binary Translation Jiann-Fuh Liaw
 
[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법
[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법
[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법GangSeok Lee
 
Mobile Hacking using Linux Drivers
Mobile Hacking using Linux DriversMobile Hacking using Linux Drivers
Mobile Hacking using Linux DriversAnil Kumar Pugalia
 
Study on Android Emulator
Study on Android EmulatorStudy on Android Emulator
Study on Android EmulatorSamael Wang
 
Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012
Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012
Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012DefCamp
 
Beyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisBeyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisFastly
 
Track c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eveTrack c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -evechiportal
 

Similar to Kernel Debugging & Profiling (20)

Kernel Debugging & Profiling
Kernel Debugging & ProfilingKernel Debugging & Profiling
Kernel Debugging & Profiling
 
Android Boot Time Optimization
Android Boot Time OptimizationAndroid Boot Time Optimization
Android Boot Time Optimization
 
BeagleBone Black Booting Process
BeagleBone Black Booting ProcessBeagleBone Black Booting Process
BeagleBone Black Booting Process
 
OSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner FischerOSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner Fischer
 
HKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightHKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with Coresight
 
Linux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingLinux User Space Debugging & Profiling
Linux User Space Debugging & Profiling
 
Virtual platform
Virtual platformVirtual platform
Virtual platform
 
Embedded Android
Embedded AndroidEmbedded Android
Embedded Android
 
Mod06 new development tools
Mod06 new development toolsMod06 new development tools
Mod06 new development tools
 
QEMU - Binary Translation
QEMU - Binary Translation QEMU - Binary Translation
QEMU - Binary Translation
 
[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법
[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법
[2009 CodeEngn Conference 03] koheung - 윈도우 커널 악성코드에 대한 분석 및 방법
 
Mobile Hacking using Linux Drivers
Mobile Hacking using Linux DriversMobile Hacking using Linux Drivers
Mobile Hacking using Linux Drivers
 
Study on Android Emulator
Study on Android EmulatorStudy on Android Emulator
Study on Android Emulator
 
Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012
Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012
Hunting and Exploiting Bugs in Kernel Drivers - DefCamp 2012
 
Genode Compositions
Genode CompositionsGenode Compositions
Genode Compositions
 
Beyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisBeyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic Analysis
 
Track c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eveTrack c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eve
 
There is more to C
There is more to CThere is more to C
There is more to C
 
Linux on System z debugging with Valgrind
Linux on System z debugging with ValgrindLinux on System z debugging with Valgrind
Linux on System z debugging with Valgrind
 
SR-IOV Introduce
SR-IOV IntroduceSR-IOV Introduce
SR-IOV Introduce
 

More from Anil Kumar Pugalia (20)

File System Modules
File System ModulesFile System Modules
File System Modules
 
Processes
ProcessesProcesses
Processes
 
System Calls
System CallsSystem Calls
System Calls
 
Introduction to Linux
Introduction to LinuxIntroduction to Linux
Introduction to Linux
 
Embedded Software Design
Embedded Software DesignEmbedded Software Design
Embedded Software Design
 
Playing with R L C Circuits
Playing with R L C CircuitsPlaying with R L C Circuits
Playing with R L C Circuits
 
Audio Drivers
Audio DriversAudio Drivers
Audio Drivers
 
Video Drivers
Video DriversVideo Drivers
Video Drivers
 
Shell Scripting
Shell ScriptingShell Scripting
Shell Scripting
 
Functional Programming with LISP
Functional Programming with LISPFunctional Programming with LISP
Functional Programming with LISP
 
Power of vi
Power of viPower of vi
Power of vi
 
"make" system
"make" system"make" system
"make" system
 
Hardware Design for Software Hackers
Hardware Design for Software HackersHardware Design for Software Hackers
Hardware Design for Software Hackers
 
RPM Building
RPM BuildingRPM Building
RPM Building
 
Linux Network Management
Linux Network ManagementLinux Network Management
Linux Network Management
 
System Calls
System CallsSystem Calls
System Calls
 
Timers
TimersTimers
Timers
 
Threads
ThreadsThreads
Threads
 
Synchronization
SynchronizationSynchronization
Synchronization
 
Processes
ProcessesProcesses
Processes
 

Recently uploaded

AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...ISPMAIndia
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut meManoj Prabakar B
 
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxLeveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxVotarikari Shravan
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxMaarten Balliauw
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolProduct School
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIEDanBrown980551
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementMimmo Squillace
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?MENGSAYLOEM1
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERNRonnelBaroc
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringMassimo Talia
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellencePrecisely
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceSusan Ibach
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsEvangelia Mitsopoulou
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
H3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxH3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxMemory Fabric Forum
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1Inbay UK
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, GoogleISPMAIndia
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsInflectra
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotelPhilippines
 

Recently uploaded (20)

AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut me
 
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxLeveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptx
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product School
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIE
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvement
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineering
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center Excellence
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data science
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applications
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
H3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxH3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptx
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company Profile
 

Kernel Debugging & Profiling

  • 1. © 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Kernel Debugging & Profiling
  • 2. 2© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. What to Expect? Kernel Debugging Tools & Techniques Kernel Profiling Tools & Mechanism Kernel Testing Ways
  • 3. 3© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Debugging by Printing printk & dmesg int klogctl(int cmd, char *bufp, int len); syslogd #include <syslog.h> void openlog(char *ident, int option, int facility); void syslog ( int priority, char *format, …); void closelog( void ); ident – typically short program name option - LOG_CONS, LOG_PERROR, LOG_PID, … facility - LOG_AUTHPRIV, LOG_CRON, LOG_DAEMON, LOG_USER, ... priority - LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR, LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG klogd – bridge between kernel ring buffer and syslogd
  • 4. 4© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Debugging by Querying Through Kernel Windows /proc Process related stuff General system information: cpuinfo, meminfo, … Drivers related info: devices, iomem, interrupts, ... /sys (Utilities: sysfsutils, sysdiag) Buses, Classes, Devices, ... Using ioctl Through specific drivers Using custom commands
  • 5. 5© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Crash Dumping & Analysis Oops Analysis CONFIG_BUG, CONFIG_KALLSYMS Call Trace cat /proc/kallsyms objdump -d | -S <obj | exe> Crash Dumping (using 2 kernel images – could be same) System Kernel: CONFIG_KEXEC, CONFIG_DEBUG_INFO Dump-capture Kernel: CONFIG_CRASH_DUMP, CONFIG_PROC_VMCORE, Other Architecture specific CONFIG_s Boot System Kernel with crashkernel=size@offset, or crashkernel=range:size Overlay Dump-capture Kernel using kexec After Crash (Simulated using SysRq+c, Or real) System reboots w/ Dump-capture Kernel Collect the dump using kdump, Or simply cp /proc/vmcore <dump file> Analyse using <gdb | crash> vmlinux <dump file>
  • 6. 6© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Kernel Hacking Related Options CONFIG_DEBUG_INFO (gcc -g) CONFIG_DEBUG_FS CONFIG_DYNAMIC_DEBUG CONFIG_DEBUG_STACKOVERFLOW CONFIG_LOCKUP_DETECTOR CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, ... CONFIG_MAGIC_SYSRQ CONFIG_EARLY_PRINTK CONFIG_DEBUG_SLAB, CONFIG_DEBUG_VM, ... CONFIG_DEBUG_STACK_USAGE CONFIG_PANIC_ON_OOPS CONFIG_PANIC_ON_OOPS_VALUE
  • 7. 7© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Kernel Debuggers Kernel Debugger (Frontend): kdb CONFIG_DEBUG_INFO, CONFIG_MAGIC_SYSRQ CONFIG_KGDB, CONFIG_KGDB_KDB, CONFIG_KGDB_SERIAL_CONSOLE Basic Operations over Serial Console Remote Debugging using kgdb (Kernel Debug Server) Remote connection & debug through gdb vmlinux gdb <kernel src>/vmlinux /proc/kcore Only to gather current kernel debug information
  • 8. 8© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Miscellaneous Tools & Techniques Early Init Debugs CONFIG_SERIAL_EARLYCON CONFIG_EARLY_PRINTK (direct on h/w) Light & Sound JTAG Hardware Protocol Debug Hardware Protocol Analyzers Network Debugging Driver level: ethtool [options] <dev i/f> Packet level: tcpdump [options, e.g. -vv] -i <dev i/f>
  • 9. 9© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Kernel Probes kprobes → CONFIG_KPROBES jprobes → Specialized Kprobes For probing function entry points kretprobes → Return Kprobes For probing function exit points
  • 10. 10© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Tracing Single Process Tracing using strace Traces the system calls made by an application Command: strace [options] <application> Excellent options to tune to get only desired output Linux Trace Toolkit Patch @ http://www.opersys.com/LTT Core Module Code using Trace Services Daemon: tracedaemon Utilities: tracereader, tracevisualizer Function Tracer CONFIG_FUNCTION_TRACER & co /sys/kernel/debug/tracing
  • 11. 11© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Profiling & Code Coverage System Profiling From /proc using LTTng Kernel Profiling: Oprofile CONFIG_PROFILING, CONFIG_OPROFILE, CONFIG_APIC Daemon: oprofiled Utilities: opcontrol, opreport, op_help, ... Kernel Performance Profiling using perf CONFIG_PERF_EVENTS & co stat, list, top, record, report, ... Kernel Code Coverage: CONFIG_GCOV_KERNEL, CONFIG_GCOV_PROFILE_ALL
  • 12. 12© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Testing Possibilities Linux Test Project (LTP) Hosted @ http://ltp.sourceforge.net Suite of around 3000 tests Exercises various parts of the Kernel Mostly automated, except net & storage ones Command: runltp -p -l logfile User Mode Linux (UML) Hosted @ http://user-mode-linux.sourceforge.net A Kernel Instance as a User Mode Process Kernel Debugging without “oops”ing Good for testing & experimenting with unstable kernels
  • 13. 13© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. What all have we learnt? Kernel Debugging Tools & Techniques Debugging by Printing & Querying Crash Dumping & Analysis Kernel Hacking Options Kernel Debuggers Miscellaneous Tools & Techniques Kernel Probes Tracing Kernel Profiling Tools & Mechanism Kernel Testing Ways
  • 14. 14© 2010-16 SysPlay Workshops <workshop@sysplay.in> All Rights Reserved. Any Queries?