Real time debugging: using non-intrusive tracepoints to debug live systems Marc Khouzam, Ericsson Canada [email_address]
Agenda <ul><li>DSF-GDB today </li></ul><ul><li>Tracepoints </li></ul><ul><ul><li>The need </li></ul></ul><ul><ul><li>The s...
DSF-GDB today <ul><li>Optimized GDB integration  </li></ul><ul><li>Standard Debugging features </li></ul><ul><li>Multi-thr...
DSF-GDB demo <ul><li>Any-binary debugging </li></ul><ul><ul><li>Debugging GDB itself </li></ul></ul>
<ul><li>TRACEPOINTS </li></ul>
The need <ul><li>Extremely low intrusiveness </li></ul><ul><ul><li>For live sites </li></ul></ul><ul><ul><li>For race cond...
The solution <ul><li>Highly efficient tracing tool using tracepoints </li></ul><ul><li>Dynamic Tracepoints </li></ul><ul><...
The solution <ul><li>GDB (GNU Debugger) </li></ul><ul><ul><li>Enhanced dynamic tracepoints </li></ul></ul><ul><ul><li>New ...
GDB’s New Tracepoint Feature <ul><li>Tracepoint support using gdbserver (on Linux) </li></ul><ul><ul><li>Tracing on the ho...
Dynamic Tracepoints (DSF-GDB Demo) <ul><li>Creation of tracepoint as is done as for breakpoints </li></ul><ul><li>Enable/D...
Dynamic Tracepoints <ul><li>Possible to define global actions (affecting all tracepoints) </li></ul><ul><li>Option to use ...
Two kinds of Dynamic Tracepoints <ul><li>Slow tracepoints using ptrace interface </li></ul><ul><ul><li>using breakpoints a...
Static Tracepoints provided by LTTng <ul><li>Using User-space LTTng library </li></ul><ul><li>Program to be traced is link...
Static Tracepoints and GDB <ul><li>Creation of tracepoint is done by designer before compilation </li></ul><ul><li>As for ...
Static Tracepoints and GDB <ul><li>Also like Dynamic Tracepoints: </li></ul><ul><li>Possible to define global actions (aff...
Trace Data Visualization (DSF-GDB Demo) <ul><li>Navigation through data records using GDB </li></ul><ul><li>Each data reco...
Next Tracepoint Features <ul><li>Disabling tracepoints during Tracing </li></ul><ul><li>Tracepoints Enhanced Visualization...
Future DSF-GDB work <ul><li>Multi-core awareness </li></ul><ul><ul><li>Reporting to the user which threads run on which co...
Future DSF-GDB work <ul><li>Bringing more GDB features to DSF-GDB </li></ul><ul><ul><li>Code patching (hot-swap?) </li></u...
Relevant Links <ul><li>CDT Tracepoint wiki </li></ul><ul><ul><li>http://wiki.eclipse.org/CDT/designs/Tracepoints </li></ul...
Questions?
Upcoming SlideShare
Loading in …5
×

Real time debugging: using non-intrusive tracepoints to debug live systems

5,451 views
5,302 views

Published on

Presentation given by Marc Khouzam, at EclipseCon 2010

In many cases, debugging a program by stopping its execution is not acceptable, delays introduced by a debugger might cause the program to change its behavior drastically, or perhaps fail, even when the code itself is correct. Troubleshooting a live system, chasing a race condition, debugging problems happening only under heavy load or on a multi-core system, or investigating user interface issues, are such examples. To address this, DSF-GDB, the new debugger integration of the CDT, is being extended to support the use of tracepoints which can dynamically be inserted and controlled in a live system.

In this talk we will cover such concepts as dynamic and static tracepoints, fast tracepoints, remote tracing, disconnected tracing, visualization of data and many more.

Beyond the topic of tracepoints, we will also discuss what advanced debugging features (e.g. multi-core awareness) are planned for the next release of the CDT and of GDB.

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,451
On SlideShare
0
From Embeds
0
Number of Embeds
76
Actions
Shares
0
Downloads
112
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Real time debugging: using non-intrusive tracepoints to debug live systems

  1. 1. Real time debugging: using non-intrusive tracepoints to debug live systems Marc Khouzam, Ericsson Canada [email_address]
  2. 2. Agenda <ul><li>DSF-GDB today </li></ul><ul><li>Tracepoints </li></ul><ul><ul><li>The need </li></ul></ul><ul><ul><li>The solution </li></ul></ul><ul><ul><ul><li>Dynamic Tracepoints and Eclipse demo </li></ul></ul></ul><ul><ul><ul><li>Static tracepoints </li></ul></ul></ul><ul><ul><ul><li>Trace data visualization and Eclipse demo </li></ul></ul></ul><ul><li>Questions </li></ul>
  3. 3. DSF-GDB today <ul><li>Optimized GDB integration </li></ul><ul><li>Standard Debugging features </li></ul><ul><li>Multi-thread/Multi-process </li></ul><ul><li>Non-stop debugging </li></ul><ul><li>Extendable Pretty-printing with GDB 7.0 </li></ul><ul><li>Linux, Windows, Mac </li></ul><ul><li>Reverse Debugging </li></ul><ul><li>Any-binary debugging (no need for a project) </li></ul>
  4. 4. DSF-GDB demo <ul><li>Any-binary debugging </li></ul><ul><ul><li>Debugging GDB itself </li></ul></ul>
  5. 5. <ul><li>TRACEPOINTS </li></ul>
  6. 6. The need <ul><li>Extremely low intrusiveness </li></ul><ul><ul><li>For live sites </li></ul></ul><ul><ul><li>For race conditions </li></ul></ul><ul><ul><li>For Real-time </li></ul></ul><ul><ul><li>For UI bugs </li></ul></ul>
  7. 7. The solution <ul><li>Highly efficient tracing tool using tracepoints </li></ul><ul><li>Dynamic Tracepoints </li></ul><ul><ul><li>Added dynamically while code is executing </li></ul></ul><ul><li>Static Tracepoints </li></ul><ul><ul><li>Added in the source code, before compilation </li></ul></ul><ul><li>Disconnected tracing </li></ul>
  8. 8. The solution <ul><li>GDB (GNU Debugger) </li></ul><ul><ul><li>Enhanced dynamic tracepoints </li></ul></ul><ul><ul><li>New control of static tracepoints </li></ul></ul><ul><li>LTTng and UST (new User Space Tracing) </li></ul><ul><ul><li>Can be controlled at run-time by GDB </li></ul></ul><ul><li>Eclipse CDT </li></ul><ul><ul><li>Extending the existing DSF-GDB integration </li></ul></ul>
  9. 9. GDB’s New Tracepoint Feature <ul><li>Tracepoint support using gdbserver (on Linux) </li></ul><ul><ul><li>Tracing on the host can still be done using gdbserver </li></ul></ul><ul><li>Tracepoints implemented by </li></ul><ul><ul><li>Breakpoints (slow dynamic tracepoints) </li></ul></ul><ul><ul><li>Jump-patching (fast dynamic tracepoints) </li></ul></ul><ul><ul><li>User-space LTTng (static tracepoints) </li></ul></ul><ul><li>Observer-mode to enforce tracing instead of debugging </li></ul>
  10. 10. Dynamic Tracepoints (DSF-GDB Demo) <ul><li>Creation of tracepoint as is done as for breakpoints </li></ul><ul><li>Enable/Disable tracepoints </li></ul><ul><li>Dynamic condition can be assigned to a tracepoint </li></ul><ul><li>Specification of data to be gathered using symbolic expressions and memory addresses (actions) </li></ul><ul><li>Pass count per tracepoint to stop tracing automatically </li></ul><ul><li>Trace-state variables that can be used in conditions and actions </li></ul><ul><li>Tracepoints are only in effect if tracing is enabled </li></ul>
  11. 11. Dynamic Tracepoints <ul><li>Possible to define global actions (affecting all tracepoints) </li></ul><ul><li>Option to use a finite trace buffer or circular trace buffer </li></ul><ul><li>Disconnected data gathering </li></ul><ul><li>On-disk trace data storage for ‘small’ amounts of data </li></ul><ul><li>Automatic timestamp collection on successful tracepoint hit </li></ul>
  12. 12. Two kinds of Dynamic Tracepoints <ul><li>Slow tracepoints using ptrace interface </li></ul><ul><ul><li>using breakpoints and automatic resuming </li></ul></ul><ul><li>Fast tracepoints using an in-process library </li></ul><ul><ul><li>using jump patches </li></ul></ul><ul><ul><li>restricted to 5 bytes instructions </li></ul></ul><ul><ul><li>will give error or use slow tracepoint if installation fails </li></ul></ul><ul><ul><li>if condition of tracepoint is false, the tracepoint will take < 100nS </li></ul></ul><ul><li>Eclipse support could be implicit or explicit </li></ul>
  13. 13. Static Tracepoints provided by LTTng <ul><li>Using User-space LTTng library </li></ul><ul><li>Program to be traced is linked with LTTng library </li></ul><ul><li>During tracing, user program calls LTTng library which calls GDB’s in-process library </li></ul><ul><li>Write data to the same in-process buffer as dynamic tracepoints </li></ul><ul><li>Can be listed by GDB </li></ul>
  14. 14. Static Tracepoints and GDB <ul><li>Creation of tracepoint is done by designer before compilation </li></ul><ul><li>As for Dynamic tracepoints: </li></ul><ul><li>Enable/Disable tracepoints dynamically </li></ul><ul><li>Dynamic condition can be assigned to a tracepoint </li></ul><ul><li>Can additionally have dynamic tracing specified using symbolic expressions and memory addresses (actions) </li></ul><ul><li>Pass count per tracepoint to stop tracing automatically </li></ul><ul><li>Trace-state variables that can be used in conditions and actions </li></ul>
  15. 15. Static Tracepoints and GDB <ul><li>Also like Dynamic Tracepoints: </li></ul><ul><li>Possible to define global actions (affecting all tracepoints) </li></ul><ul><li>Option to use a finite trace buffer or circular trace buffer </li></ul><ul><li>Disconnected data gathering </li></ul><ul><li>On-disk trace data storage for ‘small’ amounts of data </li></ul><ul><li>Automatic timestamp collection on successful tracepoint hit </li></ul>
  16. 16. Trace Data Visualization (DSF-GDB Demo) <ul><li>Navigation through data records using GDB </li></ul><ul><li>Each data record is a snapshot of debug information </li></ul><ul><li>Records can be examined using standard debugger views </li></ul><ul><ul><li>As if debugger was attached at a specific point in time </li></ul></ul><ul><ul><li>Only collected information can be shown </li></ul></ul><ul><ul><li>Highlighting of the tracepoint of interest </li></ul></ul><ul><li>All collected data of a record can also be dumped as plain text </li></ul><ul><li>Trace data can be saved to file </li></ul><ul><li>Saved trace data can be examined offline </li></ul>
  17. 17. Next Tracepoint Features <ul><li>Disabling tracepoints during Tracing </li></ul><ul><li>Tracepoints Enhanced Visualization: </li></ul><ul><ul><li>Currently the user must have an idea of what has been collected </li></ul></ul><ul><ul><li>Goal is to directly and only show what has been collected </li></ul></ul><ul><li>Fast Tracepoints on 3-byte instruction </li></ul><ul><ul><li>Currently fast tracepoints are 5-byte jumps insert in the code </li></ul></ul><ul><ul><li>New 3-byte jump to a nearby location to the 5-byte jump </li></ul></ul>
  18. 18. Future DSF-GDB work <ul><li>Multi-core awareness </li></ul><ul><ul><li>Reporting to the user which threads run on which cores </li></ul></ul><ul><li>Enhanced multi-process support </li></ul><ul><ul><li>Currently limited to single address-space targets </li></ul></ul><ul><ul><li>Will be extended to Linux </li></ul></ul><ul><li>Flexible-hierarchy breakpoint view usage </li></ul><ul><ul><li>Helios brings a new Flexible-hierarchy breakpoint view </li></ul></ul>
  19. 19. Future DSF-GDB work <ul><li>Bringing more GDB features to DSF-GDB </li></ul><ul><ul><li>Code patching (hot-swap?) </li></ul></ul><ul><ul><li>Checkpoints </li></ul></ul><ul><ul><li>Enhanced debugging console </li></ul></ul><ul><ul><li>Scripting </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>GDB is full of debugging feature, we just have to tap into that </li></ul></ul>
  20. 20. Relevant Links <ul><li>CDT Tracepoint wiki </li></ul><ul><ul><li>http://wiki.eclipse.org/CDT/designs/Tracepoints </li></ul></ul><ul><li>Features and screen shots </li></ul><ul><ul><li>http://www.eclipse.org/dsdp/dd/development/relnotes/dd_news-1.1.html </li></ul></ul><ul><ul><li>http:// wiki.eclipse.org/CDT/cdt -debug-default-integration   </li></ul></ul><ul><li>DSF-GDB feature-parity effort </li></ul><ul><ul><li>http:// wiki.eclipse.org/CDT/cdt -debug-feature-parity-effort </li></ul></ul><ul><li>Reverse Debugging Webinar </li></ul><ul><ul><li>http://live.eclipse.org/node/723 </li></ul></ul>
  21. 21. Questions?

×