Instrumentation Tools Demonstrations Questions




                         Linux Instrumentation

                       ...
Instrumentation Tools Demonstrations Questions

Outline


  1     Overview of available Instrumentation Tools
          Co...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Outline
  1     Overview of available Instru...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




What major players are in this area? DTra...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




What major players are in this area? DTra...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and users...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and users...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and users...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and users...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace Implementation


       Well document...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace Implementation


       Well document...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace Implementation


       Well document...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace on Linux?




       DTrace approach ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace on Linux?




       DTrace approach ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Enough DTrace envy, what about Linux?



...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                            ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Static probes? tracepoints




          ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Static probes? tracepoints




          ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Tracepoints




       Static probe points i...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel dynamic probes? kprobes




      ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel dynamic probes? kprobes




      ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Kprobes


       Insert a new probe into ker...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace dynamic probes? uprobes
Not mai...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace dynamic probes? uprobes
Not mai...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace dynamic probes? uprobes
Not mai...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Uprobes


      Kprobes for userspace
      ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace profiling? gprof




           ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace profiling? gprof




           ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

gprof



        Legacy, but deserves a ment...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

gprof Flat Profile




                      ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

gprof Call Graph




                       ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

kprof Profile Visualisation Tool




        ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace tracing? ptrace . . .
A bit ugl...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace tracing? ptrace . . .
A bit ugl...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace tracing? ptrace . . .
A bit ugl...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ptrace




       Means of one process obser...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ptrace—ltrace




                          ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ptrace—strace




                          ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Improved userspace tracing? utrace
Not ma...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Improved userspace tracing? utrace
Not ma...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Improved userspace tracing? utrace
Not ma...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

utrace


       Improved userspace tracing
 ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel tracing? ftrace




              ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel tracing? ftrace




              ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace
         Mainline
         Kernel tra...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace—Function Tracer




                 ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace—Task scheduling




                 ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace—Measuring task wakeup latency




   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel and Userspace tracing? LTTng
Not m...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel and Userspace tracing? LTTng
Not m...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel and Userspace tracing? LTTng
Not m...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

LTTng




      Kernel & Userspace tracing p...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

LTTng




      Kernel & Userspace tracing p...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

LTTng Screenshot
  LTTng did not work for me...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Hardware performance counters? oProfile


...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Hardware performance counters? oProfile


...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile




       System wide sampling profi...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile GUI




                            ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile Report




                         ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile Annotate




                       ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile Call Graph




                     ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something to pull everything together? Sy...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something to pull everything together? Sy...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something to pull everything together? Sy...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

SystemTap

      Powerful kernel & userspace...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

SystemTap
          Let’s face it, system ta...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something else to pull everything togethe...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something else to pull everything togethe...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something else to pull everything togethe...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf events

       All in one Performance t...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Sample perf report




                     ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Sample perf call graph




                 ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf—Available events




                  ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf—Available events




                  ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf top




                               ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf timechart




                         ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf timechart




                         ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




How does it all fit together?




        ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                            ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

   ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                            ...
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                            ...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Outline




  2     Demonstrations
 ...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes




Finding cache misses with perf


...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf



  ...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf



  ...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf




 ...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf




 ...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf
  $ p...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf
  $ p...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes




Locate sources of block I/O with ...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

perf—Locate sources of block I/O



...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

perf—Locate sources of block I/O
  #...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

perf—Locate sources of block I/O
  #...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes




Analyse a running kernel with per...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analyse a running kernel with perf a...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analysing a running kernel with perf...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analysing a running kernel with perf...
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analysing a running kernel with perf...
Instrumentation Tools Demonstrations Questions




Thankyou all for listening
     perf can be found in the Linux kernel s...
Upcoming SlideShare
Loading in...5
×

Linux Instrumentation

4,290

Published on

Numerous technologies exist for profiling and tracing live Linux systems - from the traditional and straight forward gProf and strace to the more elaborate SystemTap, oProfile and the Linux Trace Toolkit. Very recently some new technologies, perf events and ftrace, have appeared that can already largely take the place of these traditional tools and have gained mainline acceptance in the Linux community - meaning that they will become more and more relevant in the future and are already being used to shed light on real world performance issues.

This presentation provides an overview of a number of the more noteworthy instrumentation tools available for Linux and the technologies that they build upon. Some examples of using perf events to analyse a running system to help track down real world performance problems are demonstrated.

Published in: Technology, News & Politics
3 Comments
42 Likes
Statistics
Notes
No Downloads
Views
Total Views
4,290
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
370
Comments
3
Likes
42
Embeds 0
No embeds

No notes for slide

Linux Instrumentation

  1. 1. Instrumentation Tools Demonstrations Questions Linux Instrumentation Ian Munsie June 24, 2010 Ian Munsie Linux Instrumentation
  2. 2. Instrumentation Tools Demonstrations Questions Outline 1 Overview of available Instrumentation Tools Competition’s Instrumentation Tools Overview of Linux Instrumentation Tools Interactions Between Instrumentation Tools Interesting Instrumentation Tools 2 Demonstrations Finding cache misses with perf Locate sources of block I/O with tracepoints Using kprobes to analyse a running kernel Ian Munsie Linux Instrumentation
  3. 3. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Outline 1 Overview of available Instrumentation Tools Competition’s Instrumentation Tools DTrace Overview of Linux Instrumentation Tools tracepoints kprobes uprobes gprof ptrace utrace ftrace LTTng oProfile SystemTap performance events Interactions Between Instrumentation Tools Interesting Instrumentation Tools Ian Munsie Linux Instrumentation
  4. 4. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting What major players are in this area? DTrace Ian Munsie Linux Instrumentation
  5. 5. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting What major players are in this area? DTrace Ian Munsie Linux Instrumentation
  6. 6. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  7. 7. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  8. 8. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  9. 9. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  10. 10. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace Implementation Well documented probes Definition Probe: A location or action which can be hooked into in order to perform some arbitrary action Probes placed statically in source code Probe location well considered Probes in Solaris kernel, postgreSQL, x.org, . . . Can create new probes dynamically “Dynamic tracing framework” DTrace D programming language Ian Munsie Linux Instrumentation
  11. 11. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace Implementation Well documented probes Definition Probe: A location or action which can be hooked into in order to perform some arbitrary action Probes placed statically in source code Probe location well considered Probes in Solaris kernel, postgreSQL, x.org, . . . Can create new probes dynamically “Dynamic tracing framework” DTrace D programming language Ian Munsie Linux Instrumentation
  12. 12. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace Implementation Well documented probes Definition Probe: A location or action which can be hooked into in order to perform some arbitrary action Probes placed statically in source code Probe location well considered Probes in Solaris kernel, postgreSQL, x.org, . . . Can create new probes dynamically “Dynamic tracing framework” DTrace D programming language Ian Munsie Linux Instrumentation
  13. 13. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace on Linux? DTrace approach generally unsuitable for Linux kernel Linux kernel evolves rapidly Resistance to placement of static probes Unofficial Linux port does exist SystemTap can use DTrace markers Ian Munsie Linux Instrumentation
  14. 14. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace on Linux? DTrace approach generally unsuitable for Linux kernel Linux kernel evolves rapidly Resistance to placement of static probes Unofficial Linux port does exist SystemTap can use DTrace markers Ian Munsie Linux Instrumentation
  15. 15. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Enough DTrace envy, what about Linux? Ian Munsie Linux Instrumentation
  16. 16. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel oProfile tracepoints LTTng Hardware Counters uprobes perf events ftrace utrace kprobes ptrace Kernel SystemTap Modules perf gprof gdb strace DTrace Markers Userspace Ian Munsie Linux Instrumentation
  17. 17. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Static probes? tracepoints Ian Munsie Linux Instrumentation
  18. 18. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Static probes? tracepoints Ian Munsie Linux Instrumentation
  19. 19. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Tracepoints Static probe points in the kernel source Replace old Kernel Markers Infrastructure mainline Low performance impact TRACE_EVENT() macro Ian Munsie Linux Instrumentation
  20. 20. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel dynamic probes? kprobes Ian Munsie Linux Instrumentation
  21. 21. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel dynamic probes? kprobes Ian Munsie Linux Instrumentation
  22. 22. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kprobes Insert a new probe into kernel at runtime Does not do userspace probes Heavily architecture specific Can be used by loadable kernel modules Used by other instrumentation tools Provides three types of probes kprobes—Probe instruction jprobes—Probe function call kretprobes—Probe function return Mainline Ian Munsie Linux Instrumentation
  23. 23. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace dynamic probes? uprobes Not mainline yet. Ian Munsie Linux Instrumentation
  24. 24. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace dynamic probes? uprobes Not mainline yet. Ian Munsie Linux Instrumentation
  25. 25. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace dynamic probes? uprobes Not mainline yet. Ian Munsie Linux Instrumentation
  26. 26. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Uprobes Kprobes for userspace Not mainline yet—predict that may soon change Used by SystemTap Handlers run in task context Background page replacement mechanism Probed instruction single stepped in XOL area No longer depends on utrace Handlers implemented as kernel module perf interface Ian Munsie Linux Instrumentation
  27. 27. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace profiling? gprof Ian Munsie Linux Instrumentation
  28. 28. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace profiling? gprof Ian Munsie Linux Instrumentation
  29. 29. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting gprof Legacy, but deserves a mention Counts and profiles calls to functions Function granuality Compile executable with -pg flag Produce function call graph Does NOT profile libraries or kernel Kprof provides GUI to visualise call graph Ian Munsie Linux Instrumentation
  30. 30. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting gprof Flat Profile Ian Munsie Linux Instrumentation
  31. 31. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting gprof Call Graph Ian Munsie Linux Instrumentation
  32. 32. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting kprof Profile Visualisation Tool Ian Munsie Linux Instrumentation
  33. 33. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace tracing? ptrace . . . A bit ugly. Ian Munsie Linux Instrumentation
  34. 34. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace tracing? ptrace . . . A bit ugly. Ian Munsie Linux Instrumentation
  35. 35. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace tracing? ptrace . . . A bit ugly. Ian Munsie Linux Instrumentation
  36. 36. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ptrace Means of one process observing/controlling another Prominently used by gdb, strace, ltrace Ugly, limited interface Processes cannot be traced by multiple processes Signal oriented architecture Large overhead Ian Munsie Linux Instrumentation
  37. 37. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ptrace—ltrace Ian Munsie Linux Instrumentation
  38. 38. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ptrace—strace Ian Munsie Linux Instrumentation
  39. 39. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Improved userspace tracing? utrace Not mainline yet. Ian Munsie Linux Instrumentation
  40. 40. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Improved userspace tracing? utrace Not mainline yet. Ian Munsie Linux Instrumentation
  41. 41. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Improved userspace tracing? utrace Not mainline yet. Ian Munsie Linux Instrumentation
  42. 42. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting utrace Improved userspace tracing Not mainline Prominently used by SystemTap Utrace clients run in kernel, typically in module Ptrace reimplemented as a utrace client Infrastructure to monitor threads Establish “engine” for each monitored thread Event reporting Thread control Thread machine state access Ian Munsie Linux Instrumentation
  43. 43. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel tracing? ftrace Ian Munsie Linux Instrumentation
  44. 44. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel tracing? ftrace Ian Munsie Linux Instrumentation
  45. 45. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace Mainline Kernel tracing infrastructure Originated from the realtime efforts Hijacks mcount (from gprof) for own purposes Low performance impact Tracing Plugins Function Funtion graph Context switches Time interrupts disabled Time preemtion disabled Delay for high priority tasks Power state transitions Branch prediction ... Exposed through debugfs—Manipulate with echo & cat Gained event support—perf events favoured for this Ian Munsie Linux Instrumentation
  46. 46. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace—Function Tracer Ian Munsie Linux Instrumentation
  47. 47. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace—Task scheduling Ian Munsie Linux Instrumentation
  48. 48. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace—Measuring task wakeup latency Ian Munsie Linux Instrumentation
  49. 49. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel and Userspace tracing? LTTng Not mainline. Ian Munsie Linux Instrumentation
  50. 50. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel and Userspace tracing? LTTng Not mainline. Ian Munsie Linux Instrumentation
  51. 51. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel and Userspace tracing? LTTng Not mainline. Ian Munsie Linux Instrumentation
  52. 52. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting LTTng Kernel & Userspace tracing package Not mainline Adds instrumentation points into kernel Uses tracepoints, ftrace, kprobes Can plot traces Completely failed to work for me in every way Ian Munsie Linux Instrumentation
  53. 53. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting LTTng Kernel & Userspace tracing package Not mainline Adds instrumentation points into kernel Uses tracepoints, ftrace, kprobes Can plot traces Completely failed to work for me in every way Ian Munsie Linux Instrumentation
  54. 54. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting LTTng Screenshot LTTng did not work for me, but here’s what it’s supposed to do: Ian Munsie Linux Instrumentation
  55. 55. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Hardware performance counters? oProfile Ian Munsie Linux Instrumentation
  56. 56. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Hardware performance counters? oProfile Ian Munsie Linux Instrumentation
  57. 57. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile System wide sampling profiler Mainline Interface to Performance Measurement Unit GUI eases selection of performance counters Can generate gprof style call graph Ian Munsie Linux Instrumentation
  58. 58. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile GUI Ian Munsie Linux Instrumentation
  59. 59. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile Report Ian Munsie Linux Instrumentation
  60. 60. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile Annotate Ian Munsie Linux Instrumentation
  61. 61. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile Call Graph Ian Munsie Linux Instrumentation
  62. 62. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something to pull everything together? SystemTap Not mainline. Ian Munsie Linux Instrumentation
  63. 63. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something to pull everything together? SystemTap Not mainline. Ian Munsie Linux Instrumentation
  64. 64. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something to pull everything together? SystemTap Not mainline. Ian Munsie Linux Instrumentation
  65. 65. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting SystemTap Powerful kernel & userspace analysis suite Scripting language Generates, Compiles and loads kernel modules Primarily uses kprobes Utrace for userspace tracing Uprobes support Many userspace dependencies Hard to use More of interest to enterprise distros Version 1.2 recently released Prototypical support for perf events Hardware breakpoint probe support Not mainline and . . . Ian Munsie Linux Instrumentation
  66. 66. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting SystemTap Let’s face it, system tap isn’t going to be merged, so why even bring it up? Every kernel developer I have _ever_ seen agrees that all the new tracing is a million times superior. I’m sure there are system tap people who disagree, but quite frankly, I don’t see it being merged considering how little the system tap people ever did for the kernel. So if things like system tap and "security models that go behind the kernel by tying into utrace" are the reasons for utrace, color me utterly uninterested. In fact, color me actively hostile. I think that’s the worst possible situation that we’d ever be in as kernel people (namely exactly the "do things in kernel space by hiding behind utrace without having kernel people involved") Linus Ian Munsie Linux Instrumentation
  67. 67. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something else to pull everything together? perf events The future of Linux Instrumentation. Ian Munsie Linux Instrumentation
  68. 68. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something else to pull everything together? perf events The future of Linux Instrumentation. Ian Munsie Linux Instrumentation
  69. 69. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something else to pull everything together? perf events The future of Linux Instrumentation. Ian Munsie Linux Instrumentation
  70. 70. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf events All in one Performance tool Newcomer, but already mainline Interface to Performance Measurement Unit Virtual counters Integrates with tracepoints Kprobe support Scripting support SLAB subsystem profiling Lock profiling Scheduler analyser Timechart Ian Munsie Linux Instrumentation
  71. 71. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Sample perf report Ian Munsie Linux Instrumentation
  72. 72. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Sample perf call graph Ian Munsie Linux Instrumentation
  73. 73. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf—Available events Ian Munsie Linux Instrumentation
  74. 74. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf—Available events Ian Munsie Linux Instrumentation
  75. 75. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf top Ian Munsie Linux Instrumentation
  76. 76. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf timechart Ian Munsie Linux Instrumentation
  77. 77. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf timechart Ian Munsie Linux Instrumentation
  78. 78. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting How does it all fit together? Ian Munsie Linux Instrumentation
  79. 79. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel oProfile tracepoints LTTng Hardware Counters uprobes perf events ftrace utrace kprobes ptrace Kernel SystemTap Modules perf gprof gdb strace DTrace Markers Userspace Ian Munsie Linux Instrumentation
  80. 80. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace SystemTap LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  81. 81. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  82. 82. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  83. 83. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  84. 84. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  85. 85. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  86. 86. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  87. 87. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  88. 88. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  89. 89. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  90. 90. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  91. 91. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  92. 92. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  93. 93. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  94. 94. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  95. 95. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  96. 96. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  97. 97. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints—Mainline, Used by others ftrace perf events Ian Munsie Linux Instrumentation
  98. 98. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints—Mainline, Used by others ftrace—Mainline, Active perf events Ian Munsie Linux Instrumentation
  99. 99. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints—Mainline, Used by others ftrace—Mainline, Active perf events—Mainline, Very active Ian Munsie Linux Instrumentation
  100. 100. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel oProfile tracepoints LTTng Hardware Counters uprobes perf events ftrace utrace kprobes ptrace Kernel SystemTap Modules perf gprof gdb strace DTrace Markers Userspace Ian Munsie Linux Instrumentation
  101. 101. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel tracepoints Hardware Counters perf events ftrace kprobes perf Userspace Ian Munsie Linux Instrumentation
  102. 102. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Outline 2 Demonstrations Finding cache misses with perf Locate sources of block I/O with tracepoints Using kprobes to analyse a running kernel Ian Munsie Linux Instrumentation
  103. 103. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Ian Munsie Linux Instrumentation
  104. 104. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Fairly common scenario High cache misses is bad for performance Use a very simple C program as an example Task: sum every element in a large array Complication: array & program too big for L2 cache Two approaches with different array access order Use perf to observe cache misses Ian Munsie Linux Instrumentation
  105. 105. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Fairly common scenario High cache misses is bad for performance Use a very simple C program as an example Task: sum every element in a large array Complication: array & program too big for L2 cache Two approaches with different array access order Use perf to observe cache misses Ian Munsie Linux Instrumentation
  106. 106. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Ian Munsie Linux Instrumentation
  107. 107. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Ian Munsie Linux Instrumentation
  108. 108. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf $ perf record -e cache-misses ./cachetest $ perf report Ian Munsie Linux Instrumentation
  109. 109. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf $ perf record -e cache-misses ./cachetest $ perf report Ian Munsie Linux Instrumentation
  110. 110. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Locate sources of block I/O with perf and tracepoints Ian Munsie Linux Instrumentation
  111. 111. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes perf—Locate sources of block I/O System is under heavy disk I/O causing slowdowns Traditional top does not reveal offending processes Even atop only shows I/O at process granularity block:block_rq_issue tracepoint available Kernel must be compiled with block I/O tracing perf can produce call graphs to track exact origin Ian Munsie Linux Instrumentation
  112. 112. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes perf—Locate sources of block I/O # perf record -g -a -e block:block_rq_issue sleep 10 # perf report Ian Munsie Linux Instrumentation
  113. 113. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes perf—Locate sources of block I/O # perf record -g -a -e block:block_rq_issue sleep 10 # perf report Ian Munsie Linux Instrumentation
  114. 114. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analyse a running kernel with perf and kprobes Ian Munsie Linux Instrumentation
  115. 115. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analyse a running kernel with perf and kprobes Want to pull information out of a running kernel Information is not already revealed to userspace No tracepoints provide the information Live system—cannot afford downtime Kernel debugging information and source is available Let’s probe the scheduler as an example. . . Ian Munsie Linux Instrumentation
  116. 116. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analysing a running kernel with perf and kprobes # perf probe –line schedule Ian Munsie Linux Instrumentation
  117. 117. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analysing a running kernel with perf and kprobes Ian Munsie Linux Instrumentation
  118. 118. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analysing a running kernel with perf and kprobes # perf report Ian Munsie Linux Instrumentation
  119. 119. Instrumentation Tools Demonstrations Questions Thankyou all for listening perf can be found in the Linux kernel sources under /tools/perf perf is being actively developed in the “tip” kernel tree Linux Weekly News has some good further reading on ftrace—google site:lwn.net ftrace perf articles not as easy to find yet, here are some: http://lwn.net/Articles/339361—“Perfcounters added to the mainline” http://lwn.net/Articles/373842—“Scripting support for perf” http://lwn.net/Articles/382554—“A “live mode” for perf” http://lwn.net/Articles/346470—“Fun with tracepoints” http://lkml.org/lkml/2010/3/17/91—Ingo Molnar on database I/O latency Ian Munsie Linux Instrumentation
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×