SlideShare a Scribd company logo
1 of 119
Download to read offline
Instrumentation Tools Demonstrations Questions




                         Linux Instrumentation

                                       Ian Munsie


                                     June 24, 2010




                                    Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions

Outline


  1     Overview of available Instrumentation Tools
          Competition’s Instrumentation Tools
          Overview of Linux Instrumentation Tools
          Interactions Between Instrumentation Tools
          Interesting Instrumentation Tools

  2     Demonstrations
          Finding cache misses with perf
          Locate sources of block I/O with tracepoints
          Using kprobes to analyse a running kernel




                                          Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Outline
  1     Overview of available Instrumentation Tools
          Competition’s Instrumentation Tools
                DTrace
             Overview of Linux Instrumentation Tools
                tracepoints
                kprobes
                uprobes
                gprof
                ptrace
                utrace
                ftrace
                LTTng
                oProfile
                SystemTap
                performance events
             Interactions Between Instrumentation Tools
             Interesting Instrumentation Tools

                                          Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




What major players are in this area? DTrace




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




What major players are in this area? DTrace




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and userspace tracing
       “Better than Linux”
       Originated from Solaris
       Mainstream ports for FreeBSD, NetBSD, Mac OS X
       Licencing Issues with Linux
       No performance impact when probes disabled
       Minimal probe effect
  Definition
  Probe Effect: The phenomena where observing a system will
  change the behaviour of that system



                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and userspace tracing
       “Better than Linux”
       Originated from Solaris
       Mainstream ports for FreeBSD, NetBSD, Mac OS X
       Licencing Issues with Linux
       No performance impact when probes disabled
       Minimal probe effect
  Definition
  Probe Effect: The phenomena where observing a system will
  change the behaviour of that system



                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and userspace tracing
       “Better than Linux”
       Originated from Solaris
       Mainstream ports for FreeBSD, NetBSD, Mac OS X
       Licencing Issues with Linux
       No performance impact when probes disabled
       Minimal probe effect
  Definition
  Probe Effect: The phenomena where observing a system will
  change the behaviour of that system



                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace


       All purpose kernel and userspace tracing
       “Better than Linux”
       Originated from Solaris
       Mainstream ports for FreeBSD, NetBSD, Mac OS X
       Licencing Issues with Linux
       No performance impact when probes disabled
       Minimal probe effect
  Definition
  Probe Effect: The phenomena where observing a system will
  change the behaviour of that system



                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace Implementation


       Well documented probes
  Definition
  Probe: A location or action which can be hooked into in order to
  perform some arbitrary action

       Probes placed statically in source code
       Probe location well considered
       Probes in Solaris kernel, postgreSQL, x.org, . . .
       Can create new probes dynamically
       “Dynamic tracing framework”
       DTrace D programming language


                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace Implementation


       Well documented probes
  Definition
  Probe: A location or action which can be hooked into in order to
  perform some arbitrary action

       Probes placed statically in source code
       Probe location well considered
       Probes in Solaris kernel, postgreSQL, x.org, . . .
       Can create new probes dynamically
       “Dynamic tracing framework”
       DTrace D programming language


                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace Implementation


       Well documented probes
  Definition
  Probe: A location or action which can be hooked into in order to
  perform some arbitrary action

       Probes placed statically in source code
       Probe location well considered
       Probes in Solaris kernel, postgreSQL, x.org, . . .
       Can create new probes dynamically
       “Dynamic tracing framework”
       DTrace D programming language


                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace on Linux?




       DTrace approach generally unsuitable for Linux kernel
       Linux kernel evolves rapidly
       Resistance to placement of static probes
       Unofficial Linux port does exist
       SystemTap can use DTrace markers




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

DTrace on Linux?




       DTrace approach generally unsuitable for Linux kernel
       Linux kernel evolves rapidly
       Resistance to placement of static probes
       Unofficial Linux port does exist
       SystemTap can use DTrace markers




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Enough DTrace envy, what about Linux?




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                                               Non-Mainline Kernel
         oProfile

                                    tracepoints                          LTTng
Hardware
Counters
                                                                     uprobes
       perf events
                                   ftrace

                                                                                  utrace

                             kprobes


                                  ptrace
                                                      Kernel                   SystemTap
                                                     Modules
 perf    gprof                  gdb       strace            DTrace Markers
Userspace

                                    Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Static probes? tracepoints




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Static probes? tracepoints




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Tracepoints




       Static probe points in the kernel source
       Replace old Kernel Markers
       Infrastructure mainline
       Low performance impact
       TRACE_EVENT() macro




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel dynamic probes? kprobes




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel dynamic probes? kprobes




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Kprobes


       Insert a new probe into kernel at runtime
       Does not do userspace probes
       Heavily architecture specific
       Can be used by loadable kernel modules
       Used by other instrumentation tools
       Provides three types of probes
               kprobes—Probe instruction
               jprobes—Probe function call
               kretprobes—Probe function return
       Mainline



                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace dynamic probes? uprobes
Not mainline yet.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace dynamic probes? uprobes
Not mainline yet.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace dynamic probes? uprobes
Not mainline yet.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Uprobes


      Kprobes for userspace
      Not mainline yet—predict that may soon change
      Used by SystemTap
      Handlers run in task context
      Background page replacement mechanism
      Probed instruction single stepped in XOL area
      No longer depends on utrace
      Handlers implemented as kernel module
      perf interface




                                      Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace profiling? gprof




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace profiling? gprof




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

gprof



        Legacy, but deserves a mention
        Counts and profiles calls to functions
        Function granuality
        Compile executable with -pg flag
        Produce function call graph
        Does NOT profile libraries or kernel
        Kprof provides GUI to visualise call graph




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

gprof Flat Profile




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

gprof Call Graph




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

kprof Profile Visualisation Tool




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace tracing? ptrace . . .
A bit ugly.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace tracing? ptrace . . .
A bit ugly.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Userspace tracing? ptrace . . .
A bit ugly.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ptrace




       Means of one process observing/controlling another
       Prominently used by gdb, strace, ltrace
       Ugly, limited interface
       Processes cannot be traced by multiple processes
       Signal oriented architecture
       Large overhead




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ptrace—ltrace




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ptrace—strace




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Improved userspace tracing? utrace
Not mainline yet.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Improved userspace tracing? utrace
Not mainline yet.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Improved userspace tracing? utrace
Not mainline yet.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

utrace


       Improved userspace tracing
       Not mainline
       Prominently used by SystemTap
       Utrace clients run in kernel, typically in module
       Ptrace reimplemented as a utrace client
       Infrastructure to monitor threads
       Establish “engine” for each monitored thread
               Event reporting
               Thread control
               Thread machine state access




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel tracing? ftrace




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel tracing? ftrace




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace
         Mainline
         Kernel tracing infrastructure
         Originated from the realtime efforts
         Hijacks mcount (from gprof) for own purposes
         Low performance impact
         Tracing Plugins
               Function
               Funtion graph
               Context switches
               Time interrupts disabled
               Time preemtion disabled
               Delay for high priority tasks
               Power state transitions
               Branch prediction
               ...
         Exposed through debugfs—Manipulate with echo & cat
         Gained event support—perf events favoured for this
                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace—Function Tracer




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace—Task scheduling




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

ftrace—Measuring task wakeup latency




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel and Userspace tracing? LTTng
Not mainline.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel and Userspace tracing? LTTng
Not mainline.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Kernel and Userspace tracing? LTTng
Not mainline.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

LTTng




      Kernel & Userspace tracing package
      Not mainline
      Adds instrumentation points into kernel
      Uses tracepoints, ftrace, kprobes
      Can plot traces
      Completely failed to work for me in every way




                                      Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

LTTng




      Kernel & Userspace tracing package
      Not mainline
      Adds instrumentation points into kernel
      Uses tracepoints, ftrace, kprobes
      Can plot traces
      Completely failed to work for me in every way




                                      Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

LTTng Screenshot
  LTTng did not work for me, but here’s what it’s supposed to do:




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Hardware performance counters? oProfile




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Hardware performance counters? oProfile




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile




       System wide sampling profiler
       Mainline
       Interface to Performance Measurement Unit
       GUI eases selection of performance counters
       Can generate gprof style call graph




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile GUI




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile Report




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile Annotate




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

oProfile Call Graph




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something to pull everything together? SystemTap
Not mainline.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something to pull everything together? SystemTap
Not mainline.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something to pull everything together? SystemTap
Not mainline.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

SystemTap

      Powerful kernel & userspace analysis suite
      Scripting language
      Generates, Compiles and loads kernel modules
      Primarily uses kprobes
      Utrace for userspace tracing
      Uprobes support
      Many userspace dependencies
      Hard to use
      More of interest to enterprise distros
      Version 1.2 recently released
              Prototypical support for perf events
              Hardware breakpoint probe support
      Not mainline and . . .
                                      Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

SystemTap
          Let’s face it, system tap isn’t going to be merged,
      so why even bring it up? Every kernel developer I
      have _ever_ seen agrees that all the new tracing is a
      million times superior. I’m sure there are system tap
      people who disagree, but quite frankly, I don’t see it
      being merged considering how little the system tap
      people ever did for the kernel.
          So if things like system tap and "security models
      that go behind the kernel by tying into utrace" are the
      reasons for utrace, color me utterly uninterested. In
      fact, color me actively hostile. I think that’s the worst
      possible situation that we’d ever be in as kernel people
      (namely exactly the "do things in kernel space by
      hiding behind utrace without having kernel people
      involved")
          Linus
                                      Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something else to pull everything together? perf events
The future of Linux Instrumentation.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something else to pull everything together? perf events
The future of Linux Instrumentation.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




Something else to pull everything together? perf events
The future of Linux Instrumentation.




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf events

       All in one Performance tool
       Newcomer, but already mainline
       Interface to Performance Measurement Unit
       Virtual counters
       Integrates with tracepoints
       Kprobe support
       Scripting support
       SLAB subsystem profiling
       Lock profiling
       Scheduler analyser
       Timechart


                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Sample perf report




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Sample perf call graph




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf—Available events




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf—Available events




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf top




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf timechart




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

perf timechart




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting




How does it all fit together?




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                                               Non-Mainline Kernel
         oProfile

                                    tracepoints                          LTTng
Hardware
Counters
                                                                     uprobes
       perf events
                                   ftrace

                                                                                  utrace

                             kprobes


                                  ptrace
                                                      Kernel                   SystemTap
                                                     Modules
 perf    gprof                  gdb       strace            DTrace Markers
Userspace

                                    Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace
       SystemTap
       LTTng
       gprof
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap
       LTTng
       gprof
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap
       LTTng
       gprof
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng
       gprof
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng
       gprof
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile—Not going anywhere
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile—Not going anywhere
       uprobes
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile—Not going anywhere
       uprobes—Not mainline yet, likely soon. . .
       kprobes
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile—Not going anywhere
       uprobes—Not mainline yet, likely soon. . .
       kprobes—Mainline, Used by others
       tracepoints
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile—Not going anywhere
       uprobes—Not mainline yet, likely soon. . .
       kprobes—Mainline, Used by others
       tracepoints—Mainline, Used by others
       ftrace
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile—Not going anywhere
       uprobes—Not mainline yet, likely soon. . .
       kprobes—Mainline, Used by others
       tracepoints—Mainline, Used by others
       ftrace—Mainline, Active
       perf events

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting

Interesting Technologies for the Future

       DTrace—Licence issues, not suited to Linux
       SystemTap—Not mainline, enterprise focus
       LTTng—Not mainline, enterprise focus
       gprof—Limitations, not going anywhere
       ptrace—Ugly, set in stone
       utrace—Trouble going mainline
       oProfile—Not going anywhere
       uprobes—Not mainline yet, likely soon. . .
       kprobes—Mainline, Used by others
       tracepoints—Mainline, Used by others
       ftrace—Mainline, Active
       perf events—Mainline, Very active

                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                                               Non-Mainline Kernel
         oProfile

                                    tracepoints                          LTTng
Hardware
Counters
                                                                     uprobes
       perf events
                                   ftrace

                                                                                  utrace

                             kprobes


                                  ptrace
                                                      Kernel                   SystemTap
                                                     Modules
 perf    gprof                  gdb       strace            DTrace Markers
Userspace

                                    Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Non-Linux Linux Interesting


Mainline Kernel                                               Non-Mainline Kernel



                                    tracepoints
Hardware
Counters


       perf events
                                   ftrace



                             kprobes




  perf
Userspace

                                    Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Outline




  2     Demonstrations
          Finding cache misses with perf
          Locate sources of block I/O with tracepoints
          Using kprobes to analyse a running kernel




                                          Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes




Finding cache misses with perf




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf



       Fairly common scenario
       High cache misses is bad for performance
       Use a very simple C program as an example
       Task: sum every element in a large array
       Complication: array & program too big for L2 cache
       Two approaches with different array access order
       Use perf to observe cache misses




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf



       Fairly common scenario
       High cache misses is bad for performance
       Use a very simple C program as an example
       Task: sum every element in a large array
       Complication: array & program too big for L2 cache
       Two approaches with different array access order
       Use perf to observe cache misses




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf
  $ perf record -e cache-misses ./cachetest
  $ perf report




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Finding cache misses with perf
  $ perf record -e cache-misses ./cachetest
  $ perf report




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes




Locate sources of block I/O with perf and tracepoints




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

perf—Locate sources of block I/O




       System is under heavy disk I/O causing slowdowns
       Traditional top does not reveal offending processes
       Even atop only shows I/O at process granularity
       block:block_rq_issue tracepoint available
       Kernel must be compiled with block I/O tracing
       perf can produce call graphs to track exact origin




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

perf—Locate sources of block I/O
  # perf record -g -a -e block:block_rq_issue sleep 10
  # perf report




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

perf—Locate sources of block I/O
  # perf record -g -a -e block:block_rq_issue sleep 10
  # perf report




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes




Analyse a running kernel with perf and kprobes




                                     Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analyse a running kernel with perf and kprobes




       Want to pull information out of a running kernel
       Information is not already revealed to userspace
       No tracepoints provide the information
       Live system—cannot afford downtime
       Kernel debugging information and source is available
       Let’s probe the scheduler as an example. . .




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analysing a running kernel with perf and kprobes
  # perf probe –line schedule




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analysing a running kernel with perf and kprobes




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions   Cache Misses Block I/O Perf Kprobes

Analysing a running kernel with perf and kprobes
  # perf report




                                       Ian Munsie   Linux Instrumentation
Instrumentation Tools Demonstrations Questions




Thankyou all for listening
     perf can be found in the Linux kernel sources under
     /tools/perf
     perf is being actively developed in the “tip” kernel tree
     Linux Weekly News has some good further reading on
     ftrace—google site:lwn.net ftrace
     perf articles not as easy to find yet, here are some:
             http://lwn.net/Articles/339361—“Perfcounters added to the mainline”
             http://lwn.net/Articles/373842—“Scripting support for perf”
             http://lwn.net/Articles/382554—“A “live mode” for perf”
             http://lwn.net/Articles/346470—“Fun with tracepoints”
             http://lkml.org/lkml/2010/3/17/91—Ingo Molnar on database I/O latency




                                        Ian Munsie        Linux Instrumentation

More Related Content

What's hot

Linux Presentation
Linux PresentationLinux Presentation
Linux Presentationnishantsri
 
Linux Kernel - Virtual File System
Linux Kernel - Virtual File SystemLinux Kernel - Virtual File System
Linux Kernel - Virtual File SystemAdrian Huang
 
linux device driver
linux device driverlinux device driver
linux device driverRahul Batra
 
Linux Performance Analysis and Tools
Linux Performance Analysis and ToolsLinux Performance Analysis and Tools
Linux Performance Analysis and ToolsBrendan Gregg
 
Debugging linux kernel tools and techniques
Debugging linux kernel tools and  techniquesDebugging linux kernel tools and  techniques
Debugging linux kernel tools and techniquesSatpal Parmar
 
Linux presentation
Linux presentationLinux presentation
Linux presentationNikhil Jain
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveMichal Rostecki
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance AnalysisBrendan Gregg
 
The TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux KernelThe TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux KernelDivye Kapoor
 
Linux basic commands
Linux basic commandsLinux basic commands
Linux basic commandsSagar Kumar
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFBrendan Gregg
 
Unix shell scripting basics
Unix shell scripting basicsUnix shell scripting basics
Unix shell scripting basicsManav Prasad
 
Linux kernel tracing
Linux kernel tracingLinux kernel tracing
Linux kernel tracingViller Hsiao
 
Meet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracingMeet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracingViller Hsiao
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...Adrian Huang
 

What's hot (20)

Linux Presentation
Linux PresentationLinux Presentation
Linux Presentation
 
Linux Kernel - Virtual File System
Linux Kernel - Virtual File SystemLinux Kernel - Virtual File System
Linux Kernel - Virtual File System
 
linux device driver
linux device driverlinux device driver
linux device driver
 
Linux Performance Analysis and Tools
Linux Performance Analysis and ToolsLinux Performance Analysis and Tools
Linux Performance Analysis and Tools
 
Debugging linux kernel tools and techniques
Debugging linux kernel tools and  techniquesDebugging linux kernel tools and  techniques
Debugging linux kernel tools and techniques
 
Linux presentation
Linux presentationLinux presentation
Linux presentation
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep Dive
 
Basic Linux Internals
Basic Linux InternalsBasic Linux Internals
Basic Linux Internals
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
 
The TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux KernelThe TCP/IP Stack in the Linux Kernel
The TCP/IP Stack in the Linux Kernel
 
Linux basic commands
Linux basic commandsLinux basic commands
Linux basic commands
 
Qemu JIT Code Generator and System Emulation
Qemu JIT Code Generator and System EmulationQemu JIT Code Generator and System Emulation
Qemu JIT Code Generator and System Emulation
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
 
Linux - Introductions to Linux Operating System
Linux - Introductions to Linux Operating SystemLinux - Introductions to Linux Operating System
Linux - Introductions to Linux Operating System
 
Unix shell scripting basics
Unix shell scripting basicsUnix shell scripting basics
Unix shell scripting basics
 
Linux kernel tracing
Linux kernel tracingLinux kernel tracing
Linux kernel tracing
 
Meet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracingMeet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracing
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
 
Linux Internals - Part II
Linux Internals - Part IILinux Internals - Part II
Linux Internals - Part II
 
Linux introduction
Linux introductionLinux introduction
Linux introduction
 

Viewers also liked

Linux Systems Performance 2016
Linux Systems Performance 2016Linux Systems Performance 2016
Linux Systems Performance 2016Brendan Gregg
 
Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016Brendan Gregg
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and moreBrendan Gregg
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at NetflixBrendan Gregg
 
Velocity 2015 linux perf tools
Velocity 2015 linux perf toolsVelocity 2015 linux perf tools
Velocity 2015 linux perf toolsBrendan Gregg
 
Walking the Tightrope: Balancing Bias to Action and Planning
Walking the Tightrope: Balancing Bias to Action and PlanningWalking the Tightrope: Balancing Bias to Action and Planning
Walking the Tightrope: Balancing Bias to Action and PlanningDianne Marsh
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing LandscapeSasha Goldshtein
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceSUSE Labs Taipei
 
Usp友の会勉強会、ジャクソン構造図の巻(後編)
Usp友の会勉強会、ジャクソン構造図の巻(後編)Usp友の会勉強会、ジャクソン構造図の巻(後編)
Usp友の会勉強会、ジャクソン構造図の巻(後編)umidori
 
Bashのヒストリ展開を活用する
Bashのヒストリ展開を活用するBashのヒストリ展開を活用する
Bashのヒストリ展開を活用するbsdhack
 
FreeBSDのブートプロセス
FreeBSDのブートプロセスFreeBSDのブートプロセス
FreeBSDのブートプロセスbsdhack
 
2015.08.29 JUS共催勉強会資料
2015.08.29 JUS共催勉強会資料2015.08.29 JUS共催勉強会資料
2015.08.29 JUS共催勉強会資料umidori
 
Usp友の会勉強会、ジャクソン構造図の巻(前編)
Usp友の会勉強会、ジャクソン構造図の巻(前編)Usp友の会勉強会、ジャクソン構造図の巻(前編)
Usp友の会勉強会、ジャクソン構造図の巻(前編)umidori
 
シェルスクリプトを極める
シェルスクリプトを極めるシェルスクリプトを極める
シェルスクリプトを極めるbsdhack
 
Linux女子部第二回勉強会usp友の会
Linux女子部第二回勉強会usp友の会Linux女子部第二回勉強会usp友の会
Linux女子部第二回勉強会usp友の会Ryuichi Ueda
 
正規表現
正規表現正規表現
正規表現bsdhack
 

Viewers also liked (20)

Linux Systems Performance 2016
Linux Systems Performance 2016Linux Systems Performance 2016
Linux Systems Performance 2016
 
Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and more
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
Velocity 2015 linux perf tools
Velocity 2015 linux perf toolsVelocity 2015 linux perf tools
Velocity 2015 linux perf tools
 
Linux PV on HVM
Linux PV on HVMLinux PV on HVM
Linux PV on HVM
 
Walking the Tightrope: Balancing Bias to Action and Planning
Walking the Tightrope: Balancing Bias to Action and PlanningWalking the Tightrope: Balancing Bias to Action and Planning
Walking the Tightrope: Balancing Bias to Action and Planning
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
 
Usp友の会勉強会、ジャクソン構造図の巻(後編)
Usp友の会勉強会、ジャクソン構造図の巻(後編)Usp友の会勉強会、ジャクソン構造図の巻(後編)
Usp友の会勉強会、ジャクソン構造図の巻(後編)
 
Bashのヒストリ展開を活用する
Bashのヒストリ展開を活用するBashのヒストリ展開を活用する
Bashのヒストリ展開を活用する
 
FreeBSDのブートプロセス
FreeBSDのブートプロセスFreeBSDのブートプロセス
FreeBSDのブートプロセス
 
2015.08.29 JUS共催勉強会資料
2015.08.29 JUS共催勉強会資料2015.08.29 JUS共催勉強会資料
2015.08.29 JUS共催勉強会資料
 
Usp友の会勉強会、ジャクソン構造図の巻(前編)
Usp友の会勉強会、ジャクソン構造図の巻(前編)Usp友の会勉強会、ジャクソン構造図の巻(前編)
Usp友の会勉強会、ジャクソン構造図の巻(前編)
 
シェルスクリプトを極める
シェルスクリプトを極めるシェルスクリプトを極める
シェルスクリプトを極める
 
Linux女子部第二回勉強会usp友の会
Linux女子部第二回勉強会usp友の会Linux女子部第二回勉強会usp友の会
Linux女子部第二回勉強会usp友の会
 
Awk勉強会用資料
Awk勉強会用資料Awk勉強会用資料
Awk勉強会用資料
 
Tips for bash script
Tips for bash scriptTips for bash script
Tips for bash script
 
正規表現
正規表現正規表現
正規表現
 
awk入門
awk入門awk入門
awk入門
 

Similar to Linux Instrumentation

Linux Operating System (Presented in ICS Course at United International Unive...
Linux Operating System (Presented in ICS Course at United International Unive...Linux Operating System (Presented in ICS Course at United International Unive...
Linux Operating System (Presented in ICS Course at United International Unive...Md. Fahim Bin Amin
 
Enea Linux and LWRT FTF China 2012
Enea Linux and LWRT FTF China 2012Enea Linux and LWRT FTF China 2012
Enea Linux and LWRT FTF China 2012EneaSoftware
 
Linux.ppt
Linux.ppt Linux.ppt
Linux.ppt onu9
 
Profiling Multicore Systems to Maximize Core Utilization
Profiling Multicore Systems to Maximize Core Utilization Profiling Multicore Systems to Maximize Core Utilization
Profiling Multicore Systems to Maximize Core Utilization mentoresd
 
Linux/Unix Night - (PEN) Testing Toolkits (English)
Linux/Unix Night - (PEN) Testing Toolkits (English)Linux/Unix Night - (PEN) Testing Toolkits (English)
Linux/Unix Night - (PEN) Testing Toolkits (English)Jelmer de Reus
 
Advance linux presentation_0702011
Advance linux presentation_0702011Advance linux presentation_0702011
Advance linux presentation_0702011Aravindan Arun
 
Intro to Linux (for Ham Radio Operators)
Intro to Linux (for Ham Radio Operators)Intro to Linux (for Ham Radio Operators)
Intro to Linux (for Ham Radio Operators)Tanner Lovelace
 
Linux operating system ppt
Linux operating system pptLinux operating system ppt
Linux operating system pptAchyut Sinha
 
Group project linux helix
Group project linux helixGroup project linux helix
Group project linux helixJeff Carroll
 
linux operating system.pptx
linux operating system.pptxlinux operating system.pptx
linux operating system.pptxKailash510466
 
Linux operating system
Linux operating systemLinux operating system
Linux operating systemMohamed Essam
 
Ethical hacking Chapter 9 - Linux Vulnerabilities - Eric Vanderburg
Ethical hacking   Chapter 9 - Linux Vulnerabilities - Eric VanderburgEthical hacking   Chapter 9 - Linux Vulnerabilities - Eric Vanderburg
Ethical hacking Chapter 9 - Linux Vulnerabilities - Eric VanderburgEric Vanderburg
 
Introtolinux 110730150838-phpapp02
Introtolinux 110730150838-phpapp02Introtolinux 110730150838-phpapp02
Introtolinux 110730150838-phpapp02M Gabrail
 
Linux; Operating System
Linux; Operating SystemLinux; Operating System
Linux; Operating SystemJayEl Cadawas
 

Similar to Linux Instrumentation (20)

Linux Operating System (Presented in ICS Course at United International Unive...
Linux Operating System (Presented in ICS Course at United International Unive...Linux Operating System (Presented in ICS Course at United International Unive...
Linux Operating System (Presented in ICS Course at United International Unive...
 
Linux
LinuxLinux
Linux
 
Linux
LinuxLinux
Linux
 
Enea Linux and LWRT FTF China 2012
Enea Linux and LWRT FTF China 2012Enea Linux and LWRT FTF China 2012
Enea Linux and LWRT FTF China 2012
 
Linux.ppt
Linux.ppt Linux.ppt
Linux.ppt
 
Profiling Multicore Systems to Maximize Core Utilization
Profiling Multicore Systems to Maximize Core Utilization Profiling Multicore Systems to Maximize Core Utilization
Profiling Multicore Systems to Maximize Core Utilization
 
Linux/Unix Night - (PEN) Testing Toolkits (English)
Linux/Unix Night - (PEN) Testing Toolkits (English)Linux/Unix Night - (PEN) Testing Toolkits (English)
Linux/Unix Night - (PEN) Testing Toolkits (English)
 
Rhce ppt
Rhce pptRhce ppt
Rhce ppt
 
Advance linux presentation_0702011
Advance linux presentation_0702011Advance linux presentation_0702011
Advance linux presentation_0702011
 
Intro to Linux (for Ham Radio Operators)
Intro to Linux (for Ham Radio Operators)Intro to Linux (for Ham Radio Operators)
Intro to Linux (for Ham Radio Operators)
 
AZLinux.Windows2Linux en Red.es
AZLinux.Windows2Linux en Red.esAZLinux.Windows2Linux en Red.es
AZLinux.Windows2Linux en Red.es
 
Linux operating system ppt
Linux operating system pptLinux operating system ppt
Linux operating system ppt
 
Group project linux helix
Group project linux helixGroup project linux helix
Group project linux helix
 
Intro to linux
Intro to linuxIntro to linux
Intro to linux
 
linux operating system.pptx
linux operating system.pptxlinux operating system.pptx
linux operating system.pptx
 
Linux operating system
Linux operating systemLinux operating system
Linux operating system
 
Ethical hacking Chapter 9 - Linux Vulnerabilities - Eric Vanderburg
Ethical hacking   Chapter 9 - Linux Vulnerabilities - Eric VanderburgEthical hacking   Chapter 9 - Linux Vulnerabilities - Eric Vanderburg
Ethical hacking Chapter 9 - Linux Vulnerabilities - Eric Vanderburg
 
Introtolinux 110730150838-phpapp02
Introtolinux 110730150838-phpapp02Introtolinux 110730150838-phpapp02
Introtolinux 110730150838-phpapp02
 
Linux; Operating System
Linux; Operating SystemLinux; Operating System
Linux; Operating System
 
Linux
Linux Linux
Linux
 

Recently uploaded

From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
Fact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMsFact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMsZilliz
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
Fact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMsFact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMs
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 

Linux Instrumentation

  • 1. Instrumentation Tools Demonstrations Questions Linux Instrumentation Ian Munsie June 24, 2010 Ian Munsie Linux Instrumentation
  • 2. Instrumentation Tools Demonstrations Questions Outline 1 Overview of available Instrumentation Tools Competition’s Instrumentation Tools Overview of Linux Instrumentation Tools Interactions Between Instrumentation Tools Interesting Instrumentation Tools 2 Demonstrations Finding cache misses with perf Locate sources of block I/O with tracepoints Using kprobes to analyse a running kernel Ian Munsie Linux Instrumentation
  • 3. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Outline 1 Overview of available Instrumentation Tools Competition’s Instrumentation Tools DTrace Overview of Linux Instrumentation Tools tracepoints kprobes uprobes gprof ptrace utrace ftrace LTTng oProfile SystemTap performance events Interactions Between Instrumentation Tools Interesting Instrumentation Tools Ian Munsie Linux Instrumentation
  • 4. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting What major players are in this area? DTrace Ian Munsie Linux Instrumentation
  • 5. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting What major players are in this area? DTrace Ian Munsie Linux Instrumentation
  • 6. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  • 7. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  • 8. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  • 9. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace All purpose kernel and userspace tracing “Better than Linux” Originated from Solaris Mainstream ports for FreeBSD, NetBSD, Mac OS X Licencing Issues with Linux No performance impact when probes disabled Minimal probe effect Definition Probe Effect: The phenomena where observing a system will change the behaviour of that system Ian Munsie Linux Instrumentation
  • 10. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace Implementation Well documented probes Definition Probe: A location or action which can be hooked into in order to perform some arbitrary action Probes placed statically in source code Probe location well considered Probes in Solaris kernel, postgreSQL, x.org, . . . Can create new probes dynamically “Dynamic tracing framework” DTrace D programming language Ian Munsie Linux Instrumentation
  • 11. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace Implementation Well documented probes Definition Probe: A location or action which can be hooked into in order to perform some arbitrary action Probes placed statically in source code Probe location well considered Probes in Solaris kernel, postgreSQL, x.org, . . . Can create new probes dynamically “Dynamic tracing framework” DTrace D programming language Ian Munsie Linux Instrumentation
  • 12. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace Implementation Well documented probes Definition Probe: A location or action which can be hooked into in order to perform some arbitrary action Probes placed statically in source code Probe location well considered Probes in Solaris kernel, postgreSQL, x.org, . . . Can create new probes dynamically “Dynamic tracing framework” DTrace D programming language Ian Munsie Linux Instrumentation
  • 13. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace on Linux? DTrace approach generally unsuitable for Linux kernel Linux kernel evolves rapidly Resistance to placement of static probes Unofficial Linux port does exist SystemTap can use DTrace markers Ian Munsie Linux Instrumentation
  • 14. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting DTrace on Linux? DTrace approach generally unsuitable for Linux kernel Linux kernel evolves rapidly Resistance to placement of static probes Unofficial Linux port does exist SystemTap can use DTrace markers Ian Munsie Linux Instrumentation
  • 15. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Enough DTrace envy, what about Linux? Ian Munsie Linux Instrumentation
  • 16. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel oProfile tracepoints LTTng Hardware Counters uprobes perf events ftrace utrace kprobes ptrace Kernel SystemTap Modules perf gprof gdb strace DTrace Markers Userspace Ian Munsie Linux Instrumentation
  • 17. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Static probes? tracepoints Ian Munsie Linux Instrumentation
  • 18. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Static probes? tracepoints Ian Munsie Linux Instrumentation
  • 19. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Tracepoints Static probe points in the kernel source Replace old Kernel Markers Infrastructure mainline Low performance impact TRACE_EVENT() macro Ian Munsie Linux Instrumentation
  • 20. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel dynamic probes? kprobes Ian Munsie Linux Instrumentation
  • 21. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel dynamic probes? kprobes Ian Munsie Linux Instrumentation
  • 22. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kprobes Insert a new probe into kernel at runtime Does not do userspace probes Heavily architecture specific Can be used by loadable kernel modules Used by other instrumentation tools Provides three types of probes kprobes—Probe instruction jprobes—Probe function call kretprobes—Probe function return Mainline Ian Munsie Linux Instrumentation
  • 23. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace dynamic probes? uprobes Not mainline yet. Ian Munsie Linux Instrumentation
  • 24. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace dynamic probes? uprobes Not mainline yet. Ian Munsie Linux Instrumentation
  • 25. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace dynamic probes? uprobes Not mainline yet. Ian Munsie Linux Instrumentation
  • 26. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Uprobes Kprobes for userspace Not mainline yet—predict that may soon change Used by SystemTap Handlers run in task context Background page replacement mechanism Probed instruction single stepped in XOL area No longer depends on utrace Handlers implemented as kernel module perf interface Ian Munsie Linux Instrumentation
  • 27. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace profiling? gprof Ian Munsie Linux Instrumentation
  • 28. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace profiling? gprof Ian Munsie Linux Instrumentation
  • 29. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting gprof Legacy, but deserves a mention Counts and profiles calls to functions Function granuality Compile executable with -pg flag Produce function call graph Does NOT profile libraries or kernel Kprof provides GUI to visualise call graph Ian Munsie Linux Instrumentation
  • 30. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting gprof Flat Profile Ian Munsie Linux Instrumentation
  • 31. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting gprof Call Graph Ian Munsie Linux Instrumentation
  • 32. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting kprof Profile Visualisation Tool Ian Munsie Linux Instrumentation
  • 33. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace tracing? ptrace . . . A bit ugly. Ian Munsie Linux Instrumentation
  • 34. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace tracing? ptrace . . . A bit ugly. Ian Munsie Linux Instrumentation
  • 35. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Userspace tracing? ptrace . . . A bit ugly. Ian Munsie Linux Instrumentation
  • 36. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ptrace Means of one process observing/controlling another Prominently used by gdb, strace, ltrace Ugly, limited interface Processes cannot be traced by multiple processes Signal oriented architecture Large overhead Ian Munsie Linux Instrumentation
  • 37. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ptrace—ltrace Ian Munsie Linux Instrumentation
  • 38. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ptrace—strace Ian Munsie Linux Instrumentation
  • 39. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Improved userspace tracing? utrace Not mainline yet. Ian Munsie Linux Instrumentation
  • 40. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Improved userspace tracing? utrace Not mainline yet. Ian Munsie Linux Instrumentation
  • 41. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Improved userspace tracing? utrace Not mainline yet. Ian Munsie Linux Instrumentation
  • 42. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting utrace Improved userspace tracing Not mainline Prominently used by SystemTap Utrace clients run in kernel, typically in module Ptrace reimplemented as a utrace client Infrastructure to monitor threads Establish “engine” for each monitored thread Event reporting Thread control Thread machine state access Ian Munsie Linux Instrumentation
  • 43. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel tracing? ftrace Ian Munsie Linux Instrumentation
  • 44. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel tracing? ftrace Ian Munsie Linux Instrumentation
  • 45. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace Mainline Kernel tracing infrastructure Originated from the realtime efforts Hijacks mcount (from gprof) for own purposes Low performance impact Tracing Plugins Function Funtion graph Context switches Time interrupts disabled Time preemtion disabled Delay for high priority tasks Power state transitions Branch prediction ... Exposed through debugfs—Manipulate with echo & cat Gained event support—perf events favoured for this Ian Munsie Linux Instrumentation
  • 46. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace—Function Tracer Ian Munsie Linux Instrumentation
  • 47. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace—Task scheduling Ian Munsie Linux Instrumentation
  • 48. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting ftrace—Measuring task wakeup latency Ian Munsie Linux Instrumentation
  • 49. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel and Userspace tracing? LTTng Not mainline. Ian Munsie Linux Instrumentation
  • 50. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel and Userspace tracing? LTTng Not mainline. Ian Munsie Linux Instrumentation
  • 51. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Kernel and Userspace tracing? LTTng Not mainline. Ian Munsie Linux Instrumentation
  • 52. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting LTTng Kernel & Userspace tracing package Not mainline Adds instrumentation points into kernel Uses tracepoints, ftrace, kprobes Can plot traces Completely failed to work for me in every way Ian Munsie Linux Instrumentation
  • 53. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting LTTng Kernel & Userspace tracing package Not mainline Adds instrumentation points into kernel Uses tracepoints, ftrace, kprobes Can plot traces Completely failed to work for me in every way Ian Munsie Linux Instrumentation
  • 54. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting LTTng Screenshot LTTng did not work for me, but here’s what it’s supposed to do: Ian Munsie Linux Instrumentation
  • 55. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Hardware performance counters? oProfile Ian Munsie Linux Instrumentation
  • 56. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Hardware performance counters? oProfile Ian Munsie Linux Instrumentation
  • 57. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile System wide sampling profiler Mainline Interface to Performance Measurement Unit GUI eases selection of performance counters Can generate gprof style call graph Ian Munsie Linux Instrumentation
  • 58. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile GUI Ian Munsie Linux Instrumentation
  • 59. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile Report Ian Munsie Linux Instrumentation
  • 60. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile Annotate Ian Munsie Linux Instrumentation
  • 61. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting oProfile Call Graph Ian Munsie Linux Instrumentation
  • 62. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something to pull everything together? SystemTap Not mainline. Ian Munsie Linux Instrumentation
  • 63. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something to pull everything together? SystemTap Not mainline. Ian Munsie Linux Instrumentation
  • 64. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something to pull everything together? SystemTap Not mainline. Ian Munsie Linux Instrumentation
  • 65. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting SystemTap Powerful kernel & userspace analysis suite Scripting language Generates, Compiles and loads kernel modules Primarily uses kprobes Utrace for userspace tracing Uprobes support Many userspace dependencies Hard to use More of interest to enterprise distros Version 1.2 recently released Prototypical support for perf events Hardware breakpoint probe support Not mainline and . . . Ian Munsie Linux Instrumentation
  • 66. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting SystemTap Let’s face it, system tap isn’t going to be merged, so why even bring it up? Every kernel developer I have _ever_ seen agrees that all the new tracing is a million times superior. I’m sure there are system tap people who disagree, but quite frankly, I don’t see it being merged considering how little the system tap people ever did for the kernel. So if things like system tap and "security models that go behind the kernel by tying into utrace" are the reasons for utrace, color me utterly uninterested. In fact, color me actively hostile. I think that’s the worst possible situation that we’d ever be in as kernel people (namely exactly the "do things in kernel space by hiding behind utrace without having kernel people involved") Linus Ian Munsie Linux Instrumentation
  • 67. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something else to pull everything together? perf events The future of Linux Instrumentation. Ian Munsie Linux Instrumentation
  • 68. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something else to pull everything together? perf events The future of Linux Instrumentation. Ian Munsie Linux Instrumentation
  • 69. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Something else to pull everything together? perf events The future of Linux Instrumentation. Ian Munsie Linux Instrumentation
  • 70. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf events All in one Performance tool Newcomer, but already mainline Interface to Performance Measurement Unit Virtual counters Integrates with tracepoints Kprobe support Scripting support SLAB subsystem profiling Lock profiling Scheduler analyser Timechart Ian Munsie Linux Instrumentation
  • 71. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Sample perf report Ian Munsie Linux Instrumentation
  • 72. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Sample perf call graph Ian Munsie Linux Instrumentation
  • 73. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf—Available events Ian Munsie Linux Instrumentation
  • 74. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf—Available events Ian Munsie Linux Instrumentation
  • 75. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf top Ian Munsie Linux Instrumentation
  • 76. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf timechart Ian Munsie Linux Instrumentation
  • 77. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting perf timechart Ian Munsie Linux Instrumentation
  • 78. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting How does it all fit together? Ian Munsie Linux Instrumentation
  • 79. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel oProfile tracepoints LTTng Hardware Counters uprobes perf events ftrace utrace kprobes ptrace Kernel SystemTap Modules perf gprof gdb strace DTrace Markers Userspace Ian Munsie Linux Instrumentation
  • 80. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace SystemTap LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 81. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 82. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 83. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 84. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 85. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 86. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 87. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 88. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 89. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 90. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 91. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 92. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 93. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 94. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 95. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 96. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints ftrace perf events Ian Munsie Linux Instrumentation
  • 97. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints—Mainline, Used by others ftrace perf events Ian Munsie Linux Instrumentation
  • 98. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints—Mainline, Used by others ftrace—Mainline, Active perf events Ian Munsie Linux Instrumentation
  • 99. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Interesting Technologies for the Future DTrace—Licence issues, not suited to Linux SystemTap—Not mainline, enterprise focus LTTng—Not mainline, enterprise focus gprof—Limitations, not going anywhere ptrace—Ugly, set in stone utrace—Trouble going mainline oProfile—Not going anywhere uprobes—Not mainline yet, likely soon. . . kprobes—Mainline, Used by others tracepoints—Mainline, Used by others ftrace—Mainline, Active perf events—Mainline, Very active Ian Munsie Linux Instrumentation
  • 100. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel oProfile tracepoints LTTng Hardware Counters uprobes perf events ftrace utrace kprobes ptrace Kernel SystemTap Modules perf gprof gdb strace DTrace Markers Userspace Ian Munsie Linux Instrumentation
  • 101. Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting Mainline Kernel Non-Mainline Kernel tracepoints Hardware Counters perf events ftrace kprobes perf Userspace Ian Munsie Linux Instrumentation
  • 102. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Outline 2 Demonstrations Finding cache misses with perf Locate sources of block I/O with tracepoints Using kprobes to analyse a running kernel Ian Munsie Linux Instrumentation
  • 103. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Ian Munsie Linux Instrumentation
  • 104. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Fairly common scenario High cache misses is bad for performance Use a very simple C program as an example Task: sum every element in a large array Complication: array & program too big for L2 cache Two approaches with different array access order Use perf to observe cache misses Ian Munsie Linux Instrumentation
  • 105. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Fairly common scenario High cache misses is bad for performance Use a very simple C program as an example Task: sum every element in a large array Complication: array & program too big for L2 cache Two approaches with different array access order Use perf to observe cache misses Ian Munsie Linux Instrumentation
  • 106. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Ian Munsie Linux Instrumentation
  • 107. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf Ian Munsie Linux Instrumentation
  • 108. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf $ perf record -e cache-misses ./cachetest $ perf report Ian Munsie Linux Instrumentation
  • 109. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Finding cache misses with perf $ perf record -e cache-misses ./cachetest $ perf report Ian Munsie Linux Instrumentation
  • 110. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Locate sources of block I/O with perf and tracepoints Ian Munsie Linux Instrumentation
  • 111. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes perf—Locate sources of block I/O System is under heavy disk I/O causing slowdowns Traditional top does not reveal offending processes Even atop only shows I/O at process granularity block:block_rq_issue tracepoint available Kernel must be compiled with block I/O tracing perf can produce call graphs to track exact origin Ian Munsie Linux Instrumentation
  • 112. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes perf—Locate sources of block I/O # perf record -g -a -e block:block_rq_issue sleep 10 # perf report Ian Munsie Linux Instrumentation
  • 113. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes perf—Locate sources of block I/O # perf record -g -a -e block:block_rq_issue sleep 10 # perf report Ian Munsie Linux Instrumentation
  • 114. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analyse a running kernel with perf and kprobes Ian Munsie Linux Instrumentation
  • 115. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analyse a running kernel with perf and kprobes Want to pull information out of a running kernel Information is not already revealed to userspace No tracepoints provide the information Live system—cannot afford downtime Kernel debugging information and source is available Let’s probe the scheduler as an example. . . Ian Munsie Linux Instrumentation
  • 116. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analysing a running kernel with perf and kprobes # perf probe –line schedule Ian Munsie Linux Instrumentation
  • 117. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analysing a running kernel with perf and kprobes Ian Munsie Linux Instrumentation
  • 118. Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes Analysing a running kernel with perf and kprobes # perf report Ian Munsie Linux Instrumentation
  • 119. Instrumentation Tools Demonstrations Questions Thankyou all for listening perf can be found in the Linux kernel sources under /tools/perf perf is being actively developed in the “tip” kernel tree Linux Weekly News has some good further reading on ftrace—google site:lwn.net ftrace perf articles not as easy to find yet, here are some: http://lwn.net/Articles/339361—“Perfcounters added to the mainline” http://lwn.net/Articles/373842—“Scripting support for perf” http://lwn.net/Articles/382554—“A “live mode” for perf” http://lwn.net/Articles/346470—“Fun with tracepoints” http://lkml.org/lkml/2010/3/17/91—Ingo Molnar on database I/O latency Ian Munsie Linux Instrumentation