DTRACEOverviewJérôme GauthierMay 2011                  COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
AGENDA1. Context2. DTrace Framework3. System inspection and analysis4. DTrace Toolkit5. Demo                              ...
ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo                                             3     ...
CONTEXT    We need tools to debug and observe different situations …                                                      ...
CONTEXT                                          … in several areas                  pkill                       pargs    ...
CONTEXT• Some existing utilities are process based• Some are only inspecting certain parts of the system    • Disks    • V...
ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo                                             7     ...
DTRACE FRAMEWORK• DTrace : A facility for dynamic instrumentation of production systems      Another way to debug and obse...
DTRACE FRAMEWORK                                        User level            Operating System             DTRACE         ...
DTRACE FRAMEWORK• A framework for real-time analysis and observation• Designed for live production systems      A totally ...
DTRACE FRAMEWORK• Everything about DTrace is virtualized per consumer      There is no limit on the number of concurrent D...
ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo                                            12     ...
SYSTEM INSPECTION AND ANALYSISPROBES• Probes are programmable sensors placed all over the system• A probe fires when the e...
probe descriptionSYSTEM INSPECTION AND ANALYSIS                                                                           ...
SYSTEM INSPECTION AND ANALYSISPROBES• Actions are used to record data to a DTrace buffer• Different types of actions:     ...
probe descriptionSYSTEM INSPECTION AND ANALYSIS                                                                           ...
SYSTEM INSPECTION AND ANALYSISPROVIDERS                  < provider, module, function, name >• DTrace probes come from a s...
SYSTEM INSPECTION AND ANALYSISPROVIDERS• Example of some providers :         SYSCALL                                  PROC...
SYSTEM INSPECTION AND ANALYSISD LANGUAGE                                                                           /* D Pr...
SYSTEM INSPECTION AND ANALYSISD LANGUAGE : AGGREGATIONS                  @name[keys] = aggfunction(args)• Used to aggregat...
SYSTEM INSPECTION AND ANALYSISD LANGUAGE : SECURITY                                                                       ...
ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo                                            22     ...
DTRACE TOOLKIT• Collection of over 100 useful documented scripts built on top of DTrace  Framework      Developed by the O...
DTRACE TOOLKITCATEGORIES                                DTrace Toolkit                                                    ...
ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo                                            25     ...
DEMO    Command line scripts                                                               Chime scriptstcpstat           ...
Questions?jerome.gauthier@alcatel-lucent.com                              27     COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGH...
REFERENCES & ADDITIONAL INFORMATION• PowerPoint presentations    • Joyent Performance Visualization – Brendan Gregg    • D...
29COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
Upcoming SlideShare
Loading in...5
×

Dtrace Overview

974

Published on

Overview of DTrace Monitoring tool

Published in: Technology, Business

Dtrace Overview

  1. 1. DTRACEOverviewJérôme GauthierMay 2011 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  2. 2. AGENDA1. Context2. DTrace Framework3. System inspection and analysis4. DTrace Toolkit5. Demo 2 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  3. 3. ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo 3 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  4. 4. CONTEXT We need tools to debug and observe different situations … Understand the Analyze the system system utilization or performance saturation Debug an Debug a application system/kernel crash 4 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  5. 5. CONTEXT … in several areas pkill pargs pflags truss mdb pgrep pstop Process Process Process control statistics debugging System Kernel Kernel statistics statistics debugging vmstat lockstat iostat cpustat kstat kmdbExamples of some Solaris commands 5 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  6. 6. CONTEXT• Some existing utilities are process based• Some are only inspecting certain parts of the system • Disks • Virtual memory • Network • Kernel Need a tool to simplify the analysis of all parts of a system This tool must not make the system any slower 6 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  7. 7. ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo 7 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  8. 8. DTRACE FRAMEWORK• DTrace : A facility for dynamic instrumentation of production systems Another way to debug and observe the entire system and understand the big picture• Open source (CDDL = Common Development and Distribution License)• Supported platforms : Unix-like systems : 8 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  9. 9. DTRACE FRAMEWORK User level Operating System DTRACE Programs Kernel level• Examine how the system works• Track down performance problems across many layers of software• Locate the cause of aberrant behavior Does not replace or retire other system utilities 9 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  10. 10. DTRACE FRAMEWORK• A framework for real-time analysis and observation• Designed for live production systems A totally safe way to inspect live data on production systems No need to stop or restart applications Live measurement and interpretation• Over 30 000 data monitoring points spread in all system• Zero effect when disabled, safe and no system slowdown when enabled 10 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  11. 11. DTRACE FRAMEWORK• Everything about DTrace is virtualized per consumer There is no limit on the number of concurrent DTrace consumers• Possibility to create custom programs to dynamically instrument the system• Example of problems that can be detected and analyzed: High CPU utilization Acute memory pressure Abnormal I/O activity Excessive network traffic 11 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  12. 12. ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo 12 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  13. 13. SYSTEM INSPECTION AND ANALYSISPROBES• Probes are programmable sensors placed all over the system• A probe fires when the event happens Then, certain actions are executed only if the predicate expression is true• General form of a probe clause : probe description /predicate/ { actions } 13 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  14. 14. probe descriptionSYSTEM INSPECTION AND ANALYSIS /predicate/PROBES { actions }• A predicate allow action to only be taken when certain conditions are met Example : Look only for a process which has the pid=1203 Match a process which has the name firefox-bin• An action can be “ record a stack trace, a timestamp, or the argument to a function ”• When some probes fire, DTrace gathers the data and reports it back to you If no action specified, DTrace will just take note of each time the probe fires 14 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  15. 15. SYSTEM INSPECTION AND ANALYSISPROBES• Actions are used to record data to a DTrace buffer• Different types of actions: Type of actions Example Explanation trace() Records the result of trace to the bufferData recording printf() Traces a D expression stack() Records a kernel stack trace stop() Stops the process which has executed the probeDestructive raise() Signal a process at a precise point during execution panic() Force a crash dump exit() Stop tracing and exitsSpecial copyin() Creates a buffer and returns its address strlen() Returns the length of a string in bytes 15 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  16. 16. probe descriptionSYSTEM INSPECTION AND ANALYSIS /predicate/PROBES { actions }• Each probe is uniquely identified by a 4-tuple : < provider, module, function, name > Probe description The name of the module in which the probe is locatedModule Either the name of a kernel module or the name of a user library If this probe correspond to a specific program location, it’s the nameFunction of the program function in which the probe is located. The final component of the tuple is the string name of the probe thatName give you some idea of the probe’s semantic meaning.• Some probes does not have a module and function (e.g. : BEGIN, END) We can identify a probe only by its name 16 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  17. 17. SYSTEM INSPECTION AND ANALYSISPROVIDERS < provider, module, function, name >• DTrace probes come from a set of kernel modules called providers Which correspond to a particular kind of instrumentation• We can list the probes that a provider can transmit to the DTrace Framework• A provider pass the control to DTrace when you decide to enable one of its probes 17 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  18. 18. SYSTEM INSPECTION AND ANALYSISPROVIDERS• Example of some providers : SYSCALL PROC SCHED • Holds the entire • Handle process creation • CPU scheduling: sleeping, communication from and termination running threads userland to kernel space • Sending and handling • CPU time: which threads • Every system call on the signals are run by which CPU and system for how long IO MIB PROFILE • I/O system • Counters for • Time based probing at • Disk input and output management information specific interval of times requests bases • To sample anything in • I/O by device, process, • IP, IPv6, ICMP, IPSec the system every unit time size, filename 18 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  19. 19. SYSTEM INSPECTION AND ANALYSISD LANGUAGE /* D Program */• A D program consist of one or more probe clauses probe1 description /predicate/ {• D Language is like C language with some actions constructs similar with awk : } • Support ANSI C operators probe2 description /predicate/ • Support Strings { actions • D expressions are based on built-in variables : } pid, execname, timestamps, curthread probe3 description /predicate/• No control-flow constructs : loops, if statements { actions• Floating-point arithmetic is not permitted (only } Integers) 19 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  20. 20. SYSTEM INSPECTION AND ANALYSISD LANGUAGE : AGGREGATIONS @name[keys] = aggfunction(args)• Used to aggregate data and look for trends• Simple to generate report about : • Total system calls by a process or an application • Total number of read or write by process• Aggregating functions: • count() • min() • sum() • max() • avg() • quantize() 20 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  21. 21. SYSTEM INSPECTION AND ANALYSISD LANGUAGE : SECURITY D program source files userland DTrace consumer: dtrace()• D programs are compiled into a safe intermediate form that is used for DTrace safe execution execution when a probe fires environment kernel DTrace providers• Programming mistakes : DTrace will report errors and disable instrumentation It is impossible to construct a bad script that would cause damage to the system• DTrace handle run-time errors and report them (dividing by zero, dereferencing invalid memory, …) 21 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  22. 22. ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo 22 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  23. 23. DTRACE TOOLKIT• Collection of over 100 useful documented scripts built on top of DTrace Framework Developed by the OpenSolaris DTrace community• The toolkit contains : the scripts the man pages the example documentation the notes files the tutorials• The script are sorted by categories 23 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  24. 24. DTRACE TOOLKITCATEGORIES DTrace Toolkit Extra, User, System Applications Processes Network Memory System Kernel Disk CPU DTrace Framework 24 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  25. 25. ContextDTrace FrameworkSystem inspection and analysisDTrace ToolkitDemo 25 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  26. 26. DEMO Command line scripts Chime scriptstcpstat system calls System calls counter by process, TCP bytes received and send function, module, …httpdstat php calltime Measure PHP elapsed times for Real-time Apache Web Server stats functions 26 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  27. 27. Questions?jerome.gauthier@alcatel-lucent.com 27 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  28. 28. REFERENCES & ADDITIONAL INFORMATION• PowerPoint presentations • Joyent Performance Visualization – Brendan Gregg • DTrace & DTrace Toolkit – Stefan Parvu• Links • DTrace community site: http://hub.opensolaris.org/bin/view/Community+Group+dtrace/WebHome • Brendan Gregg site: http://www.brendangregg.com/dtrace.html • DTrace Blogs: http://dtrace.org/blogs/ • DTrace online guide: http://wikis.sun.com/display/DTrace/Documentation • DTrace wiki: http://www.solarisinternals.com/wiki/index.php/DTraceToolkit • DTrace training: http://dtracehol.com 28 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  29. 29. 29COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×