Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DTrace - Miracle Scotland Database Forum


Published on

Published in: Technology, Business
  • Be the first to comment

DTrace - Miracle Scotland Database Forum

  1. 1. Playing around with DTrace (… and Oracle, of course) Doug Burns [email_address]
  2. 2. Agenda <ul><li>Introduction </li></ul><ul><li>What is DTrace? </li></ul><ul><li>DTrace Basics </li></ul><ul><li>A Few Oracle Examples </li></ul><ul><li>Quirks and Issues </li></ul><ul><li>Conclusions </li></ul>
  3. 3. Introduction <ul><li>Welcome to Scotland </li></ul><ul><li>I’m a Campbell </li></ul><ul><li>Search Google for “ Campbell Glencoe ” </li></ul><ul><li>The picture you are about to see is well-chosen </li></ul>
  4. 4. The Glencoe Massacre of 1692
  5. 5. Introduction <ul><li>So now you know I am definitely not to be trusted. </li></ul><ul><ul><li>I’m a Campbell </li></ul></ul><ul><ul><li>I enjoy traditional Highland hospitality … </li></ul></ul><ul><ul><li>… and then kill my hosts </li></ul></ul><ul><li>However, you may be safe … </li></ul><ul><ul><li>I don’t drink Whisky </li></ul></ul><ul><ul><li>I don’t wear Skirts! </li></ul></ul><ul><li>See how dangerous a simple Tie can be? </li></ul>
  6. 6. What is DTrace? <ul><li>Introduction </li></ul><ul><li>What is DTrace? </li></ul><ul><li>DTrace Basics </li></ul><ul><li>A Few Oracle Examples </li></ul><ul><li>Quirks and Issues </li></ul><ul><li>Conclusions </li></ul>
  7. 7. <ul><li>Similar story to Oracle Wait Statistics </li></ul><ul><ul><li>See Juan Loiza quotation in Chapter 2 of TOTOT </li></ul></ul><ul><ul><li>A problem with a benchmark </li></ul></ul><ul><ul><li>Lots of guesses - none correct </li></ul></ul><ul><ul><li>Instrumentation helps - if you thought of it </li></ul></ul><ul><ul><li>Sometimes you have to stop to move forward </li></ul></ul><ul><li>Sometimes you need to see what’s really going on – even (or particularly) on Production systems </li></ul>What is DTrace? - History
  8. 8. <ul><li>Dynamic Tracing framework. </li></ul><ul><li>Why ‘Dynamic’? </li></ul><ul><ul><li>No need to instrument your code </li></ul></ul><ul><ul><ul><li>Although you can, by implementing a DTrace provider . </li></ul></ul></ul><ul><ul><li>When you hit a performance problem, you can enable probes </li></ul></ul><ul><ul><ul><li>Perhaps like ASH but without the constant overhead? </li></ul></ul></ul><ul><ul><ul><li>More like dbms_support.start_trace_in_session </li></ul></ul></ul><ul><li>What can I trace? </li></ul><ul><ul><li>Erm, just about everything </li></ul></ul>What is DTrace? – The Name
  9. 9. What is DTrace? – The Future <ul><li>Open Source </li></ul><ul><ul><li>Thriving Community </li></ul></ul><ul><li>DTrace is available on Solaris 10+ </li></ul><ul><ul><li>Remember that Solaris != SPARC </li></ul></ul><ul><ul><li>FreeBSD and Mac OS X - soon </li></ul></ul><ul><ul><li>AIX? </li></ul></ul><ul><ul><ul><li>Search Google for ‘AIX 6 Preview’ </li></ul></ul></ul><ul><ul><ul><li>probevue </li></ul></ul></ul><ul><ul><li>Linux? </li></ul></ul>
  10. 10. DTrace Basics <ul><li>Introduction </li></ul><ul><li>The Need for DTrace </li></ul><ul><li>DTrace Basics </li></ul><ul><li>A Few Oracle Examples </li></ul><ul><li>Quirks and Issues </li></ul><ul><li>Conclusions </li></ul>
  11. 11. DTrace Basics – D Language <ul><li>‘ D’ Scripting Language </li></ul><ul><li>A combination of C and awk? </li></ul><ul><ul><li>But don’t let that put you off ;-) </li></ul></ul><ul><li>Event-driven so you :- </li></ul><ul><ul><li>Define the Events (Probes) to trap </li></ul></ul><ul><ul><li>Write the handlers </li></ul></ul><ul><li>Let’s take a look at a simple example </li></ul><ul><ul><li>But ‘Hello World’ is not interesting </li></ul></ul>
  12. 12. DTrace Basics – Defining Probes <ul><li>Probes can be identified by name or number </li></ul><ul><li>A probe name is a 4-tuple </li></ul><ul><ul><li>provider:module:function:name </li></ul></ul><ul><ul><li>A space or asterisk is a wildcard </li></ul></ul><ul><ul><li>Some providers do not provide modules </li></ul></ul><ul><li>Examples </li></ul><ul><ul><li>pid456:libc:malloc:entry </li></ul></ul><ul><ul><li>pid1910:oracle:krccdw: </li></ul></ul><ul><ul><li>sched:::enqueue </li></ul></ul><ul><ul><li>dtrace:::BEGIN </li></ul></ul>
  13. 13. DTrace Basics – One-Liner <ul><li>Use any probe definition as a one line program </li></ul><ul><ul><li>Enable probes using -n </li></ul></ul><ul><li>dtrace -n pid1910:oracle:krccdw: </li></ul><ul><li>dtrace: description 'pid1910:oracle:krccdw:' matched 108 probes </li></ul><ul><li>CPU ID FUNCTION:NAME </li></ul><ul><li>0 66998 krccdw:entry </li></ul><ul><li>0 207834 krccdw:0 </li></ul><ul><li>0 207835 krccdw:1 </li></ul><ul><li>0 207836 krccdw:3 </li></ul><ul><li>0 207837 krccdw:6 </li></ul><ul><li>0 207838 krccdw:9 </li></ul><ul><li>0 207839 krccdw:a </li></ul><ul><li>0 207840 krccdw:b </li></ul><ul><li>0 207841 krccdw:c </li></ul>
  14. 14. DTrace Basics – One-Liner <ul><li>Make a small change in probe definition </li></ul><ul><li>dtrace -n ::oracle:krccdw:entry </li></ul><ul><li>dtrace: description '::oracle:krccdw:entry' matched 1 probe </li></ul><ul><li>CPU ID FUNCTION:NAME </li></ul><ul><li>2 43530 krccdw:entry </li></ul><ul><li>3 43530 krccdw:entry </li></ul><ul><li>1 43530 krccdw:entry </li></ul><ul><li>1 43530 krccdw:entry </li></ul><ul><li>1 43530 krccdw:entry </li></ul><ul><li>2 43530 krccdw:entry </li></ul><ul><li>2 43530 krccdw:entry </li></ul><ul><li>2 43530 krccdw:entry </li></ul><ul><li>2 43530 krccdw:entry </li></ul>
  15. 15. DTrace Basics - Providers <ul><li>sched provider </li></ul><ul><ul><li>Access to O/S scheduling information </li></ul></ul><ul><ul><li>Probes which can trap information when processes are placed on to the run queue and when they are dequeued and allowed to run. </li></ul></ul><ul><ul><ul><li>sched:::enqueue </li></ul></ul></ul><ul><ul><ul><li>sched:::dequeue </li></ul></ul></ul><ul><ul><li>Probes that can be used to monitor time spent on the CPU. </li></ul></ul><ul><ul><ul><li>sched:::on-cpu </li></ul></ul></ul><ul><ul><ul><li>sched:::off-cpu </li></ul></ul></ul>
  16. 16. DTrace Basics - Providers <ul><li>dtrace </li></ul><ul><li>io </li></ul><ul><li>pid </li></ul><ul><li>Proc </li></ul><ul><li>syscall </li></ul><ul><li>sysinfo </li></ul>
  17. 17. DTrace Basics – A Script <ul><li>A more realistic Program Structure </li></ul><ul><ul><li>#pragma D option quiet </li></ul></ul><ul><ul><li>proc:::exec </li></ul></ul><ul><ul><li>{ self->parent = execname; </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>proc:::exec-success </li></ul></ul><ul><ul><li>/self->parent != NULL/ </li></ul></ul><ul><ul><li>{ @[self->parent, execname] = count(); </li></ul></ul><ul><ul><li>self->parent = NULL; </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>proc:::exec-failure </li></ul></ul><ul><ul><li>/self->parent != NULL/ </li></ul></ul><ul><ul><li>{ self->parent = NULL; </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>END </li></ul></ul><ul><ul><li>{ printf(&quot;%-20s %-20s %s &quot;, &quot;WHO&quot;, &quot;WHAT&quot;, &quot;COUNT&quot;); </li></ul></ul><ul><ul><li>printa(&quot;%-20s %-20s %@d &quot;, @); </li></ul></ul><ul><ul><li>} </li></ul></ul>
  18. 18. DTrace Basics – A Script <ul><li>Output </li></ul><ul><ul><li># dtrace -s ./whoexec.d </li></ul></ul><ul><ul><li>^C </li></ul></ul><ul><ul><li>WHO WHAT COUNT </li></ul></ul><ul><ul><li>make.bin yacc 1 </li></ul></ul><ul><ul><li>sh sed 4 </li></ul></ul><ul><ul><li>sh tr 4 </li></ul></ul><ul><ul><li>make make.bin 4 </li></ul></ul><ul><ul><li>sh install.bin 5 </li></ul></ul><ul><ul><li>sh rm 6 </li></ul></ul><ul><ul><li>cc ir2hf 33 </li></ul></ul><ul><ul><li>cc ube 33 </li></ul></ul><ul><ul><li>sh date 34 </li></ul></ul><ul><ul><li>sh mcs 34 </li></ul></ul><ul><ul><li>cc acomp 34 </li></ul></ul><ul><ul><li>sh cc 34 </li></ul></ul><ul><ul><li>sh basename 34 </li></ul></ul><ul><ul><li>basename expr 34 </li></ul></ul><ul><ul><li>make.bin sh 87 </li></ul></ul>
  19. 19. DTrace Basics - <ul><li>DTrace Toolkit – Brendan Gregg </li></ul><ul><ul><li>A Useful Place to Start </li></ul></ul><ul><ul><ul><li>Learn the D language </li></ul></ul></ul><ul><ul><ul><li>Initial ideas of what to try </li></ul></ul></ul><ul><ul><ul><li>Reasonably well documented </li></ul></ul></ul><ul><ul><ul><li>Can run into resource limits if you apply them to oracle whole-sale (more later) </li></ul></ul></ul><ul><ul><li>Free </li></ul></ul><ul><ul><li>Constant Development </li></ul></ul><ul><ul><li>Examples will use DTrace Toolkit programs </li></ul></ul>
  20. 20. A Few Oracle Examples <ul><li>Introduction </li></ul><ul><li>The Need for DTrace </li></ul><ul><li>DTrace Basics </li></ul><ul><li>A Few Oracle Examples </li></ul><ul><li>Quirks and Issues </li></ul><ul><li>Conclusions </li></ul>
  21. 21. A Few Oracle Examples <ul><li>iotop –d md0 1 </li></ul><ul><li>iosnoop –p 1885 </li></ul>
  22. 22. A Few Oracle Examples <ul><li>topsyscall </li></ul>
  23. 23. A Few Oracle Examples <ul><li>One for Alex … </li></ul><ul><ul><li>Change Block Tracking </li></ul></ul><ul><ul><li>Find the CTWR process </li></ul></ul><ul><ul><li>ps -ef|grep ctwr |grep -v grep </li></ul></ul><ul><ul><li>oracle 1911 1 0 May 19 ? 2:47 ora_ctwr_TEST1020 </li></ul></ul>
  24. 24. A Few Oracle Examples <ul><li>Traditional approach – truss (or strace) </li></ul>
  25. 25. A Few Oracle Examples <ul><li>dtruss – significantly less overhead </li></ul>
  26. 26. A Few Oracle Examples <ul><li>dapptrace –aFp PID </li></ul>
  27. 27. Quirks and Issues <ul><li>Introduction </li></ul><ul><li>The Need for DTrace </li></ul><ul><li>DTrace Basics </li></ul><ul><li>A Few Oracle Examples </li></ul><ul><li>Quirks and Issues </li></ul><ul><li>Conclusions </li></ul>
  28. 28. Quirks and Issues <ul><li>./dapptrace -p 5728 </li></ul><ul><li>First , think about what you’re doing! </li></ul><ul><li>Edit /kernel/drv/fasttrap.conf </li></ul><ul><li>Default is 250,000 </li></ul><ul><ul><li>Increase to 1,000,000? </li></ul></ul><ul><li>update_drv fasttrap </li></ul><ul><li>Probably need a reboot </li></ul>
  29. 29. Quirks and Issues <ul><li>./dapptrace -p 5728 </li></ul><ul><li>Not zeroing variables when they’re no longer used </li></ul><ul><li>Caused by running out of finite space for variables </li></ul><ul><li>Look at tuning dynvarsize and cleanrate </li></ul>
  30. 30. Quirks and Issues <ul><li>No Overhead? </li></ul><ul><ul><li>When probes are not enabled </li></ul></ul><ul><ul><ul><li>No or very low overhead </li></ul></ul></ul><ul><ul><li>Sampling - A probe event rate of less than 1000 per second </li></ul></ul><ul><ul><ul><li>Likely to have negligible impact. </li></ul></ul></ul><ul><ul><li>Tracing </li></ul></ul><ul><ul><ul><li>“ Slow” disk events – maybe less than 0.2% CPU </li></ul></ul></ul><ul><ul><ul><li>Process creation – closer to 0% </li></ul></ul></ul><ul><ul><ul><li>Every Application Function – maybe 10%? </li></ul></ul></ul><ul><li>Source: DTrace wiki at </li></ul>
  31. 31. Quirks and Issues <ul><li>No Overhead </li></ul><ul><ul><li>A simple example </li></ul></ul><ul><ul><li>Start a sqlplus session and pick up dedicated server process </li></ul></ul><ul><ul><ul><li>dapptrace –p PID </li></ul></ul></ul><ul><ul><ul><li>SELECT sysdate FROM dual; </li></ul></ul></ul><ul><ul><li>Normally sub-second </li></ul></ul><ul><ul><li>With dapptrace, 5 – 20 seconds! </li></ul></ul><ul><li>No overhead? </li></ul>
  32. 32. Quirks and Issues <ul><li>Tracing vs. Sampling </li></ul><ul><ul><li>Hey, it’s the 10046/Statspack debate! </li></ul></ul><ul><li>DTrace can do both – horses for courses </li></ul><ul><li>DTrace can drown you in detail </li></ul><ul><ul><li>Unless you already know what you’re looking for </li></ul></ul><ul><ul><li>Aggregation Functions can also make results more manageable </li></ul></ul>
  33. 33. Conclusions <ul><li>Introduction </li></ul><ul><li>The Need for DTrace </li></ul><ul><li>DTrace Basics </li></ul><ul><li>A Few Oracle Examples </li></ul><ul><li>Quirks and Issues </li></ul><ul><li>Conclusions </li></ul>
  34. 34. Conclusions <ul><li>“ Just tell me the top things to try” </li></ul><ul><li>The Good </li></ul><ul><ul><li>You may get some quick performance wins </li></ul></ul><ul><ul><li>A very practical way to introduce you to DTrace </li></ul></ul><ul><li>The Bad </li></ul><ul><ul><li>Can only introduce you to a fraction of the field of DTrace </li></ul></ul><ul><ul><li>Every site is different, there is no one-size-fits-all set of scripts </li></ul></ul><ul><ul><li>Somewhat ill-conceived; DTrace wins often aren't that quick </li></ul></ul><ul><li>The Ugly </li></ul><ul><ul><li>A customer declaring that &quot;DTrace found nothing&quot; after only trying the quick wins </li></ul></ul>
  35. 35. Conclusions <ul><li>From the DTrace Toolkit FAQ (Docs/Faq) </li></ul><ul><ul><li>1.4. Am I now a performance tuning expert? </li></ul></ul><ul><ul><li>The DTraceToolkit does not turn people into performance tuning experts in the same way that owning a set of golf clubs won't make you a professional golfer. Experience and understanding are necessary. </li></ul></ul>
  36. 36. Conclusions <ul><li>Not the ideal cure for Compulsive Tuning Disorder 1 </li></ul><ul><ul><li>Like presenting an alcoholic with the distillery keys. </li></ul></ul><ul><ul><li>You could spend a lot of time with this. </li></ul></ul><ul><ul><ul><li>If you’re a geek! </li></ul></ul></ul><ul><li>1 Gaja Krishna Vaidyanatha - TOTOT </li></ul>
  37. 37. Conclusions <ul><li>My final and most important conclusion </li></ul><ul><li>Never Trust a Campbell! </li></ul>
  38. 38. Playing around with DTrace (… and Oracle, of course) Doug Burns [email_address]
  39. 39. References and Additional Info <ul><li>Brendan Gregg (including link to DTrace Toolkit) </li></ul><ul><ul><li> </li></ul></ul><ul><li>Solaris Internals - DTrace Topics wiki </li></ul><ul><ul><li> </li></ul></ul><ul><li>Tips, Tricks and Gotchas from DTrace authors </li></ul><ul><ul><li> </li></ul></ul>