Your SlideShare is downloading. ×
DTrace - Miracle Scotland Database Forum
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

DTrace - Miracle Scotland Database Forum

8,786
views

Published on

Published in: Technology, Business

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,786
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
119
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Transcript

    • 1. Playing around with DTrace (… and Oracle, of course) Doug Burns [email_address] http://oracledoug.com
    • 2. Agenda
      • Introduction
      • What is DTrace?
      • DTrace Basics
      • A Few Oracle Examples
      • Quirks and Issues
      • Conclusions
    • 3. Introduction
      • Welcome to Scotland
      • I’m a Campbell
      • Search Google for “ Campbell Glencoe ”
      • The picture you are about to see is well-chosen
    • 4. The Glencoe Massacre of 1692
    • 5. Introduction
      • So now you know I am definitely not to be trusted.
        • I’m a Campbell
        • I enjoy traditional Highland hospitality …
        • … and then kill my hosts
      • However, you may be safe …
        • I don’t drink Whisky
        • I don’t wear Skirts!
      • See how dangerous a simple Tie can be?
    • 6. What is DTrace?
      • Introduction
      • What is DTrace?
      • DTrace Basics
      • A Few Oracle Examples
      • Quirks and Issues
      • Conclusions
    • 7.
      • Similar story to Oracle Wait Statistics
        • See Juan Loiza quotation in Chapter 2 of TOTOT
        • A problem with a benchmark
        • Lots of guesses - none correct
        • Instrumentation helps - if you thought of it
        • Sometimes you have to stop to move forward
      • Sometimes you need to see what’s really going on – even (or particularly) on Production systems
      What is DTrace? - History
    • 8.
      • Dynamic Tracing framework.
      • Why ‘Dynamic’?
        • No need to instrument your code
          • Although you can, by implementing a DTrace provider .
        • When you hit a performance problem, you can enable probes
          • Perhaps like ASH but without the constant overhead?
          • More like dbms_support.start_trace_in_session
      • What can I trace?
        • Erm, just about everything
      What is DTrace? – The Name
    • 9. What is DTrace? – The Future
      • Open Source
        • Thriving Community
      • DTrace is available on Solaris 10+
        • Remember that Solaris != SPARC
        • FreeBSD and Mac OS X - soon
        • AIX?
          • Search Google for ‘AIX 6 Preview’
          • probevue
        • Linux?
    • 10. DTrace Basics
      • Introduction
      • The Need for DTrace
      • DTrace Basics
      • A Few Oracle Examples
      • Quirks and Issues
      • Conclusions
    • 11. DTrace Basics – D Language
      • ‘ D’ Scripting Language
      • A combination of C and awk?
        • But don’t let that put you off ;-)
      • Event-driven so you :-
        • Define the Events (Probes) to trap
        • Write the handlers
      • Let’s take a look at a simple example
        • But ‘Hello World’ is not interesting
    • 12. DTrace Basics – Defining Probes
      • Probes can be identified by name or number
      • A probe name is a 4-tuple
        • provider:module:function:name
        • A space or asterisk is a wildcard
        • Some providers do not provide modules
      • Examples
        • pid456:libc:malloc:entry
        • pid1910:oracle:krccdw:
        • sched:::enqueue
        • dtrace:::BEGIN
    • 13. DTrace Basics – One-Liner
      • Use any probe definition as a one line program
        • Enable probes using -n
      • dtrace -n pid1910:oracle:krccdw:
      • dtrace: description 'pid1910:oracle:krccdw:' matched 108 probes
      • CPU ID FUNCTION:NAME
      • 0 66998 krccdw:entry
      • 0 207834 krccdw:0
      • 0 207835 krccdw:1
      • 0 207836 krccdw:3
      • 0 207837 krccdw:6
      • 0 207838 krccdw:9
      • 0 207839 krccdw:a
      • 0 207840 krccdw:b
      • 0 207841 krccdw:c
    • 14. DTrace Basics – One-Liner
      • Make a small change in probe definition
      • dtrace -n ::oracle:krccdw:entry
      • dtrace: description '::oracle:krccdw:entry' matched 1 probe
      • CPU ID FUNCTION:NAME
      • 2 43530 krccdw:entry
      • 3 43530 krccdw:entry
      • 1 43530 krccdw:entry
      • 1 43530 krccdw:entry
      • 1 43530 krccdw:entry
      • 2 43530 krccdw:entry
      • 2 43530 krccdw:entry
      • 2 43530 krccdw:entry
      • 2 43530 krccdw:entry
    • 15. DTrace Basics - Providers
      • sched provider
        • Access to O/S scheduling information
        • Probes which can trap information when processes are placed on to the run queue and when they are dequeued and allowed to run.
          • sched:::enqueue
          • sched:::dequeue
        • Probes that can be used to monitor time spent on the CPU.
          • sched:::on-cpu
          • sched:::off-cpu
    • 16. DTrace Basics - Providers
      • dtrace
      • io
      • pid
      • Proc
      • syscall
      • sysinfo
    • 17. DTrace Basics – A Script
      • A more realistic Program Structure
        • #pragma D option quiet
        • proc:::exec
        • { self->parent = execname;
        • }
        • proc:::exec-success
        • /self->parent != NULL/
        • { @[self->parent, execname] = count();
        • self->parent = NULL;
        • }
        • proc:::exec-failure
        • /self->parent != NULL/
        • { self->parent = NULL;
        • }
        • END
        • { printf("%-20s %-20s %s ", "WHO", "WHAT", "COUNT");
        • printa("%-20s %-20s %@d ", @);
        • }
    • 18. DTrace Basics – A Script
      • Output
        • # dtrace -s ./whoexec.d
        • ^C
        • WHO WHAT COUNT
        • make.bin yacc 1
        • sh sed 4
        • sh tr 4
        • make make.bin 4
        • sh install.bin 5
        • sh rm 6
        • cc ir2hf 33
        • cc ube 33
        • sh date 34
        • sh mcs 34
        • cc acomp 34
        • sh cc 34
        • sh basename 34
        • basename expr 34
        • make.bin sh 87
    • 19. DTrace Basics -
      • DTrace Toolkit – Brendan Gregg
        • A Useful Place to Start
          • Learn the D language
          • Initial ideas of what to try
          • Reasonably well documented
          • Can run into resource limits if you apply them to oracle whole-sale (more later)
        • Free
        • Constant Development
        • Examples will use DTrace Toolkit programs
    • 20. A Few Oracle Examples
      • Introduction
      • The Need for DTrace
      • DTrace Basics
      • A Few Oracle Examples
      • Quirks and Issues
      • Conclusions
    • 21. A Few Oracle Examples
      • iotop –d md0 1
      • iosnoop –p 1885
    • 22. A Few Oracle Examples
      • topsyscall
    • 23. A Few Oracle Examples
      • One for Alex …
        • Change Block Tracking
        • Find the CTWR process
        • ps -ef|grep ctwr |grep -v grep
        • oracle 1911 1 0 May 19 ? 2:47 ora_ctwr_TEST1020
    • 24. A Few Oracle Examples
      • Traditional approach – truss (or strace)
    • 25. A Few Oracle Examples
      • dtruss – significantly less overhead
    • 26. A Few Oracle Examples
      • dapptrace –aFp PID
    • 27. Quirks and Issues
      • Introduction
      • The Need for DTrace
      • DTrace Basics
      • A Few Oracle Examples
      • Quirks and Issues
      • Conclusions
    • 28. Quirks and Issues
      • ./dapptrace -p 5728
      • First , think about what you’re doing!
      • Edit /kernel/drv/fasttrap.conf
      • Default is 250,000
        • Increase to 1,000,000?
      • update_drv fasttrap
      • Probably need a reboot
    • 29. Quirks and Issues
      • ./dapptrace -p 5728
      • Not zeroing variables when they’re no longer used
      • Caused by running out of finite space for variables
      • Look at tuning dynvarsize and cleanrate
    • 30. Quirks and Issues
      • No Overhead?
        • When probes are not enabled
          • No or very low overhead
        • Sampling - A probe event rate of less than 1000 per second
          • Likely to have negligible impact.
        • Tracing
          • “ Slow” disk events – maybe less than 0.2% CPU
          • Process creation – closer to 0%
          • Every Application Function – maybe 10%?
      • Source: DTrace wiki at solarisinternals.com
    • 31. Quirks and Issues
      • No Overhead
        • A simple example
        • Start a sqlplus session and pick up dedicated server process
          • dapptrace –p PID
          • SELECT sysdate FROM dual;
        • Normally sub-second
        • With dapptrace, 5 – 20 seconds!
      • No overhead?
    • 32. Quirks and Issues
      • Tracing vs. Sampling
        • Hey, it’s the 10046/Statspack debate!
      • DTrace can do both – horses for courses
      • DTrace can drown you in detail
        • Unless you already know what you’re looking for
        • Aggregation Functions can also make results more manageable
    • 33. Conclusions
      • Introduction
      • The Need for DTrace
      • DTrace Basics
      • A Few Oracle Examples
      • Quirks and Issues
      • Conclusions
    • 34. Conclusions
      • “ Just tell me the top things to try”
      • The Good
        • You may get some quick performance wins
        • A very practical way to introduce you to DTrace
      • The Bad
        • Can only introduce you to a fraction of the field of DTrace
        • Every site is different, there is no one-size-fits-all set of scripts
        • Somewhat ill-conceived; DTrace wins often aren't that quick
      • The Ugly
        • A customer declaring that "DTrace found nothing" after only trying the quick wins
    • 35. Conclusions
      • From the DTrace Toolkit FAQ (Docs/Faq)
        • 1.4. Am I now a performance tuning expert?
        • The DTraceToolkit does not turn people into performance tuning experts in the same way that owning a set of golf clubs won't make you a professional golfer. Experience and understanding are necessary.
    • 36. Conclusions
      • Not the ideal cure for Compulsive Tuning Disorder 1
        • Like presenting an alcoholic with the distillery keys.
        • You could spend a lot of time with this.
          • If you’re a geek!
      • 1 Gaja Krishna Vaidyanatha - TOTOT
    • 37. Conclusions
      • My final and most important conclusion
      • Never Trust a Campbell!
    • 38. Playing around with DTrace (… and Oracle, of course) Doug Burns [email_address] http://oracledoug.com
    • 39. References and Additional Info
      • Brendan Gregg (including link to DTrace Toolkit)
        • http://www.brendangregg.com/dtrace.html
      • Solaris Internals - DTrace Topics wiki
        • http://www.solarisinternals.com/wiki/index.php/DTrace_Topics
      • Tips, Tricks and Gotchas from DTrace authors
        • http://learningsolaris.com/docs/dtrace_tips_public.pdf