DTrace - Miracle Scotland Database Forum

Playing around with DTrace (… and Oracle, of course) Doug Burns [email_address] http://oracledoug.com

Agenda

Introduction

What is DTrace?

DTrace Basics

A Few Oracle Examples

Quirks and Issues

Conclusions

Introduction

Welcome to Scotland

I’m a Campbell

Search Google for “ Campbell Glencoe ”

The picture you are about to see is well-chosen

The Glencoe Massacre of 1692

Introduction

So now you know I am definitely not to be trusted.

I’m a Campbell

I enjoy traditional Highland hospitality …

… and then kill my hosts

However, you may be safe …

I don’t drink Whisky

I don’t wear Skirts!

See how dangerous a simple Tie can be?

What is DTrace?

Introduction

What is DTrace?

DTrace Basics

Quirks and Issues

Conclusions

Similar story to Oracle Wait Statistics

See Juan Loiza quotation in Chapter 2 of TOTOT

A problem with a benchmark

Lots of guesses - none correct

Instrumentation helps - if you thought of it

Sometimes you have to stop to move forward

Sometimes you need to see what’s really going on – even (or particularly) on Production systems

What is DTrace? - History

Dynamic Tracing framework.

Why ‘Dynamic’?

No need to instrument your code

Although you can, by implementing a DTrace provider .

When you hit a performance problem, you can enable probes

Perhaps like ASH but without the constant overhead?

More like dbms_support.start_trace_in_session

What can I trace?

Erm, just about everything

What is DTrace? – The Name

What is DTrace? – The Future

Open Source

Thriving Community

DTrace is available on Solaris 10+

Remember that Solaris != SPARC

FreeBSD and Mac OS X - soon

AIX?

Search Google for ‘AIX 6 Preview’

probevue

Linux?

DTrace Basics

Introduction

The Need for DTrace

DTrace Basics

Quirks and Issues

Conclusions

DTrace Basics – D Language

‘ D’ Scripting Language

A combination of C and awk?

But don’t let that put you off ;-)

Event-driven so you :-

Define the Events (Probes) to trap

Write the handlers

Let’s take a look at a simple example

But ‘Hello World’ is not interesting

DTrace Basics – Defining Probes

Probes can be identified by name or number

A probe name is a 4-tuple

provider:module:function:name

A space or asterisk is a wildcard

Some providers do not provide modules

Examples

pid456:libc:malloc:entry

pid1910:oracle:krccdw:

sched:::enqueue

dtrace:::BEGIN

DTrace Basics – One-Liner

Use any probe definition as a one line program

Enable probes using -n

dtrace -n pid1910:oracle:krccdw:

dtrace: description 'pid1910:oracle:krccdw:' matched 108 probes

CPU ID FUNCTION:NAME

0 66998 krccdw:entry

0 207834 krccdw:0

0 207835 krccdw:1

0 207836 krccdw:3

0 207837 krccdw:6

0 207838 krccdw:9

0 207839 krccdw:a

0 207840 krccdw:b

0 207841 krccdw:c

DTrace Basics – One-Liner

Make a small change in probe definition

dtrace -n ::oracle:krccdw:entry

dtrace: description '::oracle:krccdw:entry' matched 1 probe

CPU ID FUNCTION:NAME

DTrace Basics - Providers

sched provider

Access to O/S scheduling information

Probes which can trap information when processes are placed on to the run queue and when they are dequeued and allowed to run.

sched:::enqueue

sched:::dequeue

Probes that can be used to monitor time spent on the CPU.

sched:::on-cpu

sched:::off-cpu

DTrace Basics - Providers

dtrace

io

pid

Proc

syscall

sysinfo

DTrace Basics – A Script

A more realistic Program Structure

#pragma D option quiet

proc:::exec

{ self->parent = execname;

}

proc:::exec-success

/self->parent != NULL/

{ @[self->parent, execname] = count();

self->parent = NULL;

}

proc:::exec-failure

/self->parent != NULL/

{ self->parent = NULL;

}

END

{ printf("%-20s %-20s %s ", "WHO", "WHAT", "COUNT");

printa("%-20s %-20s %@d ", @);

}

DTrace Basics – A Script

Output

# dtrace -s ./whoexec.d

^C

WHO WHAT COUNT

make.bin yacc 1

sh sed 4

sh tr 4

make make.bin 4

sh install.bin 5

sh rm 6

cc ir2hf 33

cc ube 33

sh date 34

sh mcs 34

cc acomp 34

sh cc 34

sh basename 34

basename expr 34

make.bin sh 87

DTrace Basics -

DTrace Toolkit – Brendan Gregg

A Useful Place to Start

Learn the D language

Initial ideas of what to try

Reasonably well documented

Can run into resource limits if you apply them to oracle whole-sale (more later)

Free

Constant Development

Examples will use DTrace Toolkit programs

Introduction

The Need for DTrace

DTrace Basics

Quirks and Issues

Conclusions

iotop –d md0 1

iosnoop –p 1885

topsyscall

One for Alex …

Change Block Tracking

Find the CTWR process

ps -ef|grep ctwr |grep -v grep

oracle 1911 1 0 May 19 ? 2:47 ora_ctwr_TEST1020

Traditional approach – truss (or strace)

dtruss – significantly less overhead

dapptrace –aFp PID

Quirks and Issues

Introduction

The Need for DTrace

DTrace Basics

Quirks and Issues

Conclusions

Quirks and Issues

./dapptrace -p 5728

First , think about what you’re doing!

Edit /kernel/drv/fasttrap.conf

Default is 250,000

Increase to 1,000,000?

update_drv fasttrap

Probably need a reboot

Quirks and Issues

./dapptrace -p 5728

Not zeroing variables when they’re no longer used

Caused by running out of finite space for variables

Look at tuning dynvarsize and cleanrate

Quirks and Issues

No Overhead?

When probes are not enabled

No or very low overhead

Sampling - A probe event rate of less than 1000 per second

Likely to have negligible impact.

Tracing

“ Slow” disk events – maybe less than 0.2% CPU

Process creation – closer to 0%

Every Application Function – maybe 10%?

Source: DTrace wiki at solarisinternals.com

Quirks and Issues

No Overhead

A simple example

Start a sqlplus session and pick up dedicated server process

dapptrace –p PID

SELECT sysdate FROM dual;

Normally sub-second

With dapptrace, 5 – 20 seconds!

No overhead?

Quirks and Issues

Tracing vs. Sampling

Hey, it’s the 10046/Statspack debate!

DTrace can do both – horses for courses

DTrace can drown you in detail

Unless you already know what you’re looking for

Aggregation Functions can also make results more manageable

Conclusions

Introduction

The Need for DTrace

DTrace Basics

Quirks and Issues

Conclusions

Conclusions

“ Just tell me the top things to try”

The Good

You may get some quick performance wins

A very practical way to introduce you to DTrace

The Bad

Can only introduce you to a fraction of the field of DTrace

Every site is different, there is no one-size-fits-all set of scripts

Somewhat ill-conceived; DTrace wins often aren't that quick

The Ugly

A customer declaring that "DTrace found nothing" after only trying the quick wins

Conclusions

From the DTrace Toolkit FAQ (Docs/Faq)

1.4. Am I now a performance tuning expert?

The DTraceToolkit does not turn people into performance tuning experts in the same way that owning a set of golf clubs won't make you a professional golfer. Experience and understanding are necessary.

Conclusions

Not the ideal cure for Compulsive Tuning Disorder 1

Like presenting an alcoholic with the distillery keys.

You could spend a lot of time with this.

If you’re a geek!

1 Gaja Krishna Vaidyanatha - TOTOT

Conclusions

My final and most important conclusion

Never Trust a Campbell!

Playing around with DTrace (… and Oracle, of course) Doug Burns [email_address] http://oracledoug.com

References and Additional Info

Brendan Gregg (including link to DTrace Toolkit)

http://www.brendangregg.com/dtrace.html

Solaris Internals - DTrace Topics wiki

http://www.solarisinternals.com/wiki/index.php/DTrace_Topics

Tips, Tricks and Gotchas from DTrace authors

http://learningsolaris.com/docs/dtrace_tips_public.pdf

DTrace - Miracle Scotland Database Forum

0 comments

Notes on slide 1

2 Favorites