Dynamic Performance Tuning and Troubleshooting With DTrace Ed 2 (Activity Guide)

Dynamic Performance Tuning
and Troubleshooting With
DTrace
Activity Guide
SA-327-S10 Rev B
D61802GC20
Edition 2.0
D65097
Sudaryanto Sudaryanto (sudaryanto@infracom-techฺcom) has a
non-transferable license to use this Student Guideฺ
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Copyright © 2009, 2010, Oracle and/or its affiliates. All rights reserved.
Disclaimer
This document contains proprietary information, is provided under a license agreement containing restrictions on use and
disclosure, and is protected by copyright and other intellectual property laws. You may copy and print this document solely for
your own use in an Oracle training course. The document may not be modified or altered in any way. Except as expressly
permitted in your license agreement or allowed by law, you may not use, share, download, upload, copy, print, display,
perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express
authorization of Oracle.
The information contained in this document is subject to change without notice. If you find any problems in the document,
please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This
document is not warranted to be error-free.
Sun Microsystems, Inc. Disclaimer
This training manual may include references to materials, offerings, or products that were previously offered by Sun
Microsystems, Inc. Certain materials, offerings, services, or products may no longer be offered or provided.Oracle and its
affiliates cannot be held responsible for any such references should they appear in the text provided.
Restricted Rights Notice
If this documentation is delivered to the U.S. Government or anyone using the documentation on behalf of the U.S.
Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS
The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are
restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective
owners.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro
Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used
under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark
licensed through X/Open Company, Ltd.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

This page intentionally left blank.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

v
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision B
Contents
About This Workbook ................................................. Lab Preface-vii
Course Goals......................................................................Lab Preface-vii
Conventions ......................................................................Lab Preface-viii
Typographical Conventions.....................................Lab Preface-viii
DTrace Fundamentals.............................................................. Lab 1-1
Objectives ......................................................................................Lab 1-1
Exercise: Listing Probes and Writing Simple D Scripts................Lab 1-2
Task 1 – Reviewing the Module...........................................Lab 1-2
Task 2 – Listing Probes ........................................................Lab 1-3
Task 3 – Writing D Scripts ...................................................Lab 1-3
Exercise Summary .........................................................................Lab 1-5
Exercise Solutions..........................................................................Lab 1-6
Task 1 – Reviewing the Module...........................................Lab 1-6
Task 2 – Listing Probes ........................................................Lab 1-7
Task 3 – Writing D Scripts ...................................................Lab 1-7
Using DTrace ............................................................................ Lab 2-1
Objectives ......................................................................................Lab 2-1
Exercise: Using vminfo, sysinfo, io, and syscall ProvidersLab 2-2
Task – Writing D Scripts ......................................................Lab 2-2
Task – Writing D Scripts ......................................................Lab 2-5
Finding System Problems With DTrace ................................. Lab 3-1
Objective........................................................................................Lab 3-1
Exercise: Writing D Scripts and Reviewing DTrace Privileges....Lab 3-2
Task 1 – Writing D Scripts to Find System Problems..........Lab 3-2
Task 2 – Reviewing the Least Privilege Facility ................. Lab 3-4
Task 1 – Writing D Scripts to Find System Problems..........Lab 3-6
Task 2 – Reviewing the Least Privilege Facility ............... Lab 3-16
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

vi Troubleshooting to Improve Performance Using DTrace for System Administrators
Troubleshooting DTrace Problems.........................................Lab 4-1
Objectives ......................................................................................Lab 4-1
Exercise: Performance Issues, Buffer Management, and Debugging D
Scripts .........................................................................................Lab 4-2
Task 1 – Reviewing Performance Issues ..............................Lab 4-2
Task 2 – Running Scripts for Buffer Management.............. Lab 4-3
Task 3 – Debugging the DTrace Script ................................Lab 4-3
Task 1 – Reviewing Performance Issues ..............................Lab 4-5
Task 2 – Running Scripts for Buffer Management...............Lab 4-5
Task 3 – Debugging the DTrace Script ................................Lab 4-6
Using DTrace to Explain Behaviors ........................................Lab 5-1
Objectives ......................................................................................Lab 5-1
Introduction....................................................................................Lab 5-2
Exercise 1: fault 2 Using a Lot of CPU Time...........................Lab 5-3
Exercise 2: Disk Busy With Read Traffic .....................................Lab 5-3
Exercise 3: Overhead of Running clock() Function? ..................Lab 5-3
Task 1....................................................................................Lab 5-3
Task 2....................................................................................Lab 5-3
Task 3....................................................................................Lab 5-4
Exercise 4: Too Many fork(2) System Calls?..............................Lab 5-4
Exercise 5: cat Program Hangs....................................................Lab 5-4
Lab ........................................................................................Lab 5-8
Exercise 6: Processes Responsible for Network Traffic................Lab 5-8
Task 1....................................................................................Lab 5-9
Task 2....................................................................................Lab 5-9
Exercise 1 Solution ......................................................................Lab 5-10
Task 1..................................................................................Lab 5-15
Task 2..................................................................................Lab 5-16
Task 3..................................................................................Lab 5-17
Task 1..................................................................................Lab 5-33
Task 2..................................................................................Lab 5-34
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

LabPreface-vii
LabPreface
AboutThisWorkbook
Course Goals
Upon completion of this course, you should be able to:
• Describe DTrace features
• Describe the DTrace architecture
• Describe how DTrace works
• Examine performance problems using DTrace
• Use DTrace to obtain information about systems calls
• Use DTrace to access kernel variables
• Use DTrace to obtain information about I/O
• Use DTrace to do anonymous tracing
• Use DTrace to do speculative tracing
• Explain privileges necessary to run DTrace
• Describe how to lessen the DTrace performance impact
• Describe how to use and tune DTrace buffers
• Debug DTrace scripts
• Describe the DTrace Toolkit
• Describe the most useful DTrace Toolkit scripts
This workbook presents the lab exercises for each module of
the“Troubleshooting to Improve Performance Using DTrace for System
Administrators” Student Guide.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Conventions
Lab Preface-viii Troubleshooting to Improve Performance Using DTrace for System Administrators
Conventions
The following conventions are used in this course to represent various training
elements and alternative learning resources.
Typographical Conventions
Courier is used for the names of commands, files, directories, programming
code, and on-screen computer output; for example:
Use ls -al to list all files.
system% You have mail.
Courier is also used to indicate programming constructs, such as class names,
methods, and keywords; for example:
The getServletInfo method is used to get author information.
The java.awt.Dialog class contains Dialog constructor.
Courier bold is used for characters and numbers that you type; for example:
To list the files in this directory, type:
# ls
Courier bold is also used for each line of programming code that is referenced
in a textual description; for example:
1 import java.io.*;
2 import javax.servlet.*;
3 import javax.servlet.http.*;
Notice the javax.servlet interface is imported to allow access to its life
cycle methods (Line 2).
Courier italics is used for variables and command-line placeholders that are
replaced with a real name or value; for example:
To delete a file, use the rm filename command.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Conventions
About This Workbook Lab Preface-ix
Courier italic bold is used to represent variables whose values are to be
entered by the student as part of an activity; for example:
Type chmod a+rwx filename to grant read, write, and execute rights for
filename to world, group, and users.
Palatino italics is used for book titles, new words or terms, or words that you
want to emphasize; for example:
Read Chapter 6 in the User’s Guide.
These are called class options.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

1-1
Lab 1
DTraceFundamentals
Objectives
Upon completion of this lab, you should be able to:
• Display knowledge of DTrace fundamentals
• List the probes DTrace uses
• Write simple D program scripts
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise: Listing Probes and Writing Simple D Scripts
1-2 Troubleshooting to Improve Performance Using DTrace for System Administrators
In this exercise, you complete the following tasks:
• Answer review questions about the module
• List DTrace probes using various criteria
• Write simple D program scripts
Preparation
Ask your instructor for the root password for your machine.
Task 1 – Reviewing the Module
Answer the following questions to review your understanding of the information
in this module.
1. Describe the main features of DTrace.
_____________________________________________________________
_____________________________________________________________
2. Define a transient failure.
_____________________________________________________________
_____________________________________________________________
3. List some tools that have been used in the past to debug transient failures.
_____________________________________________________________
_____________________________________________________________
4. List some items that can be recorded in an action.
_____________________________________________________________
_____________________________________________________________
5. How do you fully specify a probe?
_____________________________________________________________
_____________________________________________________________
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

DTrace Fundamentals 1-3
6. List the major components of DTrace?
_____________________________________________________________
_____________________________________________________________
7. What dtrace(1M) option allows you to enable all probes from a given
module?
_____________________________________________________________
_____________________________________________________________
8. What are the units of the built-in timestamp D variable?
_____________________________________________________________
_____________________________________________________________
9. What should be the first line of the ds.d script in order to run it as follows:
# ./ds.d
_____________________________________________________________
_____________________________________________________________
Task 2 – Listing Probes
Complete the following steps:
1. Using the dtrace(1M) command, list every probe. How would you count
the number of probes provided by your system?
_____________________________________________________________
_____________________________________________________________
2. Run the dtrace(1M) command to list all probes from the TS module.
_____________________________________________________________
_____________________________________________________________
3. Run a command to list all probes from the lockstat provider.
_____________________________________________________________
_____________________________________________________________
Task 3 – Writing D Scripts
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

1. Write a D script that displays “Hello World.” Run it with and without the
-q option of dtrace(1M).
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
2. Write a D script that displays the PIDs and names of all processes issuing
the kill(2) system call. Start another terminal window and test your script
by starting a few sleep 900 commands in the background and then kill
them with the shell kill pid command and the pkill sleep command.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Summary
Exercise Summary
?
!
Discussion – Take a few minutes to discuss what experiences, issues, or
discoveries you had during the lab exercise.
• Experiences
• Interpretations
• Conclusions
• Applications
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Exercise Solutions
This section provides the answers to the exercise tasks.
Task 1 – Reviewing the Module
Review the following answers:
1. Describe the main features of DTrace.
It enables dynamic modification of the system to record arbitrary data.
DTrace has low overhead which promotes the tracing of production
systems. It is completely safe to use. It can be used on the kernel or
applications.
2. Define a transient failure.
A transient failure is any unacceptable behavior that does not result in fatal
failure of the system.
3. List some tools that have been used in the past to debug transient failures.
truss, TNF, pstack, and prstat
4. List some items that can be recorded in an action.
PID and executable name of the current process, nanoseconds since boot
timestamp, running thread’s priority, and many more.
5. How do you fully specify a probe?
provider:module:function:name
6. List the major components of DTrace.
Probes, providers, consumers, and the D language
7. What dtrace(1M) option allows you to enable all probes from a given
module?
dtrace -m module_name
8. What are the units of the built-in timestamp D variable?
Nanoseconds
9. What should be the first line of the ds.d script to run it as follows:
# ./ds.d
#!/usr/sbin/dtrace -s
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Task 2 – Listing Probes
Review the following solutions:
1. Using the dtrace(1M) command, list every probe. How would you count
the number of probes provided by your system?
# dtrace -l
...
# dtrace -l | wc -l
2. Run the dtrace(1M) command to list all probes from the TS module.
# dtrace -l -m TS
3. Run a command to list all probes from the lockstat provider.
# dtrace -l -P lockstat
Task 3 – Writing D Scripts
1. Write a D script that displays “Hello World.” Run it with and without the
-q option of dtrace(1M).
# cat hello.d
BEGIN
{
trace("Hello Worldn");
}
# dtrace -s hello.d
dtrace: script 'hello.d' matched 1 probe
CPU ID FUNCTION:NAME
0 1 :BEGIN Hello World
^C
# dtrace -q -s hello.d
Hello World
^C
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
2. Write a D script that displays the PIDs and names of all processes issuing
the kill(2) system call. Start another terminal window, and test your script
by starting a few sleep 900 commands in the background and then killing
them with the shell kill pid command or the pkill sleep command.
# cat kill.d
syscall::kill:entry
{
trace(pid);
trace(execname);
}
# chmod +x kill.d
# ./kill.d
dtrace: script './kill.d' matched 1 probe
0 78 kill:entry 5083 bash
0 78 kill:entry 349 utmpd
0 78 kill:entry 5128 pkill
^C
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

2-1
Lab 2
UsingDTrace
Objectives
Upon completion of this lab, you should be able to write D scripts that use the
vminfo, sysinfo, io, and syscall providers.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise: Using vminfo, sysinfo, io, and syscall Providers
Exercise: Using vminfo, sysinfo, io, and syscall
Providers
In this exercise, you write D scripts that use the vminfo, sysinfo, io, and
syscall providers.
Preparation
Ask your instructor for the root password for your machine. Change to the
directory containing the Module 2 lab files. (Ask your instructor for the path
name.)
Task – Writing D Scripts
Write a D script named paging.d that outputs the same information as the pi
and po fields of the vmstat(1M) command. These fields represent the amount of
kilobytes being paged in and paged out per second. Write your script to accept
exactly one argument, which is the interval time in seconds (like the vmstat
command). Use the pgpgin and pgpgout probes with the arg0 argument.
You can test your script by running in another window:
# (find / -name fubar & mkfile 100m /tmp/junk)&
assuming your current system free memory is close to 100M. You can
see your current system free memory by running:
# vmstat 1 2
kthr memory page disk faults
cpu
r b w swap free re mf pi po fr de sr s0 s2 s1 -- in sy cs us
sy id
0 0 0 819472 134816 0 1 0 0 0 0 0 0 0 0 0 414 10 46 0
1 99
0 0 0 815864 102016 11 31 81 0 0 0 0 0 0 0 0 459 98 93 1
3 96
and using the last line of free which is in Kbytes and converting
to Mbytes.
3. Write a D script that displays the total number of cow_fault and
sysfork events that occur every five seconds, to show that when the
number of fork system calls increases so does the number of “copy on
write” faults. Test your script by running many date and sleep 1
commands in the background in another terminal window.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise: Using vminfo, sysinfo, io, and syscall Providers
Using DTrace 2-3
4. Using the io provider probes with the lquantize aggregation function,
write a D script that displays a graph of the time taken in milliseconds for
every device read. Make the scale of the distribution graph range from 0 to
50 milliseconds (ms), in increments of 1ms. Make the “key” for the
aggregation be the literal string: “Read elapsed time:”.
a. Test your script by running the following command in another
terminal window: grep fubar /usr/share/man/sman1/*.
b. Run the iosnoop.d script (with another similar grep command,
grep fubar /usr/demo/dtrace/*) to verify that most of the
reads are under 1ms. (Note: because of file caching you only get one
try. If you do not see the grep commands or times under 1ms in the
iosnoop.d output try another directory with many unread text files
such as /usr/lib/adb/sparcv9/*).
5. Look at the /usr/demo/dtrace/dateprof.d script. Read the
description of the vtimestamp built-in variable in Appendix B. Start the
following sh(1) command in one window:
while : ; do date; done
while you run the dateprof.d script in another window. Interrupt both
commands with <Ctrl>C after 10 seconds.
6. Rewrite the timesys.d D script shown on page 2-47 in your student guide
so that it accepts the executable command name as an argument instead of
only working with the grep command. Test your script with an ls
command that you enter in another terminal window.
7. Write a pagefault.d D script that follows all the functions used in
handling a page fault. Make it trace starting with the kernel function:
pagefault(). Invoke the script with the -F option of the dtrace(1M)
command.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Summary
Exercise Summary
?
!
Discussion – Take a few minutes to discuss the experiences you had during the
lab exercise, and any issues or discoveries that arose.
• Experiences
• Interpretations
• Conclusions
• Applications
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Using DTrace 2-5
Exercise Solutions
Task – Writing D Scripts
1. Write a D script named paging.d that outputs the same information as the
pi and po fields of the vmstat(1M) command. These fields represent the
amount of kilobytes being paged in and paged out per second. Write your
script to accept exactly one argument, which is the interval time in seconds
(like the vmstat command). Use the pgpgin and pgpgout probes with
the arg0 argument.
# cat -n paging.d
1 #!/usr/sbin/dtrace -qs
2 /*
3 * Usage: ./paging.d interval
4 */
5 BEGIN
6 {
7 printf("%8s %8sn", "pi", "po");
8 i = 0;
9 po = 0;
10 pi = 0;
11 }
12
13 tick-1sec
14 {
15 ++i;
16 }
17
18 vminfo:::pgpgin
19 {
20 pi = pi + arg0;
21 }
22
23 vminfo:::pgpgout
24 {
25 po = po + arg0;
26 }
27
28 tick-1sec
29 /i == $1/
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
30 {
31 printf("%8d %8dn", (pi*8)/i, (po*8)/i);
32 i = 0;
33 pi = 0;
34 po = 0;
35 }
You can test your script by running in another window:
# (find / -name fubar & mkfile 100m /tmp/junk)&
assuming your current system free memory is close to 100M. You can
see your current system free memory by running:
# vmstat 1 2
kthr memory page disk faults
cpu
r b w swap free re mf pi po fr de sr s0 s2 s1 -- in sy cs us
sy id
0 0 0 819472 134816 0 1 0 0 0 0 0 0 0 0 0 414 10 46 0
1 99
0 0 0 815864 102016 11 31 81 0 0 0 0 0 0 0 0 459 98 93 1
3 96
and using the last line of free which is in Kbytes and converting
to Mbytes.
# ./paging.d 5
pi po
0 0
20 11448
0 1126
771 0
51 0
^C
2. Write a D script that displays the total number of cow_fault and
sysfork events that occur every five seconds, to show that when the
number of fork system calls increases so does the number of “copy on
write” faults. Test your script by running many date and sleep 1
commands in the background in another terminal window.
# cat -n cow.d
2
3 BEGIN
4 {
5 printf("%6s %8sn", "cows", "forks");
6 }
7
8 vminfo:::cow_fault
9 {
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Using DTrace 2-7
10 ++c;
11 }
12
13 sysinfo:::sysfork
14 {
15 ++f;
16 }
17
18 tick-5sec
19 {
20 printf("%6d %8dn", c, f);
21 c = 0;
22 f = 0;
23 }
# ./cow.d
cows forks
198 9
66 3
16 1
0 0
465 21
0 0
^C
3. Using the io provider probes with the lquantize aggregation function,
write a D script that displays a graph of the time taken in milliseconds for
every device read. Make the scale of the distribution graph range from 0 to
50 milliseconds (ms), in increments of 1ms. Make the “key” for the
aggregation be the literal string: “Read elapsed time:”.
a. Test your script by running the following command in another
terminal window: grep fubar /usr/share/man/sman1/*.
# cat -n io.d
2
3 io:::start
4 / args[0]->b_flags&B_READ /
5 {
6 start[args[0]->b_edev, args[0]->b_blkno] = timestamp;
7 }
8
9 io:::done
10 /start[args[0]->b_edev, args[0]->b_blkno]/
11 {
12 elapsed = (timestamp - start[args[0]->b_edev, args[0]->b_blkno])/1000000;
13 @["Read elapsed time:"] = lquantize(elapsed,0,50,1);
14 }
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
# ./io.d
^C
Read elapsed time:
value ------------- Distribution ------------- count
< 0 | 0
0 |@@@@@@@@@@@@@@@@ 101
1 |@ 9
2 |@@ 10
3 |@@ 12
4 |@@@ 17
5 |@@@ 18
6 |@@@@@ 33
7 |@@@ 16
8 |@@@ 16
9 |@ 5
10 |@ 5
11 | 1
12 | 0
13 | 1
14 | 2
15 | 0
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Using DTrace 2-9
b. Run the iosnoop.d script (with another similar grep command,
grep fubar /usr/demo/dtrace/*) to verify that most of the
reads are under 1ms. (Note: because of file caching you only get one
try. If you do not see the grep commands or times under 1ms in the
iosnoop.d output try another directory with many unread text files
such as /usr/lib/adb/sparcv9/*).
# iosnoop.d
COMMAND PID FILE DEVICE RW MS
bash 540 <unknown> sd2 R 6.859
bash 540 /usr/demo/dtrace sd2 R 4.286
grep 564 /usr/demo/dtrace/applicat.d sd2 R 0.346
grep 564 /usr/demo/dtrace/badopen.d sd2 R 0.296
grep 564 /usr/demo/dtrace/begin.d sd2 R 0.295
grep 564 /usr/demo/dtrace/callout.d sd2 R 0.489
grep 564 /usr/demo/dtrace/clause.d sd2 R 0.324
grep 564 /usr/demo/dtrace/clear.d sd2 R 0.291
grep 564 /usr/demo/dtrace/countdown.d sd2 R 0.274
grep 564 /usr/demo/dtrace/counter.d sd2 R 0.288
grep 564 /usr/demo/dtrace/dateprof.d sd2 R 0.309
grep 564 /usr/demo/dtrace/delay.d sd2 R 0.273
grep 564 /usr/demo/dtrace/denorm.d sd2 R 0.282
grep 564 /usr/demo/dtrace/end.d sd2 R 0.286
grep 564 /usr/demo/dtrace/error.d sd2 R 0.284
grep 564 /usr/demo/dtrace/errorpath.d sd2 R 0.278
grep 564 /usr/demo/dtrace/find.d sd2 R 0.287
grep 564 /usr/demo/dtrace/firebird.d sd2 R 0.313
...
grep 564 /usr/demo/dtrace/lquantize.d sd2 R 0.287
grep 564 /usr/demo/dtrace/lwptime.d sd2 R 0.273
grep 564 /usr/demo/dtrace/normalize.d sd2 R 0.273
grep 564 /usr/demo/dtrace/nscd.d sd2 R 0.313
grep 564 /usr/demo/dtrace/pri.d sd2 R 0.285
grep 564 /usr/demo/dtrace/printa.d sd2 R 0.327
grep 564 /usr/demo/dtrace/pritime.d sd2 R 0.285
grep 564 /usr/demo/dtrace/prof.d sd2 R 0.282
grep 564 /usr/demo/dtrace/profpri.d sd2 R 0.276
grep 564 /usr/demo/dtrace/progtime.d sd2 R 0.285
grep 564 /usr/demo/dtrace/putnext.d sd2 R 0.292
grep 564 /usr/demo/dtrace/qlen.d sd2 R 0.283
grep 564 /usr/demo/dtrace/qtime.d sd2 R 0.283
grep 564 /usr/demo/dtrace/renormalize.d sd2 R 0.286
grep 564 /usr/demo/dtrace/restest.d sd2 R 0.309
grep 564 /usr/demo/dtrace/ring.d sd2 R 0.284
grep 564 /usr/demo/dtrace/rtime.d sd2 R 0.287
grep 564 /usr/demo/dtrace/rwinfo.d sd2 R 0.310
grep 564 /usr/demo/dtrace/rwtime.d sd2 R 0.287
grep 564 /usr/demo/dtrace/sig.d sd2 R 0.278
grep 564 /usr/demo/dtrace/soffice.d sd2 R 0.309
grep 564 /usr/demo/dtrace/spec.d sd2 R 0.273
...
^C
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
4. Look at the /usr/demo/dtrace/dateprof.d script. Read the
description of the vtimestamp built-in variable in Appendix B. Start the
following sh(1) command in one window:
while : ; do date; done
while you run the dateprof.d script in another window. Interrupt both
commands with <Ctrl>C after 10 seconds.
# while : ; do date; done
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
Wed Mar 2 20:02:59 MST 2005
^C
In the other window:
# dtrace -s dateprof.d
dtrace: script 'dateprof.d' matched 228 probes
^C
system calls over time
0 | 0
100 | 2
200 |@@ 1009
300 |@@ 945
400 |@ 891
500 |@@ 1172
600 |@@ 1030
700 |@ 588
800 | 163
900 |@ 874
1000 | 98
1100 | 135
1200 |@ 364
1300 | 74
...
6800 | 241
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Using DTrace 2-11
6900 |@ 389
7000 |@ 461
7100 |@ 418
7200 |@ 343
7300 | 262
7400 | 107
7500 | 67
7600 | 31
7700 | 14
7800 | 6
7900 | 5
8000 | 5
8100 | 8
8200 | 6
8300 | 4
8400 | 2
...
5. Re-write the timesys.d D script shown on page 2-47 in your student
guide so that it accepts the executable command name as an argument
instead of only working with the grep command. Test your script with an
ls command that you enter in another terminal window.
# cat -n timesys2.d
2 BEGIN
3 {
4 printf("nSystem Call Times for %s:nn", $$1);
5 printf("%20st%10sn", "Syscall", "Microseconds");
6 }
7
8 syscall:::entry
9 /execname == $$1/
10 {
11 self->name[probefunc] = timestamp;
12 self->start = 1;
13 }
14
15 syscall:::return
16 /self->start/
17 {
18 printf("%20st%10dn", probefunc, (timestamp - self-
>name[probefunc])/1000);
19 self->start = 0;
20 }
21
22 syscall::rexit:entry
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
23 /execname == $$1/
24 {
25 exit(0);
26 }
# ./timesys2.d ls
System Call Times for ls:
Syscall Microseconds
mmap 49
resolvepath 45
resolvepath 63
stat 39
open 53
stat 33
open 30
mmap 37
...
setcontext 34
getrlimit 23
getpid 17
setcontext 23
brk 23
brk 27
stat 45
gtime 20
ioctl 76
brk 19
...
write 69
write 67
write 68
write 94
write 68
write 66
write 65
write 66
write 65
#
6. Write a pagefault.d D script that follows all the functions used in
handling a page fault. Have it trace starting with the kernel function:
pagefault(). Invoke the script with the -F option of the dtrace(1M)
command.
# cat -n pagefault.d
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Using DTrace 2-13
1 #!/usr/sbin/dtrace -s
2
3 fbt::pagefault:entry
4 {
5 self->start = 1;
6 }
7
8 fbt::pagefault:return
9 /self->start/
10 {
11 exit(0);
12 }
13
14 fbt:::
15 /self->start/
16 {
17 }
# dtrace -F -s pagefault.d
dtrace: script 'pagefault.d' matched 38108 probes
CPU FUNCTION
0 -> pagefault
0 -> as_fault
0 -> as_segat
0 -> avl_find
0 -> as_segcompar
0 <- as_segcompar
...
0 -> fop_getpage
0 -> ufs_getpage
0 -> ufs_lockfs_begin_getpage
0 -> tsd_get
0 <- tsd_get
0 -> tsd_agent_get
0 <- tsd_agent_get
0 -> ufs_lockfs_is_under_rawlockfs
0 -> mutex_owner
0 <- mutex_owner
0 <- ufs_lockfs_is_under_rawlockfs
0 <- ufs_lockfs_begin_getpage
0 -> rw_owner
0 <- rw_owner
...
0 <- page_lookup
0 -> page_lookup_create
0 -> page_try_reclaim_lock
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
0 <- page_try_reclaim_lock
0 -> page_reclaim
0 -> page_list_sub
0 -> page_sub
0 <- page_sub
0 -> page_ctr_sub
0 <- page_ctr_sub
...
0 <- sfmmu_select_tsb_szc
0 -> sfmmu_hat_exit
0 <- sfmmu_hat_exit
0 <- sfmmu_check_page_sizes
0 <- hat_memload
0 -> page_unlock
0 <- page_unlock
0 <- segvn_faultpage
0 <- segvn_fault
0 <- as_fault
0 <- pagefault
#
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

3-1
Lab 3
FindingSystemProblemsWithDTrace
Objective
• Write D scripts to support finding system problems
• Display knowledge of the Least Privilege facility
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise: Writing D Scripts and Reviewing DTrace Privileges
Exercise: Writing D Scripts and Reviewing DTrace
Privileges
In this exercise, you perform the following tasks:
• Write D scripts to support finding system problems
• Review the Least Privilege facility
Preparation
Ask your instructor for the root password for your machine.
Task 1 – Writing D Scripts to Find System Problems
Write the following scripts:
1. Write a D script that displays the value of the global kernel integer
desscan each time the schedpaging() kernel function returns. After
starting your script, enter the following commands in another terminal
window. Use the amount of system free memory in the mkfile command.
If the mkfile command does not terminate after 20 seconds, interrupt the
command with <Ctrl>C.
# vmstat 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr f0 s0 s2 s1 in sy cs us sy id
0 0 0 479112 261072 2 14 33 0 0 0 8 0 0 4 0 421 64 47 23 1 76
0 0 60 548760 368000 1 16 77 0 0 0 0 0 0 4 0 426 54 64 0 1 99
0 0 60 548728 367968 0 2 0 0 0 0 0 0 0 0 0 402 32 38 0 0 100
0 0 60 548712 367952 0 1 0 0 0 0 0 0 0 0 0 401 32 40 0 0 99
0 0 60 548712 367952 0 0 0 0 0 0 0 0 0 0 0 402 36 43 0 0 100
^C
# mkfile 367952k /tmp/junk
^C
# rm /tmp/junk
2. Read the description of the readch sysinfo probe in Table 2-3 of Module
2. Write a D script that displays the number of new readch events every
five seconds. Make your script confirm that readch tracks the number of
successful returns from the read(2), readv(2), and pread(2) system calls.
Note that there is a pread64 system call. Test your script by running the
following pipeline in another sh(1) terminal window (enter a <Return>
after the ending | on the first line). Terminate both commands with
<Ctrl>C after 40 seconds.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Finding System Problems With DTrace 3-3
# find / -type f | xargs file | grep text | cut -f1 -d: |
> xargs ls -l | awk '$5 > 100000 {print $NF}’
3. Write a D script that uses the start probe of the io provider to trace all
reads whose file offset is greater than 100000. Have it display the
executable name, path name, and file offset value. Test your script by
running the same pipeline used in question 2. Terminate both commands
with <Ctrl>C after one minute.
4. Write a D script to be used with anonymous tracing that performs the
following: on entry to the kernel function pageout_scanner(), have it
print the kernel fastscan and pageout_sample_lim integer variables.
Then, at every 1/10th of a second for 30 iterations, make it print the value
of the pageout_sample_cnt kernel variable. After the 30 iterations,
make the script print fastscan again and exit.
5. Write a D script to be used with anonymous tracing that prints the
timestamp built-in variable when the following kernel functions are
entered: main(), sched(), pageout(), and fsflush(). Also make it print
the timestamp variable when the init(1M) command issues the
umask(2) system call.
6. Write a D script to be used with speculative tracing that displays the entire
system call code path any time any system call fails with an errno code of
EINVAL. Add the following D statement, which increases the number of
speculative buffers, near the top of your D script:
#pragma D option nspec=200
Test your script by issuing the following command in another terminal
window: telnet localhost. Have the script exit after it issues one
commit().
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Task 2 – Reviewing the Least Privilege Facility
Answer the following questions:
1. What is the least privilege required to use the syscall provider on your
own processes?
_____________________________________________________________
2. What is the least privilege required to use the pid provider on your own
processes?
_____________________________________________________________
3. What privilege is required to use the stop() built-in function on your own
processes?
_____________________________________________________________
4. What privilege is required to use the panic() function?
_____________________________________________________________
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Summary
Exercise Summary
?
!
Discussion – Take a few minutes to discuss what experiences, issues, or
discoveries you had during the lab exercise.
• Experiences
• Interpretations
• Conclusions
• Applications
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Exercise Solutions
Task 1 – Writing D Scripts to Find System Problems
1. Write a D script that displays the value of the global kernel integer
desscan each time the schedpaging() kernel function returns. After
starting your script, enter the following commands in another terminal
window. Use the amount of system free memory in the mkfile command.
If the mkfile command does not terminate after 20 seconds, interrupt the
command with <Ctrl>C.
# vmstat 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr f0 s0 s2 s1 in sy cs us sy id
0 0 0 479112 261072 2 14 33 0 0 0 8 0 0 4 0 421 64 47 23 1 76
0 0 60 548760 368000 1 16 77 0 0 0 0 0 0 4 0 426 54 64 0 1 99
0 0 60 548728 367968 0 2 0 0 0 0 0 0 0 0 0 402 32 38 0 0 100
0 0 60 548712 367952 0 1 0 0 0 0 0 0 0 0 0 401 32 40 0 0 99
0 0 60 548712 367952 0 0 0 0 0 0 0 0 0 0 0 402 36 43 0 0 100
^C
# mkfile 367952k /tmp/junk
^C
# rm /tmp/junk
# cat desscan.d
#!/usr/sbin/dtrace -qs
fbt::schedpaging:return
{
printf("%dn", `desscan);
}
# ./desscan.d
25
25
25
...
7920
7920
7920
...
4834
3661
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
2521
2042
1285
806
25
25
^C
2. Read the description of the readch sysinfo probe in Table 2-3 of Module
2. Write a D script that displays the number of new readch events every
five seconds. Make your script confirm that readch tracks the number of
successful returns from the read(2), readv(2), and pread(2) system calls.
Note that there is a pread64 system call. Test your script by running the
following pipeline in another sh(1) terminal window (enter a <Return>
after the ending | on the first line). Terminate both commands with
<Ctrl>C after 40 seconds.
# find / -type f | xargs file | grep text | cut -f1 -d: |
> xargs ls -l | awk '$5 > 100000 {print $NF}’
# cat -n reads2.d
2
3 BEGIN
4 {
5 rc = 0;
6 sysreads = 0;
7 }
8
9 sysinfo:::readch
10 {
11 ++rc;
12 }
13
14 syscall::read:return, syscall::readv:return,
syscall::pread*:return
15 /arg0 != -1/
16 {
17 ++sysreads;
18 }
19
20 tick-5sec
21 {
22 printf("%d %dn", rc, sysreads);
23 rc = 0;
24 sysreads = 0;
25 }
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
# ./reads2.d
1793 1789
2141 2141
1677 1677
1523 1523
1115 1115
1350 1350
1719 1719
^C
3. Write a D script that uses the start probe of the io provider to trace all
reads whose file offset is greater than 100000. Make it display the
executable name, path name, and file offset value. Test your script by
running the same pipeline used in question 2. Terminate both commands
with <Ctrl>C after one minute.
# cat offset.d
#!/usr/sbin/dtrace -qs
io:::start
/args[0]->b_flags&B_READ && args[2]->fi_offset > 100000/
{
printf("%s %s %dn", execname, args[2]->fi_pathname,
args[2]->fi_offset);
}
# ./offset.d
ls /lib/libc.so.1 884736
dtrace /lib/sparcv9/libc.so.1 884736
xargs /lib/ld.so.1 172032
file /lib/libelf.so.1 122880
file /lib/libc.so.1 884736
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
^C
4. Write a D script to be used with anonymous tracing to perform the
following: On entry to the kernel function pageout_scanner(), make it
print the kernel fastscan and pageout_sample_lim variables. Then, at
every 1/10th of a second for 30 iterations, make it print the value of the
pageout_sample_cnt kernel variable. After the 30 iterations, make the
script print fastscan again and exit.
# cat -n fastscan.d
2
3 fbt::pageout_scanner:entry
4 {
5 printf("nfastscan at start: %dn", `fastscan);
6 printf("pageout_sample_lim: %dn", `pageout_sample_lim);
7 started = 1;
8 }
9
10 tick-100ms
11 /started/
12 {
13 printf("pageout_sample_cnt: %dn", `pageout_sample_cnt);
14 ++n;
15 }
16
17 tick-100ms
18 /n > 29/
19 {
20 printf("nfastscan: %dn", `fastscan);
21 exit(0);
22 }
# dtrace -As fastscan.d
dtrace: saved anonymous enabling in /kernel/drv/dtrace.conf
dtrace: added forceload directives to /etc/system
dtrace: run update_drv(1M) or reboot to enable changes
sys61# reboot
...
# grep enabling /var/adm/messages
...
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Feb 27 14:20:52 sys63 dtrace: [ID 566105 kern.notice] NOTICE: enabling
probe 0 (fbt::pageout_scanner:entry)
probe 1 (:::tick-100ms)
probe 2 (:::tick-100ms)
probe 3 (dtrace:::ERROR)
# dtrace -ae
0 7067 pageout_scanner:entry
fastscan at start: 8192
pageout_sample_lim: 4
0 27 :tick-100ms pageout_sample_cnt: 0
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
0 27 :tick-100ms
fastscan: 31682
5. Write a D script to be used with anonymous tracing that prints the
timestamp built-in variable when the following kernel functions are
entered: main(), sched(), pageout(), and fsflush(). Also make it print
the timestamp variable when the init(1M) command issues the
umask(2) system call.
The main() probe will never fire because DTrace does not start before you enter
main().
# cat -n start0123.d
2
3 fbt::main:entry
4 {
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
5 printf("nmain: %dn", timestamp);
6 }
7
8 fbt::sched:entry
9 {
10 printf("sched: %dn", timestamp);
11 }
12
13 fbt::pageout:entry
14 {
15 printf("pageout: %dn", timestamp);
16 }
17
18 fbt::fsflush:entry
19 {
20 printf("fsflush: %dn", timestamp);
21 }
22
23 syscall::umask:entry
24 /pid == 1/
25 {
26 printf("init umask(): %dn", timestamp);
27 }
# dtrace -As start0123.d
dtrace: cleaned up old anonymous enabling in /kernel/drv/dtrace.conf
dtrace: cleaned up forceload directives in /etc/system
dtrace: saved anonymous enabling in /kernel/drv/dtrace.conf
dtrace: added forceload directives to /etc/system
dtrace: run update_drv(1M) or reboot to enable changes
# reboot
...
# grep enabling /var/adm/messages
...
probe 0 (fbt::main:entry)
probe 1 (fbt::sched:entry)
probe 2 (fbt::pageout:entry)
probe 3 (fbt::fsflush:entry)
probe 4 (syscall::umask:entry)
probe 5 (dtrace:::ERROR)
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
# dtrace -ae
0 8048 sched:entry sched: 3049104719
0 10534 pageout:entry pageout: 3049163676
0 14018 fsflush:entry fsflush: 3050390489
0 129 umask:entry init umask(): 4705474648
The previous output indicates that process ID 1 (init) starts about
1.6 seconds after process ID 3 (fsflush)
6. Write a D script to be used with speculative tracing that displays the entire
system call code path any time any system call fails with an errno code of
EINVAL. Add the following statement, which increases the number of
speculative buffers, near the top of your D script:
Test your script by issuing the following command in another terminal
window: telnet localhost. Make the script exit after it issues one
commit() command.
# cat -n einvalspec.d
2
3 #pragma D option flowindent
4 #pragma D option nspec=200
5
6 syscall:::entry
7 {
8 self->spec = speculation();
9 speculate(self->spec);
10 }
11
12 fbt:::
13 /self->spec/
14 {
15 speculate(self->spec); /* default action */
16 }
17
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
18 syscall:::return
19 /self->spec && arg0 != -1/
20 {
21 discard(self->spec);
22 self->spec = 0;
23 }
24
25 syscall:::return
26 /self->spec && arg0 == -1 && errno == EINVAL/
27 {
28 commit(self->spec);
29 committed = 1;
30 }
31
32 syscall:::return
33 /committed/
34 {
35 exit(0);
36 }
37
38 syscall:::return
39 /self->spec && arg0 == -1 && errno != EINVAL/
40 {
41 discard(self->spec);
42 self->spec = 0;
43 }
# ./einvalspec.d
dtrace: script './einvalspec.d' matched 33587 probes
CPU FUNCTION
0 -> fcntl
0 -> getf
0 -> set_active_fd
0 <- set_active_fd
0 <- getf
0 -> fd_too_big
0 -> rctl_action
0 <- rctl_action
0 -> rctl_action_entity
0 -> rctl_set_find
0 <- rctl_set_find
0 -> rctl_entity_obtain_entity_p
0 <- rctl_entity_obtain_entity_p
0 -> rctl_global_action
0 <- rctl_global_action
0 -> rcop_no_action
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
0 <- rcop_no_action
0 -> rctl_local_action
0 <- rctl_local_action
0 <- rctl_action_entity
0 <- fd_too_big
0 -> releasef
0 -> clear_active_fd
0 <- clear_active_fd
0 -> cv_broadcast
0 <- cv_broadcast
0 <- releasef
0 -> set_errno
0 <- set_errno
0 <- fcntl
0 <= fcntl
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Task 2 – Reviewing the Least Privilege Facility
1. What is the least privilege required to use the syscall provider on your
own processes?
dtrace_user
2. What is the least privilege required to use the pid provider on your own
processes?
dtrace_proc
3. What privilege is required to use the stop() built-in function on your own
process?
dtrace_proc
4. What privilege is required to use the panic() function?
Must be super-user
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

4-1
Lab 4
TroubleshootingDTraceProblems
Objectives
• Display knowledge regarding DTrace performance issues
• Run scripts related to buffer management
• Debug D scripts
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise: Performance Issues, Buffer Management, and Debugging D Scripts
Exercise: Performance Issues, Buffer Management, and
Debugging D Scripts
In this exercise, you complete the following tasks:
• Answer questions regarding DTrace performance issues
• Run scripts related to buffer management
• Debug D scripts
Preparation
Ask your instructor for the root password for your machine. Change to the
directory containing the Module 5 lab files. (Ask your instructor for the path
name.)
Task 1 – Reviewing Performance Issues
Answer the following questions:
1. The two scripts shown under the section “Using Aggregations” in Module 5
are not equivalent. Describe how the two scripts are different.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
2. How can you change the following predicate to make it cacheable?
/curpsinfo->pr_fname == "init"/
_____________________________________________________________
_____________________________________________________________
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise: Performance Issues, Buffer Management, and Debugging D Scripts
Troubleshooting DTrace Problems 4-3
Task 2 – Running Scripts for Buffer Management
Perform the following tasks:
1. Invoke the dtrace(1M) command to run the stress.d script with the
fill buffer policy set and no other buffer management options set.
Redirect the output of dtrace to a file. What size is the file when dtrace
exits?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
ring buffer policy set and a buffer size of 100 Kbytes. Redirect the output
of dtrace to a file. What is the size of the output file?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
Task 3 – Debugging the DTrace Script
Perform the following tasks:
1. Run the script1.d D script. Correct the syntax error.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
2. Explain what the script2.d D script does.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
3. The timereads.d D script does not, in general, produce correct read
times. Fix the script so that the times are correct.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Summary
Exercise Summary
?
!
Discussion – Take a few minutes to discuss the experiences you had during the
lab exercise, and any issues or discoveries that arose.
• Experiences
• Interpretations
• Conclusions
• Applications
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
Troubleshooting DTrace Problems 4-5
Exercise Solutions
Task 1 – Reviewing Performance Issues
1. The two scripts shown under the section “Using Aggregations” in Module 5
are not equivalent. Describe how the two scripts are different.
With the associative array script, you do not get counts of the system calls
for processes that do not exit, such as daemon processes. Also, if a process
is started and exits multiple times, you get separate counts for each instance
with the associative array script as opposed to one total count with the
aggregation script.
2. How can you change the following predicate to make it cacheable?
/curpsinfo->pr_fname == "init"/
You can change the predicate as follows:
/pid == 1/
or
/execname == “init”/
Task 2 – Running Scripts for Buffer Management
fill buffer policy set and no other buffer management options set.
Redirect the output of dtrace to a file. What size is the file when dtrace
exits?
# dtrace -x bufpolicy=fill -s stress.d>/var/tmp/stress.out
dtrace: script 'stress.d' matched 32631 probes
# ls -l /var/tmp/stress.out
-rw-r--r-- 1 root root 6068875 Mar 14 09:11 /var/tmp/stress.out
2. Invoke dtrace(1M) to run the stress.d script with the ring buffer
policy set and a buffer size of 100Kbytes. Redirect the output of dtrace to
a file. Let dtrace run for one minute before you abort it. What is the size
of the output file?
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise Solutions
# dtrace -x bufpolicy=ring -b 100k -s stress.d>/var/tmp/stress.out
dtrace: script 'stress.d' matched 32631 probes
# ls -l /var/tmp/stress.out
-rw-r--r-- 1 root root 396828 Mar 14 09:14 /var/tmp/stress.out
Task 3 – Debugging the DTrace Script
1. Run the script1.d D script. Correct the syntax error.
Add a semicolon to the end of the printf(...);
2. Explain what the script2.d D script does.
It displays the average write(2) size per executable.
3. The timereads.d D script does not, in general, produce correct read
times. Fix the script so the times are correct.
# cat -n timereads2.d
2
3 syscall::read:entry
4 {
5 self->start = timestamp;
6 }
7
8 syscall::read:return
9 /self->start/
10 {
11 printf("%s read took %dusn", execname, (timestamp - self-
>start)/1000);
12 self->start = 0;
13 }
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

5-1
Lab 5
UsingDTracetoExplainBehaviors
Objectives
• Use Dtrace, and other tools when needed, to explain application and kernel
behavior.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Introduction
Introduction
In Lab 5, “Using DTrace to Explain Behaviors”, you will run applications and use
kernel modules. While the application, kernel module, or both, are running, you
will use DTrace, and other tools where needed, to explain behaviors you see.
Note – Source code for the applications or kernel module isn’t available.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 1: fault 2 Using a Lot of CPU Time
Using DTrace to Explain Behaviors 5-3
Exercise 1: fault 2 Using a Lot of CPU Time
Your customer is running an application (fault2). fault2 is using a lot of CPU
time (as shown by prstat). Determine why this is happening. To run fault2,
enter the following:
# ./fault2 &
9718
Exercise 2: Disk Busy With Read Traffic
The customer runs iostat(1M) and notices that one of the disks is busy, mostly
with read traffic. The customer wants to know which process is doing the reads,
and whether the process is doing sequential or random accesses.
Exercise 3: Overhead of Running clock() Function?
Task 1
Your boss was reading about Solaris. She read that there is a function, called
clock(), in the kernel that runs every 10 milliseconds. The clock() function
keeps track of time in a system. Your boss wants to know what is the overhead of
running this function every 10 milliseconds. She wants to know the minimum,
maximum, and average time spent in the clock() routine.
Task 2
Your boss looks at the times and notices that the minimum and average times are
low, but the maximum time is quite a bit longer. You need to show her the
distributions of timings.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 4: Too Many fork(2) System Calls?
Task 3
Your boss understands the distributions, but now wants to know why clock()
takes longer. To find out, use speculative tracing, and only commit if the time
spent by clock is longer than the time you’ve seen so far. Trace the function
entries and returns made during clock() processing. Don’t worry if you can’t
explain what the routines do. Some of the routine functioning will be obvious
based on the routine name.
Exercise 4: Too Many fork(2) System Calls?
A user noted that when the ps(1) command is used 2 or 3 times in a row, the ps
PID is much higher each time. It appears that something is doing lots of fork(2)
system calls. In addition, ps -elf shows processes that are there (or not) for
very short times. Find out what is causing the fork(2) calls.
Exercise 5: cat Program Hangs
Your customer installed a new device driver on a system. Out of curiosity, they
tried to cat the device file. The cat program hung and cannot be killed without
a reboot.
1. Use DTrace to identify how the problem occurs.
2. To understand why cat hangs, use mdb(1) to examine the thread, or
threads, that are hanging.
3. Run the cat program with the device file as an argument.
# cat /devices/pseudo/semex@0:semex
cat is hung.
1. Try control-c or kill -9 on the pid for cat.
2. Start mdb on the running kernel in another window:
# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc
pcplusmp scsi_vhci zfs sockfs ip hook neti sctp arp usba uhci sd fctl md
lofs audiosup fcip fcp random cpc crypto logindmux ptm ufs sppp ipc ]
<-- you may want to use "::log file" to log the output -->
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

> ::threadlist -v <-- dump a list of all kernel threads, with stack
traces
ADDR PROC LWP CLS PRI WCHAN
fec1fc20 fec1f398 fec21580 0 96 0
PC: _resume_from_idle+0xb1 CMD: sched
stack pointer for thread fec1fc20: fec3b014
swtch+0x188()
sched+0x3fd()
main+0x3bc()
_locore_start+0x2da()
<-- output omitted -->
e21fbdc0 fec1f398 0 0 60 e9b907bc
PC: _resume_from_idle+0xb1 TASKQ: semex_semex_taskq0
stack pointer for thread e21fbdc0: e21fbcc4
swtch+0x188()
sema_p+0x1a6()
semex_f0+0x3a()
taskq_thread+0x192()
thread_start+8()
dfe09440 d573d9f0 d5701030 3 59 e9b907b4
PC: _resume_from_idle+0xb1 CMD: cat /devices/pseudo/semex@0:semex
stack pointer for thread dfe09440: d4576b7c
swtch+0x188()
turnstile_block+0x70b()
mutex_vector_enter+0x28f()
semex_open+0xc2()
dev_open+0x27()
spec_open+0x517()
fop_open+0xa2()
vn_openat+0x633()
copen+0x403()
open64+0x20()
sys_sysenter+0x106()
>
The cat command is blocked on a mutex: in mutex_vector_enter(). The
WCHAN field is the address of the synchronization variable on which a thread is
blocked. For cat, this should be the address the mutex it is waiting for. Look at
the mutex.
> e9b907b4::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
e9b907b4 adapt e21fbdc0 - - yes
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

The mutex is an adaptive mutex. For the default, see mutex(9F). See who
owns the mutex.
> e21fbdc0::threadlist -v
e21fbdc0 fec1f398 0 0 60 e9b907bc
PC: _resume_from_idle+0xb1 TASKQ:
semex_semex_taskq0
swtch+0x188()
sema_p+0x1a6()
semex_f0+0x3a()
thread_start+8()
The owner of the mutex is blocked on a semaphore. Take a look at the
semaphore. Again, this is in the WCHAN field for the semex_semex_taskq0
thread.
> *e9b907bc::print -t ksema_t
{
void *[2] _opaque = [ 0, 0xe21fbdc0 ]
}
> e21fbdc0::whatis
e21fbdc0 is e21fbdc0+0, allocated as a thread structure
> e21fbdc0::threadlist -v
e21fbdc0 fec1f398 0 0 60 e9b907bc
PC: _resume_from_idle+0xb1 TASKQ: semex_semex_taskq0
swtch+0x188()
sema_p+0x1a6()
semex_f0+0x3a()
thread_start+8()
A ksema_t is implemented as a sema_impl_t.
> e9b907bc::print -t sema_impl_t
{
struct _kthread *s_slpq = 0xe21fbdc0 <-- the waiting
semex_semex_taskq0
int s_count = 0 <-- the "value" of the semaphore.
}
>
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

The sema_p(9f) operation tries to decrement the semiphore value. If the
semaphore value is 0, the sema_p() call blocks. The caller of sema_p(9f)
remains blocked until a sema_v(9f) operation occurs on the semaphore.
sema_v() increments the semaphore value, and if there is someone blocked in
sema_p(), it will unblock one thread, which in turn will run and decrement the
semaphore value back to 0.
The cat command is blocked waiting for a mutex to become unlocked. The
owner of the mutex is a taskq(9f) thread that is waiting on a semaphore. See
semaphore(9f).
Look at the semex_open() function that is waiting for the mutex held by the
semex_semex_taskq0 thread. The following code is from x86. x64 and
SPARC will differ in details, but the same functions (delay, mutex_enter, delay,
sema_v) are called.
> semex_open+c2::dis
semex_open+0xa2: addl $0x18,%esp
semex_open+0xa5: pushl $0x0
semex_open+0xa7: pushl $0xc8
semex_open+0xac: call +0x68a119f <delay>
semex_open+0xb1: addl $0x8,%esp
semex_open+0xb4: movl -0x8(%ebp),%eax
semex_open+0xb7: pushl $0x0
semex_open+0xb9: addl $0x4,%eax
semex_open+0xbc: pushl %eax
semex_open+0xbd: call +0x680183e <mutex_enter>
semex_open+0xc2: addl $0x8,%esp
semex_open+0xc5: pushl $0x0
semex_open+0xc7: pushl $0xc8
semex_open+0xcc: call +0x68a117f <delay>
semex_open+0xd1: addl $0x8,%esp
semex_open+0xd4: movl -0x8(%ebp),%eax
semex_open+0xd7: pushl $0x0
semex_open+0xd9: addl $0xc,%eax
semex_open+0xdc: pushl %eax
semex_open+0xdd: call +0x696385e <sema_v>
semex_open+0xe2: addl $0x8,%esp
At semex_open+0xdd, a sema_v() the operation is done. Without source code
(or a lot more digging with mdb), you can’t tell if this is the same semaphore as
for the sema_p for which the semex_semex_taskq0 thread is waiting. The value
of the semaphore can be for 2 reasons:
1. A sema_p() operation was done on the semaphore, or
2. The initial semaphore value, set in sema_init(9f), is 0.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 6: Processes Responsible for Network Traffic
Lab
1. Write a DTrace script to determine if the value of the semaphore is 0 on
initialization, or is set to 0 by a prior call to sema_p. For this, use
anonymous tracing to trace the sema_init() calls, since the initialization is
probably done when the module is loaded.
2. Only trace sema_init() calls that are made by the semex module. Assume
that the problem is completely contained within semex. This will greatly
decrease the amount of output.
3. Add a trace for mutex_init(9f) calls made via semex.
4. Once the module is loaded, examine the anonymous trace output to see the
value of the semaphore.
5. Write a second script that traces sema_p, sema_v, mutex-acquire, and
mutex-release done via the semex module.
6. Run the second script.
7. Start the cat on the device file.
8. Explain what the output tells you.
Using netstat -in 2, you see a lot of traffic on the network. You'd like to
know which process, or processes, are responsible, and the port numbers and IP
addresses involved in the traffic. You can get the IP addresses and port numbers
from snoop(1M), but it doesn't give you information about the processes
involved. Here is example netstat output:
# netstat -in 2
input eri0 output input (Total) output
packets errs packets errs colls packets errs packets errs colls
28797736 0 24082106 0 0 28798023 0 24082393 0 0
439 0 358 0 0 439 0 358 0 0
20 0 78 0 0 20 0 78 0 0
443 0 361 0 0 443 0 361 0 0
442 0 355 0 0 442 0 355 0 0
^C
#
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Task 1
Using DTrace, determine the process, or processes, generating the network
traffic. You may use scripts in the DTrace Toolkit.
Task 2
Using DTrace, determine the port numbers and IP addresses the processes are
using.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 1 Solution
Exercise 1 Solution
Start by using hotuser from the DTrace Toolkit. Most of the time is being spent
in the stat(2) system call.
Note – On x86, it may be xstat().
# ./hotuser -p `pgrep fault2`
Sampling... Hit Ctrl-C to end.
FUNCTION COUNT PCNT
libc.so.1`openat 1 0.6%
libc.so.1`syscall 1 0.6%
libc.so.1`lmutex_unlock 1 0.6%
fault2`check_motd 1 0.6%
libc.so.1`chdir 1 0.6%
libc.so.1`lmalloc 2 1.1%
libc.so.1`strncmp 3 1.7%
libc.so.1`fstat64 4 2.2%
libc.so.1`memset 5 2.8%
fault2`stat 7 3.9%
libc.so.1`readdir 9 5.0%
libc.so.1`readdir64 12 6.7%
libc.so.1`strlen 18 10.0%
fault2`find_oracle_index_log 24 13.3%
libc.so.1`_xstat 91 50.6%
#
Which files are being stat-ed? Use dtruss from the toolkit to find out:
# ./dtruss -c -p `pgrep fault2`
SYSCALL(args) = return
xstat(0x2, 0xCEA3586A, 0x804793C) = 0 0
getdents64(0x3, 0xCEA34000, 0x2000) = 0 0
close(0x3) = 0 0
chdir(0x8061004, 0x80479CC, 0xCEBFC7B4) = 0 0
xstat(0x2, 0x8050E24, 0x8047944) = 0 0
chdir(0x806100C, 0x80479CC, 0xCEBFC7B4) = -1 Err#2
fsat(0x0, 0xFFD19553, 0x8050E2C) = 3 0
fcntl(0x3, 0x2, 0x1) = 0 0
fstat64(0x3, 0x8047850, 0xCEB80000) = 0 0
getdents64(0x3, 0xCEA34000, 0x2000) = 6664 0
xstat(0x2, 0xCEA34012, 0x804793C) = 0 0
xstat(0x2, 0xCEA3402A, 0x804793C) = 0 0
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 1 Solution
dtrace: 44882 drops on CPU 1
close 419
fcntl 419
fsat 419
fstat64 419
chdir 838
getdents64 838
xstat 96737
#
stat(2) is being called often. In the dtruss output, there is a chdir(2) that is
failing. Try errinfo on chdir(2).
# ./errinfo -p `pgrep fault2`
EXEC SYSCALL ERR DESC
fault2 chdir 2 No such file or directory
fault2 chdir 2 No such file or directory
#
This doesn't tell you the directory on which chdir(2) fails. Try a one-liner to
capture the chdir directory.
# dtrace -n 'syscall::chdir:entry/execname == "fault2"/
{printf("chdir to directory: %s", copyinstr(arg0));}
syscall::chdir:return/execname == "fault2"/
{printf("returns errno = %dn", errno);}'
dtrace: description 'syscall::chdir:entry' matched 2 probes
0 50 chdir:entry chdir to directory: /etc
0 51 chdir:return returns errno = 0
0 50 chdir:entry chdir to directory:
/opt/oracle10.6/product/data/logs
0 51 chdir:return returns errno = 2
#
There are two chdir(2) calls being made:
1. One to /etc
2. The other to /opt/oracle10.6/product/data/logs.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 1 Solution
The program does a chdir(2), then calls getdents(2) to read directory entries,
then probably is doing stat(2) calls on files in that directory. Find the directory
used for getdents(2). The getdents(2) system call is being done on file
descriptor 3 (from the dtruss output above). Use pfiles(1) to see which
directory is file descriptor 3.
# pfiles `pgrep fault2`
991:./fault2
Current rlimit: 256 file descriptors
0: S_IFCHR mode:0620 dev:301,0 ino:490566937 uid:101 gid:7 rdev:24,4
O_RDWR|O_LARGEFILE
/dev/pts/4
O_RDWR|O_LARGEFILE
/dev/pts/4
O_RDWR|O_LARGEFILE
/dev/pts/4
3: S_IFDIR mode:0755 dev:182,65538 ino:49 uid:0 gid:3 size:230
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
/etc
#
The getdents(), and probably stat(2) calls, are being made on files in the
/etc directory. Probably the program is not checking the return value on the
chdir (/opt/oracle10.6/product/data/logs) calls, and keeps trying.
Note – Creating the /opt/oracle10.6/product/data/logs directory will
not cause the program to exit. The program will exit when it finds a particular
file. For extra credit, determine the name of the file that will cause the program to
exit.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 2 Solution
Exercise 2 Solution
# iostat -xnpc 2
cpu
us sy wt id
17 6 0 77
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
15.0 9.7 689.1 306.7 0.8 0.4 32.3 14.4 10 22 c7d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8t0d0
0.1 0.0 1.4 0.1 0.0 0.0 0.5 4.8 0 0 c9t0d0p0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t0d0p1
0.1 0.1 1.4 0.1 0.0 0.0 0.4 3.7 0 0 c9t0d0
cpu
us sy wt id
36 8 0 56
59.8 60.3 108.4 2808.1 3.1 1.3 26.2 11.0 22 86 c7d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t0d0p0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t0d0p1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t0d0
cpu
us sy wt id
31 1 0 68
63.0 0.0 31.5 0.0 0.0 0.7 0.0 11.3 0 71 c7d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t0d0p0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t0d0p1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t0d0
#
One disk is read from and is busy more than 50% of the time. Use iosnoop:
# ./iosnoop
UID PID D BLOCK SIZE COMM PATHNAME
0 1097 R 6393027 512 fault3 <none>
0 1097 R 15612892 512 fault3 <none>
0 1097 R 13622687 512 fault3 <none>
0 1097 R 4677774 512 fault3 <none>
0 1097 R 5618859 512 fault3 <none>
#
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 2 Solution
fault3 is doing reads, and they appear random. Try iotop to see what it tells
you.
# ./iotop -C
Tracing... Please wait.
2009 Oct 14 19:24:13, load: 0.06, disk_r: 150 KB, disk_w: 0 KB
UID PID PPID CMD DEVICE MAJ MIN D BYTES
0 0 0 cmdk0 102 0 512
0 1097 1094 fault3 cmdk0 102 0 R 154624
0 1097 1094 fault3 cmdk0 102 0 R 148992
0 1097 1094 fault3 cmdk0 102 0 R 152064
#
Most of the I/O is from fault3, and it is all read traffic. Look at iopattern
output.
# ./iopattern
%RAN %SEQ COUNT MIN MAX AVG KR KW
100 0 57 512 512 512 28 0
100 0 63 512 512 512 31 0
100 0 55 512 512 512 27 0
100 0 60 512 512 512 30 0
99 1 117 512 131072 6043 25 665
#
Most of the I/O is reads, and almost all of it is random. The fault3 program is
doing most of the random input. This may not indicate a problem. It depends on
what the application is trying to accomplish.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 3 Solution
Exercise 3 Solution
Task 1
The following is a DTrace script which keeps track of timings for the clock()
routine.
clock.d
===========
#pragma D option quiet
clock:entry
{
self->ts = vtimestamp;
}
clock:return
/self->ts/
{
@av = avg(vtimestamp-self->ts);
@mx = max(vtimestamp-self->ts);
@mn = min(vtimestamp-self->ts);
self->ts = 0;
}
END
{
printa("Average time is %@d nsecsn", @av);
printa("Maximum time is %@d nsecsn", @mx);
printa("Minimum time is %@d nsecsn", @mn);
}
# ./clock.d
Average time is 4299 nsecs
Maximum time is 80037 nsecs
Minimum time is 1136 nsecs
#
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 3 Solution
Task 2
Use an aggregation to show the distribution of timings.
clockagg.d
==========
#pragma D option quiet
clock:entry
{
self->ts = vtimestamp;
}
clock:return
/self->ts/
{
@q = quantize(vtimestamp-self->ts);
self->ts = 0;
}
END
{
printa("Time Distribution is %@d nsecsn", @q);
}
# ./clockagg.d
Time Distribution is
512 | 0
1024 |@@@@ 114
2048 |@@@@@@@@@@@@@@@@@@@@@@@@@@ 848
4096 |@@@@@@@@@ 283
8192 |@ 37
16384 | 0
32768 | 0
65536 | 13
131072 | 0
nsecs
#
Most of the times are short. Occasionally the time takes longer.
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 3 Solution
Task 3
For “Task 3”, the script is more complicated. Save the timestamp on entry into
clock. While in clock(), trace any function that is called. On return, if the time
spent is longer than the longest time seen, commit the speculation. Otherwise
discard it. Here is a script to do this:
longclock.d
===========
#pragma D option cleanrate=501
#pragma D option flowindent
clock:entry
{
self->ts = timestamp;
self->spec = speculation();
speculate(self->spec);
}
fbt:::
/self->spec/
{
speculate(self->spec);
}
clock:return
/self->spec && (timestamp-self->ts)> longest/
{
commit(self->spec);
longest = timestamp-self->ts;
self->ts = 0;
self->spec = 0;
}
clock:return
/self->spec/
{
discard(self->spec);
self->ts = 0;
self->spec = 0;
}
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 3 Solution
Let this script run for a little while. At first, there will be lots of output.
Eventually, the longest time will be reached, and the script will stop sending
output. At that time, kill the script and analyze the results.
# ./longclock.d
dtrace: script './longclock.d' matched 58436 probes
dtrace: cleaning rate reduced to 5000 hz
CPU FUNCTION
0 -> clock
0 -> set_anoninfo
0 <- set_anoninfo
0 -> set_freemem
0 -> wakeup_pcgs
0 <- wakeup_pcgs
0 <- set_freemem
0 -> clock_tick_schedule
0 -> clock_tick_execute_common
0 -> clock_tick_process
0 -> thread_free_prevent
0 <- thread_free_prevent
0 -> thread_free_allow
0 <- thread_free_allow
0 -> clock_tick
0 -> ts_tick
0 -> thread_lock
0 -> splr
0 -> apic_setspl
0 -> psm_get_cpu_id
0 <- psm_get_cpu_id
0 -> get_local_apic_pri
0 <- get_local_apic_pri
0 <- apic_setspl
0 <- splr
0 <- thread_lock
0 -> disp_lock_exit_nopreempt
0 -> do_splx
0 -> apic_setspl
0 -> psm_get_cpu_id
0 <- psm_get_cpu_id
0 -> get_local_apic_pri
0 <- get_local_apic_pri
0 <- apic_setspl
0 <- do_splx
0 <- disp_lock_exit_nopreempt
0 <- ts_tick
0 -> task_cpu_time_incr
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Exercise 3 Solution
0 <- task_cpu_time_incr
0 -> rm_asrss
0 -> hat_get_mapped_size
0 <- hat_get_mapped_size
0 <- rm_asrss
0 <- clock_tick
0 <- clock_tick_process
0 -> clock_tick_process
0 -> thread_free_prevent
0 <- thread_free_prevent
0 -> thread_free_allow
0 <- thread_free_allow
0 <- clock_tick_process
0 <- clock_tick_execute_common
0 -> do_interrupt
0 -> tlb_service
0 <- tlb_service
0 -> apic_intr_enter
0 <- apic_intr_enter
0 -> hilevel_intr_prolog
0 -> apic_intr_exit
0 -> psm_get_cpu_id
0 <- psm_get_cpu_id
0 <- apic_intr_exit
0 <- hilevel_intr_epilog
0 -> dosoftint
0 -> dosoftint_prolog
0 <- dosoftint_prolog
0 <- dosoftint
0 <- do_interrupt
0 -> sys_rtt_common
0 <- sys_rtt_common
0 <- clock_tick_schedule
0 <- clock
Note – This output may be different for the Solaris version on which you run the
script. The above may be the shortest, but it is the first, so it’s the longest so far.
0 -> clock
0 -> do_interrupt
0 -> tlb_service
0 <- tlb_service
0 -> apic_intr_enter
0 <- apic_intr_enter
Unauthorized
reproduction
or
distribution
prohibitedฺ
Copyright©
2011,
Oracle
and/or
its
affiliatesฺ

Dynamic Performance Tuning and Troubleshooting With DTrace Ed 2 (Activity Guide)

Dynamic Performance Tuning and Troubleshooting With DTrace Ed 2 (Activity Guide)

Recommended

Recommended

More Related Content

Similar to Dynamic Performance Tuning and Troubleshooting With DTrace Ed 2 (Activity Guide)

Similar to Dynamic Performance Tuning and Troubleshooting With DTrace Ed 2 (Activity Guide) (20)

Recently uploaded

Recently uploaded (20)

Dynamic Performance Tuning and Troubleshooting With DTrace Ed 2 (Activity Guide)