1.
DTrace
dtrace.conf(12)
SVP, Engineering
bryan@joyent.com
Bryan Cantrill
@bcantrill

2.
Welcome to dtrace.conf!
• ~80 participants, representing:
• ~35 companies, ~7 operating systems, ~3 language
runtimes, ~3 databases, ~2 networking infrastructure
companies, ~2 universities
• …and 1 industry analyst
• Twitter: #dtraceconf
• Thank you to our sponsors: Stone Cobra, DEY for
illumos, Nexenta, Basho and the Joyent Cloud
• Thanks to @substack for the T-shirt design
• Huge thank you to @deirdres for organizing dtrace.conf!

3.
DTrace Version: 2005-2008
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2005 2006 2007 2008 2009 2010 2011

4.
DTrace Version: 2005-2010
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2005 2006 2007 2008 2009 2010 2011

5.
DTrace Version: 2005-present
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2005 2006 2007 2008 2009 2010 2011

6.
DTrace in the past year
• Added a log-linear quantization action (llquantize())
• Added KVM support with vmregs[] variable
• Added the CTF-aware print action (ht: @ericschrock)
• A bunch of little stuff:
• Allowed tracemem() to take a variable-size
• Added toupper()/tolower()
• Allowed lltostr() to take an optional base
• A bunch of bug fixes:
• Fixed USDT in the non-global zone from the global zone
• Added USDT provider reaping (a long-time issue)

7.
DTrace: Some open problems
• Implementing (finishing) dynamic translators
• Instrumenting hardware-virtualized guests from the host
• Support dynamic USDT providers
• Integrating with user-level CTF
• Supporting user-level postmortem DTrace
• Transparent instrumentation of multiple processes
• Unifying theme: better instrumentation up-stack

8.
Dynamic translators
• DTrace uses translators to translate arguments from
their implementation to a stable (synthetic) structure
• These translators are all static — they are found (and
compiled) at compile time by libdtrace
• Implementation-specific .d files must therefore be
delivered into the DTrace library path
• If it must be said, this is a mess: different versions of
implementations canʼt co-exist
• Static translation also means that having DTrace privs in
the non-global zone is insufficient to get to useful
(translated) in-kernel structures

9.
Dynamic translators, cont.
• For as long as we have had translators, we have talked
of making them dynamic
• Idea: translation definition lives in the binary (either the
application or the kernel), and the DIF use of the
translator is symbolic — DTrace figures it out at run-time
• Difficult in that it requires changes at many levels of the
DTrace stack (compiler, in-kernel, app build chain)
• Good news: some compiler work has already been done
(see use of DIF_OP_XLATE and DIF_OP_XLARG —
only generated when -xlate=dynamic!)
• Bad news: no in-kernel consumer (yet) — and the
compiler work is almost certainly incomplete...

10.
Instrumenting VM guests
• DTrace is present on several OSs that also support first
class hardware virtualization (e.g., KVM)
• We have taken a first stab with the vmregs[] variable
and exit instrumentation — but there is much more to do
• The guest is just executing on host-accessible physical;
what guest cooperation do we need to be able to
instrument that?
• What would the mechanism for guest cooperation be?
• Can we meaningfully instrument guests without any
guest cooperation at all?

11.
Supporting dynamic USDT providers
• Chris Andrews (@chrisandrews) developed a very
useful technique for defining providers in dynamic
languages: the provider DOF is generated dynamically
• Chris has done this for node (node-dtrace-provider),
Ruby (ruby-dtrace) and Perl (Devel::DTrace::Provider)
• This has proven itself in other software, e.g. Joyentʼs
Mark Cavage (@mcavage) has used this to great effect
in node-restify and ldap.js
• Is there something that we can or should do to further
generalize this technique or better support it?
• What is the overlap with dynamic translators?

12.
Integrating with user-level CTF
• Currently, DTrace does not consume user-level CTF
• When $target is specified, it should in theory be possible
to consume these definitions in lieu of or in addition to
kernel CTF
• How to know the types of arguments to pid-provider
probes? (Could a first cut be made without this?)
• How should user-level types be scoped? (Double back-
tick as in symbols for helpers?)
• If done implicitly, should we take a swing at implicit
copyin() at the same time?

13.
User-level postmortem support
• Postmortem support in DTrace — like anonymous
tracing (and speculative tracing!) — is one of those
features that you donʼt need until you need it…
• ...at which point it pretty much saves your butt
• Is there interest in bringing this to user-level?
• Would involve extending “bufpolicy=ring” when $target is
set and $target dumps core to write ring buffer to core
file in new section
• New section would be consumed via mdbʼs ::dtrace

14.
Instrumenting multiple processes
• Currently, instrumenting multiple processes requires
either explicit enumeration of pid probes or USDT
• Might be useful to follow-on-fork with the pid provider
and/or specify multiple processes to instrument
• There are lots of weird corner cases (e.g., what does
$target evaluate to if used in a predicate?)
• We have used USDT to largely skirt this issue; have we
merely bought ourselves time, or does USDT largely
obviate the need for this?

15.
On with dtrace.conf!