SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
2.
Welcome to dtrace.conf!
• ~80 participants, representing:
• ~35 companies, ~7 operating systems, ~3 language
runtimes, ~3 databases, ~2 networking infrastructure
companies, ~2 universities
• …and 1 industry analyst
• Twitter: #dtraceconf
• Thank you to our sponsors: Stone Cobra, DEY for
illumos, Nexenta, Basho and the Joyent Cloud
• Thanks to @substack for the T-shirt design
• Huge thank you to @deirdres for organizing dtrace.conf!
6.
DTrace in the past year
• Added a log-linear quantization action (llquantize())
• Added KVM support with vmregs[] variable
• Added the CTF-aware print action (ht: @ericschrock)
• A bunch of little stuff:
• Allowed tracemem() to take a variable-size
• Added toupper()/tolower()
• Allowed lltostr() to take an optional base
• A bunch of bug fixes:
• Fixed USDT in the non-global zone from the global zone
• Added USDT provider reaping (a long-time issue)
7.
DTrace: Some open problems
• Implementing (finishing) dynamic translators
• Instrumenting hardware-virtualized guests from the host
• Support dynamic USDT providers
• Integrating with user-level CTF
• Supporting user-level postmortem DTrace
• Transparent instrumentation of multiple processes
• Unifying theme: better instrumentation up-stack
8.
Dynamic translators
• DTrace uses translators to translate arguments from
their implementation to a stable (synthetic) structure
• These translators are all static — they are found (and
compiled) at compile time by libdtrace
• Implementation-specific .d files must therefore be
delivered into the DTrace library path
• If it must be said, this is a mess: different versions of
implementations canʼt co-exist
• Static translation also means that having DTrace privs in
the non-global zone is insufficient to get to useful
(translated) in-kernel structures
9.
Dynamic translators, cont.
• For as long as we have had translators, we have talked
of making them dynamic
• Idea: translation definition lives in the binary (either the
application or the kernel), and the DIF use of the
translator is symbolic — DTrace figures it out at run-time
• Difficult in that it requires changes at many levels of the
DTrace stack (compiler, in-kernel, app build chain)
• Good news: some compiler work has already been done
(see use of DIF_OP_XLATE and DIF_OP_XLARG —
only generated when -xlate=dynamic!)
• Bad news: no in-kernel consumer (yet) — and the
compiler work is almost certainly incomplete...
10.
Instrumenting VM guests
• DTrace is present on several OSs that also support first
class hardware virtualization (e.g., KVM)
• We have taken a first stab with the vmregs[] variable
and exit instrumentation — but there is much more to do
• The guest is just executing on host-accessible physical;
what guest cooperation do we need to be able to
instrument that?
• What would the mechanism for guest cooperation be?
• Can we meaningfully instrument guests without any
guest cooperation at all?
11.
Supporting dynamic USDT providers
• Chris Andrews (@chrisandrews) developed a very
useful technique for defining providers in dynamic
languages: the provider DOF is generated dynamically
• Chris has done this for node (node-dtrace-provider),
Ruby (ruby-dtrace) and Perl (Devel::DTrace::Provider)
• This has proven itself in other software, e.g. Joyentʼs
Mark Cavage (@mcavage) has used this to great effect
in node-restify and ldap.js
• Is there something that we can or should do to further
generalize this technique or better support it?
• What is the overlap with dynamic translators?
12.
Integrating with user-level CTF
• Currently, DTrace does not consume user-level CTF
• When $target is specified, it should in theory be possible
to consume these definitions in lieu of or in addition to
kernel CTF
• How to know the types of arguments to pid-provider
probes? (Could a first cut be made without this?)
• How should user-level types be scoped? (Double back-
tick as in symbols for helpers?)
• If done implicitly, should we take a swing at implicit
copyin() at the same time?
13.
User-level postmortem support
• Postmortem support in DTrace — like anonymous
tracing (and speculative tracing!) — is one of those
features that you donʼt need until you need it…
• ...at which point it pretty much saves your butt
• Is there interest in bringing this to user-level?
• Would involve extending “bufpolicy=ring” when $target is
set and $target dumps core to write ring buffer to core
file in new section
• New section would be consumed via mdbʼs ::dtrace
14.
Instrumenting multiple processes
• Currently, instrumenting multiple processes requires
either explicit enumeration of pid probes or USDT
• Might be useful to follow-on-fork with the pid provider
and/or specify multiple processes to instrument
• There are lots of weird corner cases (e.g., what does
$target evaluate to if used in a predicate?)
• We have used USDT to largely skirt this issue; have we
merely bought ourselves time, or does USDT largely
obviate the need for this?