This document reflects on the evolution and impact of DTrace, a dynamic tracing framework integrated into Solaris in 2003 and later open-sourced in 2005. It emphasizes successful strategies in developing DTrace, such as prioritizing production systems, using a domain-specific language, and actively utilizing the tool during its development. The authors also acknowledge the luck involved in open-sourcing DTrace and its subsequent adoption across various operating systems.

1. A ScyllaDB Community
DTrace at 21:
Reflections on Fully-grown Software
Bryan Cantrill
CTO, Oxide Computer Company

2. OXIDE
DTrace in adulthood
• On September 3, 2003, we integrated DTrace into Solaris
• DTrace became open source in 2005, and has found its way into quite a
few systems (e.g., Mac and Windows) – and has influenced many more
• DTrace is broadly done and firmly in adulthood – and we’ve been lucky
enough to be using it more or less continuously for its entire life
• The benefit of hindsight allows us to reflect on the stuff we got right, the
things we figured out – and the ways in which we got lucky

3. OXIDE
We got right: Focus on production systems
• From the outset, our focus for DTrace was on production systems: we
knew that production systems have performance and other pathologies
that cannot be readily reproduced elsewhere!
• To be acceptable for production, instrumentation has to be absolutely
safe above all else: misuse cannot result in system failure!
• Facility has to always be available – can’t rely on recompiling anything,
downloading binaries, downloading symbol tables, etc.
• These constraints led to deep integration with the operating system

4. OXIDE
We got right: Dynamic instrumentation
• We believed from the outset that instrumentation should be dynamic
• Part of this comes from the production constraint: it was essential that
DTrace have zero probe effect when disabled
• We also felt strongly that we should be able to instrument software that
had had no modifications to support it – and in arbitrary contexts
• And because we wanted to also replace several existing tools, we
separated out the methodology of instrumentation from the framework
that consumed them

5. OXIDE
We got right: Organizational approach
• We had been thinking about DTrace long before we started – and over
the years, integrated the foundation that we knew we would need
• But it couldn’t be done as a side-project – we needed to focus
• Instead of attempting to timeline an entire project, we made the case in
late 2001 to allow for two of us to focus for six months
• After six months we had the proof points to add a third engineer – and
to allow us to remain focussed on it for several more years
• The team was always very small (three people!) and not in an office

6. OXIDE
We figured out: A domain specific language
• We knew we wanted to have expressive power in the actions taken on
instrumentation, but something we figured out early was the need to
have our own domain specific language
• We were heavily inspired in syntax by AWK, a little language that
continues to have an outsized influence
• We dubbed our language “D” (and its intermediate form “DIF”), not
knowing that Walter Bright was concurrently working on a language of
the same name!

7. OXIDE
We figured out: A domain specific virtual machine
• We had assumed that we would transpile DIF to native instructions for
execution, but it became quickly clear that executing DIF in a domain
specific virtual machine in the kernel would be a huge win
• Not only did this allow us to move quickly by adding powerful concepts
to DIF (e.g., thread-local variables), it allowed us to achieve the safety
constraint with extensive run-time checks
• We assure completion of DIF by making it Turing incomplete – DIF has
no backwards branches!

8. OXIDE
We got right: We used it ourselves
• We used DTrace heavily ourselves – and used it as early as possible in
its development to find issues in the operating system and beyond
• This led to a more robust system – and one’s whose emphasis is utility
• Every feature in DTrace is a direct consequence of concrete need!
• This has led to many features that may seem arcane – but they are only
arcane until you need them: thread-local variables, anonymous tracing,
speculative tracing, postmortem tracing, arbitrary instruction tracing, etc.

9. OXIDE
We figured out: Statically-defined tracing
• While the origin of DTrace is dynamic (our early tagline was “Concise
answers to arbitrary questions”), it is overwhelming to deal with the
implementation of the system to ask
• We saw the need for statically-defined tracing (SDT) at points of
semantic interest (CPU scheduling, performing I/O, etc.)
• SDT probes coupled with type information via CTF and structure
translators allowed for true interface stability, allowing users to
instrument in terms of semantics and not implementation

10. OXIDE
We figured out: Application-level instrumentation
• When we set out, it was with a focus on kernel-level instrumentation
• Kernel-level instrumentation is necessary not just for kernel-level issues,
but also to observe system-wide effects of application-level issues…
• …but also insufficient: we needed application-level instrumentation
• This is especially valuable with statically-defined tracing; user-level SDT
(USDT) allows for programs themselves to be instrumented in a
semantically stable fashion by their own users
• USDT is essential at Oxide! See https://github.com/oxidecomputer/usdt

11. OXIDE
We got right: Writing our own documentation
• In 2003, documentation was really the only way to learn how to use a
system – especially one that was sophisticated and proprietary
• We made the deliberate decision to write all of our own documentation
• The DTrace documentation (https://illumos.org/books/dtrace) was all
written by the three DTrace engineers: the documentation was
authoritative, canonical – and we found many bugs in writing the docs!

12. OXIDE
We figured out: Writing a paper
• In 2003 (right after the integration!), we attended an academic
conference (AADEBUG – RIP!) which inspired us to write a canonical
academic paper on DTrace
• The resulting paper, Dynamic Instrumentation of Production Systems,
was presented at the USENIX Annual Technical Conference in 2004
• We wrote a broader paper for practitioners in ACM Queue in 2006,
Hidden in Plain Sight, that made the case for software observability
• There is tremendous value in rigorously describing your ideas!

13. OXIDE
We got lucky: Open source
• DTrace was born an entirely proprietary system, but there had been
conversations internally about open sourcing the operating system as
early as 1997 – and by 2003, there was urgency around it
• Open sourcing big, proprietary software is not easy – and if DTrace had
remained proprietary, it would have died
• We got very lucky that Sun not only prioritized open sourcing the
operating system, but led that with DTrace first (in January 2005)

14. OXIDE
We got lucky: Ports to other systems
• DTrace is not easy – it is very tightly integrated with the operating
system, and depends on many OS facilities
• Initially ported to FreeBSD by the late John Birrell – and then to MacOS,
QNX, Linux, and Windows
• These ports required significant effort by veteran technologists!
• The different ports have taken different liberties (e.g., the DTrace port to
Linux now uses eBPF as a backend), but all shared by the goal of
dynamic instrumentation in production

15. OXIDE
We got lucky: DTrace endures
• We feel lucky to still be using DTrace everyday – and especially to be
using it on our thorniest problems!
• We feel lucky to be bringing new people into DTrace (older than DTrace
itself – but not by much!), new languages (Rust!) and new systems
• We feel lucky that it’s open source, which granted DTrace eternal life
• We feel lucky to still be working together – join Adam Leventhal and me
on our podcast Oxide and Friends!

16. Bryan Cantrill
bryan@oxidecomputer.com
@bcantrill{.bsky.social,.mastodon.social}
https://oxide.computer