From DTrace to Linux

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

TRACING SUMMIT
EUROPE
From
DTrace
To
Linux:
Oct, 2014
What
can
Linux
learn
from
DTrace?
Brendan
Gregg
Senio...

Brendan
Gregg
• DTrace
contribu?ons
include:
– Primary
author
of
the
DTrace
book
– DTraceToolkit
– dtrace-‐cl...

Agenda
1. DTrace
– What
is
DTrace,
really?
– Who
is
DTrace
for,
really?
– Why
doesn’t
Linux
have
DTrace?
–...

What
is
DTrace,
really?

Technology
+
Marke?ng
(Like
many
other
company
products)

Prior
Technology
Kerninst:
kernel
dynamic
tracing,
Solaris
2.5.1,
1999

Prior
Technology
Early
dynamic
tracers
weren’t
safe

Prior
Technology
• Also:
– Sun’s
TNF
– DProbes:
user
+
kernel
dynamic
tracing
– Linux
Trace
Toolkit
(LTT)
–...

Technology
• DTrace:
– Safe
for
produc?on
use
• You
might
step
on
your
foot
(overhead),
but
you
won’t
shoo...

A
Typical
Conversa?on…
“Does
Linux
have
DTrace
yet?”
“That’s
a
pity”
“DTrace
is
awesome!”
“I’m
not
sure”
...

Marke?ng

Early
Marke?ng

Early
Marke?ng
• DTrace
had
awesome
marke?ng
– People
s?ll
want
it
but
don’t
really
know
why
• Early
marke...

Marke?ng
Evolved
• Sun
marke?ng
become
innova?ve
– Engineering
blogs,
BigAdmin
– Marke?ng
staff
who
used
and ...

Later
Marke?ng

Later
Marke?ng
• Many
ini?a?ves
by
Deirdré
Straughan:
– Social
media,
blogs,
events,
the
ponycorn
mascot,
.....

Who
is
DTrace
for,
really?

DTrace
end-‐users:
Current
Es?mated
DTrace
guide
users:
~100
Script
end-‐users:
~5,000
Note:
91.247%
of
st...

DTrace
end-‐users:
Current
• DTrace
guide
users:
~100
– Understand
the
~400
page
Dynamic
Tracing
Guide
– De...

DTrace
end-‐users:
Future
eg,
Oracle
ZFS
Storage
Appliance
Analy?cs

DTrace
end-‐users:
Future
Possible
Future
DTrace
guide
users:
~100
Script
end-‐users:
~5,000
GUI
end-‐user...

Company
Usage

Company
Usage
• Prac?cal
usage
for
most
companies:
– A)
A
performance
team
(or
person)
• Acquires
useful
sc...

Why
doesn’t
Linux
have
DTrace?

Why
doesn’t
Linux
have
a
DTrace
equivalent?
4
Answers…

1.
It
does
(sort
of)
Jrace
perf_events

1.
It
does
(sort
of)
• Linux
has
changed
– In
2005,
numerous
Linux
issues
were
difficult
or
impossible
to...

2.
Technical
semantic error: missing x86_64
kernel/module debuginfo

2.
Technical
• Linux
is
a
more
difficult
environment
– Solaris
always
has
symbols,
via
CTF,
which
DTrace
u...

3.
Linux
isn’t
a
Company
“All
the
wood
behind
one
arrow”
–
Sco^
McNealy,
CEO,
Sun
Microsystems

3.
Linux
isn’t
a
Company
• Linus
can
refuse
patches,
but
can’t
stop
projects
– The
tracing
wood
is
split ...

4.
No
Trace
Race

4.
No
Trace
Race
• Post
2001,
Solaris
was
losing
ground
to
Linux.
Sun
desperately
needed
differen?ators
to...

Why
doesn’t
Linux
have
DTrace
itself?
2
Answers…

1.
The
CDDL
From: Claire Giordano <claire.giordano@sun.com>
To: license-discuss@opensource.org
[Open
Source
Ini?a?v...

1.
The
CDDL
• Linux
tradi?onally
includes
the
tracer/profiler
in
the
(GPL)
kernel,
but
the
DTrace
license
...

2.
DTrace
ports

2.
DTrace
ports
• There
are
two
ports,
but
both
currently
incomplete
• A)
h^ps://github.com/dtrace4linux/linux...

What
with
DTrace
worked
well?
5
Key
items…

1.
Produc?on
Safety

1.
Produc?on
Safety
• DTrace
architecture
– Restricted
probe
context:
no
kernel
facility
calls,
restricted
in...

2.
All
the
wood
behind
one
arrow
DTrace

2.
All
the
wood
behind
one
arrow
• Can
Linux
learn
this?
– Can
we
vote
some
off
the
Linux
tracing
islan...

3.
In-‐Kernel
Aggrega?ons
value ------------- Distribution ------------- count!
4096 | 0!
8192 |@@@@@@@@@@@@@@@@@@@@...

3.
In-‐Kernel
Aggrega?ons
• Changed
how
inves?ga?ons
are
conducted
– rapid,
live
analysis
• Low
overhead:
– ...

4.
Many
Example
Scripts

4.
Many
Example
Scripts

4.
Many
Example
Scripts
• Scripts
serve
many
needs:
– tools:
ready
to
run
– examples:
learn
by-‐example
– ...

4.
Many
Example
Scripts
• Linux
can
learn:
– Many
users
will
just
run
scripts,
not
write
them
– People
wa...

5.
Marke?ng

5.
Marke?ng
• DTrace
was
effec?vely
marketed
in
many
ways
– Tradi?onal,
social,
blogs,
scripts,
ponycorn,
… ...

Cute
Tracing
Logos
dtrace4linux
perf_events
ktap
LTTng
SystemTap
Jrace
Ponies
by
Deirdré
Straughan,
using:
h...

Other
Things
• Programmable/scriptable
• Built-‐in
stability
seman?cs

What
with
DTrace
didn’t
work
well?

What
with
DTrace
didn’t
work
well?
5
Key
Issues…

1.
Adop?on

1.
Adop?on
• Few
customers
ever
wrote
DTrace
scripts
– DTrace
should
have
been
used
more
than
it
was
– Su...

1.
Adop?on
• Linux
can
learn:
– Adop?on
is
about
more
than
just
the
technology
• Documenta?on,
marke?ng,
tr...

2.
Training
This
is
to
cer?fy
that
Brendan
Gregg
Has
Completed
the
Sun
Educa?onal
course
DTrace
is
a
Sol...

2.
Training
• Early
training
was
not
very
effec?ve
– Sun
began
including
the
DTraceToolkit
in
courses,
with...

3.
GUIs

3.
GUIs
• Dozens
of
performance
monitoring
products,
but
almost
no
meaningful
DTrace
support
• A
couple
of ...

3.
GUIs
• Linux
can
learn:
– Real
adop?on
possible
through
scripts
&
GUIs
– Use
the
GUI
to
add
value
to ...

3.
GUIs
• Heat
maps
are
an
example
must-‐have
use
case
for
trace
data

4.
Overheads
While
the
DTrace
technology
is
awesome,
it
does
have
some
minor
technical
challenges
as
well

4.
Overheads
• While
op?mized,
for
many
targets
the
DTrace
CPU
overheads
can
s?ll
be
too
high
– Scheduler ...

5.
Syscall
Provider

5.
Syscall
Provider
• Solaris
DTrace
instrumented
the
trap
table,
and
called
it
the
syscall
provider
– Whic...

Other
Issues
• The
lack
of:
– Bounded
loops
(like
SystemTap)
– Kernel
instruc?on
tracing
(like
perf_events)
...

From
DTrace
to
Linux
Tracers
(2014)

• Massive
AWS
EC2
Linux
cloud,
with
FreeBSD
appliances
for
content
delivery
• Performance
is
cri?cal:
>50M
...

System
Tracing
at
Ne7lix
• Present:
– Jrace
can
serve
many
needs
– perf_events
some
more,
esp.
with
debugin...

1.
Jrace

1.
Jrace
• Tracing
and
profiling:
/sys/kernel/debug/tracing
– added
by
Steven
Rostedt
and
others
since
2.6.27...

perf-‐tools:
iosnoop
• Block
I/O
(disk)
events
with
latency:
# ./iosnoop –ts!
Tracing block I/O. Ctrl-C to end.!...

perf-‐tools:
iolatency
• Block
I/O
(disk)
latency
distribu?ons:
# ./iolatency !
Tracing block I/O. Output every 1...

perf-‐tools:
opensnoop
• Trace
open()
syscalls
showing
filenames:
# ./opensnoop -t!
Tracing open()s. Ctrl-C to en...

perf-‐tools:
kprobe
• Just
wrapping
capabili?es
eases
use.
Eg,
kprobes:
# ./kprobe 'p:open do_sys_open filename=...

1.
Jrace
• Sugges?ons:
– I’m
blogging
and
so
can
you!
– Func?on
profiler:
• Can
these
in-‐kernel
counts
be...

2.
perf_events

2.
perf_events
• In-‐kernel,
tools/perf,
mul?-‐tool,
“perf”
command
• Experiences:
– Stable,
powerful,
reliabl...

perf
CPU
Flame
Graph
Broken
Java
stacks
(missing
frame
pointer)
Kernel
TCP/IP
GC
Idle
Time
thread
Locks
e...

2.
perf_events
• Sugges?ons:
– Support
for
func?on
argument
symbols
without
a
full
debuginfo
– Rework
scrip?n...

3.
SystemTap

3.
SystemTap
• The
most
powerful
of
the
tracers
• Used
for
the
deepest
custom
tracing
– Especially
Java
ho...

4.
eBPF

4.
eBPF
• Extended
BPF:
programs
on
tracepoints
– High
performance
filtering:
JIT
– In-‐kernel
summaries:
ma...

4.
eBPF
• Experiences:
– Can
have
lower
CPU
overhead
than
DTrace
– Very
powerful:
really
custom
maps
– Asse...

Other
Tracers
• Experiences
and
sugges?ons:
– ktap
– LTTng
– Oracle
Linux
DTrace
– dtrace4linux
– sysdig

The
Tracing
Landscape,
Oct
2014
Scope
&
Capability
(less
brutal)
Ease
of
use
sysdig
perf
Jrace
eBPF
ktap ...

Summary
• DTrace
is
an
awesome
technology
– Which
has
also
had
awesome
marke?ng
• Tradi?onal,
social,
sales,...

Acks
• dtrace.conf
X-‐ray
pony
art
by
substack
• h^p://www.raspberrypi.org/
rasberry
PI
image
• h^p://en.wikip...

Links
• h^ps://www.usenix.org/legacy/event/usenix04/tech/general/full_papers/
cantrill/cantrill.pdf
• Jrace
&
perf-‐...

Thanks
• Ques?ons?
• h^p://slideshare.net/brendangregg
• h^p://www.brendangregg.com
• bgregg@ne7lix.com
• @brendangre...

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 98 Ad

1 of 98 Ad