OSCON2008 Full-stack Introspection Crash Course

Slide 1: Full-stack Introspection Crash Course / Operations

Slide 2: Why listen to Theo? • I wrote a book (okay it has nothing to do with performance) • I lead the performance and scalability practice at OmniTI • It’s my job to “un”break things... yes really. • I build high-performance software. • Why not? • Listening never hurt, you can judge worth on your own.

Slide 3: Full-stack? • How do you see the whole stack, all at the same time? • DTrace • What’s the “whole” stack? LAMP? • Linux... err, not yet. (Mac OS X, FreeBSD, Solaris) • Apache... yup. • MySQL? Sure, but PostgreSQL is cooler. • PHP... and Perl... and Java... and Ruby... and... • anything with compiled with frames: C, C++, etc.

Slide 4: What is DTrace? • DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems • DTrace allow you to write scripts that can consume output from various traceable points in a running system: • kernel function boundaries, • syscalls, I/O, network, mib, nfs, scheduler, vm, etc. • arbitrary points in user-land code (entry/return/op codes) • user-space tracepoints • all at the same time

Slide 5: What is DTrace really? • It is an Oracle of knowledge. A peephole into the “real world.” • The quality of the answers it provides are limited by the quality of the questions it is asked. • DTrace can be a bit frustrating: • Syntax is simple: Awk/C • Concept is simple: watch stuff. • Implementation is safe: designed “production safe” • Learning to ask good questions is hard.

Slide 6: DTrace is a rabbit hole

Slide 7: Think Scientifically • Use the scientific method (no, I’m not rehashing lower education) • Be rigorous and analytical: • Use your gut to define the hypothesis. • Use repeatable tests to gather evidence to prove or disprove (PLEASE: don’t change more than one thing at a time) • Use experience to draw conclusions. • The proof of th' pudding's seen i' the eating.

Slide 8: Three Aproaches • SAFEST: • Start closest to the user; repeat the problem; track it down. • SORCERY: • Start at the problem; solve it; prove you were right. • Requires hypothesizing that the problem is what it is. • EXPERIENCED: • Start in the middle; look for symptoms; form a targeted hypothesis • Apply Sorcery

Slide 9: DTrace only works for real • Huh? • Contrived problems have contrived solutions. • DTrace is input driven: • garbage in, garbage out • (contrived in, contrived out) • Retracing the steps of a previous introspection exercise is boring • I’ll do my best to make it interesting... maybe you can help.

Slide 10: DTrace incantations Clatto Verata N... Necktie... Nickel... It's an "N" word, it's definitely an "N" word!

Slide 11: Thanks! • Sun & DTrace (esp. Bryan Cantrill, Adam Leventhal and Mike Shapiro) • Apple & FreeBSD for porting DTrace • Paul Fox, for serious effort toward porting it to Linux: http://www.crisp.demon.co.uk/blog/ ftp://crisp.dynalias.com/pub/release/website/dtrace • OmniTI For being an awesome place to work... We’re hiring.