Dynamic Tracing of your AMP web site

Dynamic Tracing your AMP site Sriram Natarajan Engineer Sun Microsystems Inc

What is Dtrace ?

Old school performance analysis

Collect data as required.
Analyze data looking for bottlenecks, trends and correlations.
Speculate on remedies
Evaluate and repeat the process

Does not replace any of the existing tools like vmstat, mpstat, prstat, sar etc..
Largely new functionality to ask questions.

How does it work ?

Ability to instrument binary code on the fly
Zero probe effect when Dtrace is not running.
Does a fast trap into Dtrace probe to minimize performance impact.

A simple Dtrace script #!/usr/sbin/dtrace -qs syscall:::entry { @num[probefunc] = count() }

Dtrace format ● provider:module:function:name ● -P provider ● -m module ● -f function ● -n name ● -s script ● -p pid ● -c run command ● -l list ● -v verbose

How to write a Dtrace script

Sample Dtrace scripts.

/usr/demo/dtrace (Solaris 10/OpenSolaris)

Dtracetoolkit includes task specific Dtrace scripts for OpenSolaris.

/opt/DTT (OpenSolaris)
http://www.opensolaris.org/os/community/dtrace/dtracetoolkit/

Customize and built upon these sample scripts.
MySQL sample Dtrace scripts are available at

http://dev.mysql.com/doc/refman/6.0/en/dba-dtrace-mysqld-ref.html

Dtrace for AMP stack

Apache Dtrace probes are made possible by Apache Dtrace extension
MySQL / PHP Dtrace probes are enabled within engine
Fired when execution flow reaches probe location.
No overhead for inactive probes.
Provide hard to get under the hood information in a simple way.
Some performance hit depending on number of probes fired

Sample MySQL D Script #!/usr/sbin/dtrace -qs dtrace:::BEGIN { printf("%-40s %2s %-9s ", "Query", "QC", "Time(ms)"); } mysql*:::query-start { self->query = copyinstr(arg0); self->querystart = timestamp; self->qc = 0; } mysql*:::query-cache-hit { self->qc = 1; } mysql*:::query-cache-miss { self->qc = 0; } mysql*:::query-done { printf("%-40s %-2s %-9d ",self->query,(self->qc ? "Y" : "N"), (timestamp - self->querystart) / 1000000); }

Sample MySQL D Script Output select * from t1 order by i limit 10 N 1072 set global query_cache_size=262144 N 0 select * from t1 order by i limit 10 N 781 select * from t1 order by i limit 10 Y 0 select * from t1 order by i limit 30 Y 0 select * from t1 order by i limit 10 Y 0 select * from t1 order by i limit 10 Y 0

MySQL Dtrace probes connection-start(id, user, hostname) connection-done(status, id) command-start(id, command_id) command-end(status) query-start(query, id, db, user, host) query-done(status) query-parse-start(query) query-parse-done(status) query-cache-hit(query, row) query-cache-miss(query) query-exec-start(query, id, db, host, exec_type) query-exec-done(status) insert/update/delete-row-start(db, table, status) insert/update/delete-row-done(db, table, status) select/insert/update/multi-update-start(query, status, rows, rowsmatched, rowschanged) select/insert/update/multi-update-done(query, status, rows, rowsmatched, rowschanged)

Sample PHP D Script php*:::compile-file-entry /arg0 != NULL/ { self->compile_file = copyinstr(arg0); self->compile_start_time = timestamp; } php*:::compile-file-return /self->compile_file == copyinstr(arg0)/ { self->compile_elapsed = ((timestamp - self->compile_start_time) / 1000000); @compile_time[self->compile_file] = quantize(self->compile_elapsed); self->compile_file = 0; self->compile_file_translated = 0; self->compile_start_time = 0; } tick-60s { printf(" Exclusive compile elapsed times, "); printa(" %s, %@d ", @compile_time); }

PHP Dtrace probes request-startup(request-file, request-uri, request-method) request-end(request-file, request-uri, request-method) compile-file-entry(compile-file, compile-file-translated) compile-file-return(compile-file, compile-file-translated) function-entry(function-name, request-file, lineno, classname, scope) function-end(function-name, request-file, lineno, classname, scope) execute-entry(request-file, lineno) execute-end(request-file, lineno) error(errormsg, request-file, lineno) execption-thrown(classname) exception-catched(classname)

Ongoing PHP Dtrace work

Probes now integrated within PHP engine – thanks to Wez Furlong and David Soria Parra
More probes planned at these places:

To track GC events.
To track Object create / destroy life cycle.
To track Session life cycle.

Release sample Dtrace scripts to webstack community.

Apache Dtrace Probes accept_connection(uint*) log_request(uint*, char*, char*) receive_request (uint*) create_child() check_user_credentials(uint*) check_access(uint*) check_authorization(uint*)

How to try it out ?