It802 bruning

3,252 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,252
On SlideShare
0
From Embeds
0
Number of Embeds
1,673
Actions
Shares
0
Downloads
32
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

It802 bruning

  1. 1. Using DTrace to InstrumentYour System Max Bruning,Training Director, Joyent Wednesday, March 26, 14
  2. 2. Topics Covered • DTrace • What it does. • Uses • How Instruments uses it • Examples Wednesday, March 26, 14
  3. 3. Tools Used in This Talk • DTrace • Allows runtime observability into entire software stack • Instruments • performance, analysis, and testing tool for dynamically tracing and profiling OS X and iOS code • Many of the instruments tools use DTrace (some use llvm) Wednesday, March 26, 14
  4. 4. What is DTrace? • Tool that allows one to dynamically instrument code from application level and into the kernel. • Can be used safely on production systems. • Uses: • Performance Analysis • Debugging • Code coverage • Find out what is happening in your software • Available on OS X, SmartOS, illumos, *BSD, Solaris and derivatives. • Useful with C, Objective-C, C++, as well as higher level languages (java, perl, node, python, etc.). Wednesday, March 26, 14
  5. 5. Terminology •Probe - An instrumentation point in the code •Dynamic and Static probes are provided, and new ones can be added •A probe is specified by a 4-tuple: •provider:module:function:probename{action} •Action - Executed when a probe fires •Predicate - Optional boolean to determine whether or not to execute the action •Example: syscall::read:entry/pid == 713/{trace();} Wednesday, March 26, 14
  6. 6. Some Simple Examples •Show system calls made by a running process # dtrace -n ‘syscall:::entry/pid==26442/{}’ dtrace: description 'syscall:::entry' matched 234 probes CPU ID FUNCTION:NAME 1 10157 write:entry 1 10155 read:entry 4 10155 read:entry ... # dtrace -n 'syscall:::entry/execname == "Safari"/{@[probefunc]=count();}' dtrace: description 'syscall:::entry' matched 456 probes (^C) ... mmap 183 psynch_mutexdrop 470 psynch_mutexwait 470 kevent64 6149 workq_kernreturn 6326 • Count system calls made by a running safari process Wednesday, March 26, 14
  7. 7. An Example Measuring System Call Latency•systime.d #!/usr/sbin/dtrace -s #pragma D option quiet syscall:::entry /execname == "node"/ { self->ts = timestamp; } syscall:::return /self->ts/ { @[probefunc] = quantize(timestamp - self->ts); self->ts = 0; } END { ! printa("SYSCALL NSECS # OF OCCURANCESn%s%@lxn", @); } Wednesday, March 26, 14
  8. 8. An Example Measuring System Call Latency (Continued)# ./systime.d ... SYSCALL NSECS # OF OCCURANCES read value ------------- Distribution ------------- count 1024 | 0 2048 |@@@@@@@@@@@@@ 3 4096 |@@@@@@@@@ 2 8192 |@@@@@@@@@ 2 16384 |@@@@ 1 32768 | 0 ... 33554432 | 0 67108864 | 0 134217728 |@@@@ 1 268435456 | 0 ... Wednesday, March 26, 14
  9. 9. DTrace Providers for OS X # dtrace -l | grep -v PROVIDER | awk '{print $2}' | sort -u | sed -e 's/[0-9].*//' | uniq | paste - - - - | awk '{ printf "%20-s %20-s %20-s %20-sn", $1, $2, $3, $4 }' AppContainer AppSandbox Cocoa_Autorelease Cocoa_Layout CoreData CoreImage CoreText HALC_IO HALS_Engine HALS_IO HALS_System JavaScriptCore MetadataFramework MetadataImporter MobileDevice NSApplication NSTrackingArea NSXPCConnection NSXPCListener NSXPCProxy PDEProbes PMTiming PackageKit PrintCore QLSeamlessClosing QLSeamlessOpening QLThumbnail QuartzComposer QuickTimeX RawCamera RemoteViewServices SoftwareUpdate Suggestions afc cache codesign dtrace fbt garbage_collection io ip ldap_rb_stats lockstat mach_trap magmalloc mds node objc_runtime odprobes opencl_api opencl_cpu perl pid plockstat proc profile sched sdt security_debug security_exception security_machserver securityd syscall syspolicy tcp vminfo • Providers not in grey are OS X specific (?) • More providers may exist, and more can be added Wednesday, March 26, 14
  10. 10. Instruments • Performance, analysis, and testing tool for dynamically tracing and profiling OS X and iOS code • Many of the instruments tools use DTrace (some use llvm) • Instruments provides a GUI for implementing your own instruments using DTrace Wednesday, March 26, 14
  11. 11. Instruments and DTrace Instruments using Dtrace Sudden termination, Carbon Events, Core Data Saves/ Fetches/Faults, Cocoa Events/Layout, File Locks/Attributes, File Activity, Directory I/O, Garbage Collection, Reads/ Writes, Shared Memory, Process, Custom Instruments Instruments that Could be implemented using DTrace Time Profiler, Network Activity Monitor, Sampler, Memory/ Disk/CPU/Activity Monitor,Thread States,VM Operations, System Calls, Scheduling Wednesday, March 26, 14
  12. 12. Example Instrument - File Activity Wednesday, March 26, 14
  13. 13. Example Instrument - File Activity (Continued) • Edit as needed and save or cancel. Wednesday, March 26, 14
  14. 14. Example Instrument - File Activity (Continued) Wednesday, March 26, 14
  15. 15. Example Instrument - DTrace Script$ cat fileact.d #!/bin/bash # # To run this script, please type the following command while in the same # directory (or specify the full path rather than './'): # # sudo ./fileact.d -o dtrace_output.txt # # After the output file has been generated (here, it's dtrace_output.txt but you # may call it whatever you like), call up the trace document where # this script was exported from and choose the "DTrace Data Import..." option # from the "File" pulldown menu to import that data. # TOOLCHAINS=default /usr/sbin/dtrace $* -C -Z -s /dev/stdin << "End-Of-File" #pragma D option switchrate=1msec ... Wednesday, March 26, 14
  16. 16. Example Instrument - Running the D Script $ sudo ./fileact.d -o fileact.out •Run your workload while this is running •When ready, control-c or kill •Import fileact.out into Instruments •Modifying the script outside of Instruments may cause output to not be importable •Importing of arbitrary D script output into Instruments does not work Wednesday, March 26, 14
  17. 17. Example Instrument - D Script Output <dtrace_output_begin/> 1395363306364638000 3A85E3B6-1514-4ECC-8EC8-40FB699DCAF9 <e> -1 4 2 1246115 1395363306366033000 fstat64 dtrace 6 </e> <s> libsystem_kernel.dylib`fstat$INODE64+0xa libsystem_c.dylib`__smakebuf+0x2a libsystem_c.dylib`__swsetup+0x84 libsystem_c.dylib`__v2printf+0x55 libsystem_c.dylib`__xvprintf+0x279 libsystem_c.dylib`vfprintf+0x49 libdtrace.dylib`dt_printf+0xf9 libdtrace.dylib`dt_printf_format+0x19c libdtrace.dylib`dtrace_fprintf+0x33 libdtrace.dylib`dt_consume_cpu+0xc47 libdtrace.dylib`dtrace_consume+0x3cb libdtrace.dylib`dtrace_work+0x7a dtrace`main+0x1c9c libdyld.dylib`start+0x1 dtrace`0x7 </s> <e> -1 6 6 1246115 1395363306367047000 stat64 dtrace /Library/Symbols </e> <s> libsystem_kernel.dylib`stat$INODE64+0xa ... Wednesday, March 26, 14
  18. 18. Example D Script That “mimics” Time Profiler #!/usr/sbin/dtrace -qs BEGIN { printf("%-24s %-8s %-16s %-24s %-24sn", "Timestamp", "Depth", "Process", "Hot Frame", "Caller"); } profile-1ms /arg0/ { printf("%-24Y %-8d %-16s %-24p %-24pn", walltimestamp, stackdepth, execname, arg0, caller); stack(); } profile-1ms /arg1/ { printf("%-24Y %-8d %-16s %-24p %-24pn", walltimestamp, stackdepth, execname, arg1, caller); ustack(); } Wednesday, March 26, 14
  19. 19. Output from Profiler D Script $ sudo ./prof1.d > prof1.out ^C bash-3.2$ ls -l prof1.out -rw-r--r-- 1 max staff 33694762 Mar 21 11:20 prof1.out bash-3.2$ less prof1.out Timestamp Depth Process Hot Frame Caller 2014 Mar 21 11:20:04 6 dtrace ffffff80290d41b2 ffffff80291f3ae1 mach_kernel`0xffffff80290d4060+0x152 mach_kernel`VNOP_IOCTL+0x150 mach_kernel`0xffffff80291f39b0+0x131 mach_kernel`fo_ioctl+0x43 mach_kernel`ioctl+0x466 mach_kernel`unix_syscall64+0x1f3 mach_kernel`hndl_unix_scall64+0x16 2014 Mar 21 11:20:04 1 kernel_task ffffff80290dd76e 0 mach_kernel`machine_idle+0x1de mach_kernel`call_continuation+0x17 ... Wednesday, March 26, 14
  20. 20. DTrace Scripts Available on OS X$ man -k dtrace bitesize.d(1m) - analyse disk I/O size by process. Uses DTrace cpuwalk.d(1m) - Measure which CPUs a process runs on. Uses DTrace creatbyproc.d(1m) - snoop creat()s by process name. Uses DTrace dappprof(1m) - profile user and lib function usage. Uses DTrace dapptrace(1m) - trace user and library function usage. Uses DTrace diskhits(1m) - disk access by file offset. Uses DTrace dispqlen.d(1m) - dispatcher queue length by CPU. Uses DTrace dtrace(1) - generic front-end to the DTrace facility dtruss(1m) - process syscall details. Uses DTrace errinfo(1m) - print errno for syscall fails. Uses DTrace execsnoop(1m) - snoop new process execution. Uses DTrace fddist(1m) - file descriptor usage distributions. Uses DTrace filebyproc.d(1m) - snoop opens by process name. Uses DTrace hotspot.d(1m) - print disk event by location. Uses DTrace ... •42 scripts available, and more can be found at http://www.dtracebook.com/index.php/Main_Page Wednesday, March 26, 14
  21. 21. Example - dtruss • Trace system calls $ sudo dtruss -n Mail PID/THRD SYSCALL(args) = return 3113/0x47af: select(0x60, 0x61000020C600, 0x61000020C680, 0x0, 0x0) = 1 0 3113/0x47af: recvfrom(0x22, 0x1132ADAA0, 0x100) = 1 0 3113/0x1a3059: kevent64(0x3, 0x11623DE18, 0x1) = 1 0 3113/0x46b0: workq_kernreturn(0x20, 0x0, 0x1) = 0 0 3113/0x46b0: kevent64(0x3, 0x0, 0x0) = 1 0 3113/0x179187: fcntl(0x8, 0x8, 0x117BB8900) = 0 0 3113/0x179187: fcntl(0x8, 0x8, 0x117BB8920) = 0 0 3113/0x179187: fcntl(0x8, 0x8, 0x117BB8A60) = 0 0 3113/0x179187: pread(0x6, "r0", 0x1000, 0xC8C0000) = 4096 0 3113/0x179187: workq_kernreturn(0x20, 0x0, 0x1) = 0 0 3113/0x179187: pwrite(0x7, "0", 0x1018, 0x3DFC90) = 4120 0 3113/0x179187: pwrite(0x7, "0", 0x1018, 0x3E0CA8) = 4120 0 3113/0x1a28d7: open("/Users/max/Library/Mail/V2/IMAP-max@joyent.com@imap.gmail.com/ [Gmail].mbox/All Mail.mbox/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/Data/8/7/1/Messages/ 178177.emlx0", 0x8000, 0x0) = 46 0 ... Wednesday, March 26, 14
  22. 22. Example - iosnoop • Trace Disk I/O Events $ sudo iosnoop -a STRTIME DEVICE MAJ MIN UID PID D BLOCK SIZE PATHNAME ARGS 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132761912 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132761952 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132761976 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762576 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762840 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132763224 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132763320 8192 ??/Messages/177915.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132764944 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132765360 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132765472 12288 ??/Messages/177917.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132767000 8192 ??/Messages/177918.emlx Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132767152 16384 ??/Messages/177924.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762608 12288 ??/Messages/177913.emlx Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132764920 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132767016 8192 ??/Messages/177919.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762896 12288 ??/Messages/177914.emlx Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132765384 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132766960 20480 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 0 128 R 24160560 4096 ??/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/ reverseDirectoryStore mds_stores0 ... Wednesday, March 26, 14
  23. 23. Example - opensnoop • Trace open(2) calls on files $ sudo opensnoop -a TIME STRTIME UID PID FD ERR PATH ARGS 276685505483 2014 Mar 21 15:39:59 0 128 5 0 . mds_stores0 276685505676 2014 Mar 21 15:39:59 0 128 5 0 . mds_stores0 276688258585 2014 Mar 21 15:40:02 502 17327 10 0 /Users/max/Library/Saved Application State/org.gnu.Aquamacs.savedState/data.data Aquamacs0 276688258829 2014 Mar 21 15:40:02 502 17327 10 0 /Users/max/Library/Saved Application State/org.gnu.Aquamacs.savedState/windows.plist Aquamacs0 276688259202 2014 Mar 21 15:40:02 502 17327 10 0 /Users/max/Library/Saved ... Wednesday, March 26, 14
  24. 24. Example - hfsslower • Trace slow reads/writes on hfs File System (from DTrace book) $ sudo ./dtbook_scripts/Chap5/hfsslower.d -x dynvarsize=25m 1 TIME PROCESS D KB ms FILE 2014 Mar 22 14:09:48 mds W 0 3 journal.1 2014 Mar 22 14:09:50 mdworker R 0 4 Info.plist/..namedfork/rsrc 2014 Mar 22 14:10:42 bash R 0 4 ps/..namedfork/rsrc 2014 Mar 22 14:12:50 vmware-vmx W 15 4 Virtual Disk-s001.vmdk 2014 Mar 22 14:12:50 vmware-vmx W 1 4 Virtual Disk-s001.vmdk 2014 Mar 22 14:12:50 vmware-vmx W 5 4 Virtual Disk-s001.vmdk 2014 Mar 22 14:13:19 Mail W 1325 4 156241.emlx.tmp 2014 Mar 22 14:13:58 fontd R 0 4 GillSans.ttc/..namedfork/rsrc 2014 Mar 22 14:14:09 Mail W 1912 7 156197.emlx.tmp 2014 Mar 22 14:14:21 Mail R 1913 2 156197.emlx 2014 Mar 22 14:14:21 Keynote W 0 4 data.data 2014 Mar 22 14:14:26 AppleSpell R 0 4 findNames/..namedfork/rsrc 2014 Mar 22 14:14:34 mdworker R 0 4 Info.plist/..namedfork/rsrc 2014 Mar 22 14:14:49 mds W 0 3 journal.1 2014 Mar 22 14:14:56 Keynote R 256 1 using_dtrace.key Wednesday, March 26, 14
  25. 25. Example - soconnect_mac.d • Trace connect(2) calls (from DTrace book) $ sudo ./dtbook_scripts/Chap6/soconnect_mac.d Password: PID PROCESS FAM ADDRESS PORT LAT(us) RESULT dtrace: error on enabled probe ID 2 (ID 1035: syscall::connectx:entry): invalid address (0x0) in action #1 at DIF offset 32 166 acwebsecagent 2 XXX.XXX.XXX.XXX 80 41 In progress 166 acwebsecagent 2 XXX.XXX.XXX.XXX 80 48 In progress dtrace: error on enabled probe ID 2 (ID 1035: syscall::connectx:entry): invalid address (0x0) in action #1 at DIF offset 32 dtrace: error on enabled probe ID 2 (ID 1035: syscall::connectx:entry): invalid address (0x0) in action #1 at DIF offset 32 166 acwebsecagent 2 XXX.XXX.XXX.XXX 80 63 In progress dtrace: error on enabled probe ID 2 (ID 1035: syscall::connectx:entry): invalid address (0x0) in action #1 at DIF offset 3 ... Wednesday, March 26, 14
  26. 26. Example - soconnect_mac.d (Continued) • What is “dtrace: error on enabled probe ID 2 (ID 1035: syscall::connectx:entry): invalid address (0x0) in action #1 at DIF offset 32” ? • probe ID 2 is the second probe: syscall::connect*:entry { /* assume this is sockaddr_in until we can examine family */ this->s = (struct sockaddr_in *)copyin(arg1, sizeof (struct sockaddr)); this->f = this->s->sin_family; } Wednesday, March 26, 14
  27. 27. Example - soconnect_mac.d (Continued) • What is arg1 to connectx()? $ sudo dtrace -n 'syscall::connectx:entry{printf("arg0 = %x arg1 = %x arg2 = %x arg3 = %x arg4 = %xn", arg0, arg1, arg2, arg3, arg4);}' dtrace: description 'syscall::connectx:entry' matched 1 probe CPU ID FUNCTION:NAME 2 1035 connectx:entry arg0 = 14 arg1 = 0 arg2 = 0 arg3 = 7fc42bd1fd10 arg4 = 10 4 1035 connectx:entry arg0 = 1a arg1 = 0 arg2 = 0 arg3 = 7fc42be471a0 arg4 = 10 0 1035 connectx:entry arg0 = 10 arg1 = 0 arg2 = 0 arg3 = 11b5380e0 arg4 = 10 7 1035 connectx:entry arg0 = 14 arg1 = 0 arg2 = 0 arg3 = 7fc429725bf0 arg4 = 10 ... Wednesday, March 26, 14
  28. 28. Example - soconnect_mac.d (Continued) • What is arg3 to connectx()? • Assume it is a sockaddr_in. #!/usr/sbin/dtrace -qs inline int af_inet = 2; dtrace:::BEGIN { printf("%-6s %-16s %-3s %-16s %-5s %8sn", "PID", "PROCESS", "FAM", "ADDRESS", "PORT", "LAT(us)"); } syscall::connectx:entry { this->s = (struct sockaddr_in *)copyin(arg3, sizeof (struct sockaddr)); this->f = this->s->sin_family; self->inconnectx = 1; } Wednesday, March 26, 14
  29. 29. Example - soconnect_mac.d (Continued) syscall::connectx:entry /this->f == af_inet/ { self->family = this->f; self->port = (this->s->sin_port & 0xFF00) >> 8; self->port |= (this->s->sin_port & 0xFF) << 8; this->a = (uint8_t *)&this->s->sin_addr; this->addr1 = strjoin(lltostr(this->a[0] + 0ULL), strjoin(".", strjoin(lltostr(this->a[1] + 0ULL), "."))); this->addr2 = strjoin(lltostr(this->a[2] + 0ULL), strjoin(".", strjoin(lltostr(this->a[1] + 0ULL), "."))); this->addr2 = strjoin(lltostr(this->a[2] + 0ULL), strjoin(".", lltostr(this->a[3] + 0ULL))); self->address = strjoin(this->addr1, this->addr2); self->start = timestamp; } Wednesday, March 26, 14
  30. 30. Example - soconnect_mac.d (Continued) syscall::connectx:return /self->start/ { this->delta = (timestamp - self->start) / 1000; printf("%-6d %-16s %-3d %-16s %-5d %8dn", pid, execname, self->family, self->address, self->port, this->delta); self->family = 0; self->address = 0; self->port = 0; self->start = 0; } Wednesday, March 26, 14
  31. 31. Example - soconnect_mac.d (Continued) $ sudo ./connectx.d PID PROCESS FAM ADDRESS PORT LAT(us) 8801 com.apple.WebKi 2 XXX.XXX.XXX.XXX 80 41 8801 com.apple.WebKi 2 XXX.XXX.XXX.XXX 80 34 8801 com.apple.WebKi 2 XXX.XXX.XXX.XXX 80 43 21473 ocspd 2 XXX.XXX.XXX.XXX 80 97 8801 com.apple.WebKi 2 XXX.XXX.XXX.XXX 80 41 ... Wednesday, March 26, 14
  32. 32. Flame Graphs • Visualization of profiled code paths • Shows distribution of code paths over time • https://github.com/brendangregg/FlameGraph/ Wednesday, March 26, 14
  33. 33. An Example Flame Graph •Colors have no meaning •Left to right is not time sequenced •Moving cursor over item shows function, number of times that function occurs , and percentage. •Longer horizontal bands implies longer time in that function or a function called from that function Wednesday, March 26, 14
  34. 34. Building Flame Graphs Example - DTrace Script#!/usr/sbin/dtrace -s #pragma D option quiet profile:::profile-97hz/arg0 != 0/ { @k[stack()] = count(); } profile:::profile-97hz/arg1 != 0 && execname != "dtrace"/ { @u[ustack()] = count(); } tick-60s {exit(0);} Wednesday, March 26, 14
  35. 35. Building Flame Graphs Example (Continued) $ git clone https://github.com/brendangregg/FlameGraph/ $ sudo ./prof.d -o prof.out $ ls -l prof3.out -rw-r--r-- 1 root staff 2286203 Mar 24 11:35 prof.out $ ./stackcollapse.pl prof.out | ./flamegraph.pl > prof.svg $ open prof3.svg $ •Alternatively $ git clone https://github.com/davepacheco/node-stackvis ... $ cd node-stackvis $ make ... $ stackvis dtrace flamegraph-svg < prof.out > prof1.svg Wednesday, March 26, 14
  36. 36. Node.js - Example • Node.js (and other higher level languages) has probes built in. • Some Node.js modules also have DTrace support • Example is from https://github.com/mcavage/node-restify, (git clone and then npm install) Wednesday, March 26, 14
  37. 37. Node.js - Example Continued /* restifyserver.js */ var restify = require('restify'); var server = restify.createServer({ name: 'myapp', version: '1.0.0' }); server.use(restify.acceptParser(server.acceptable)); server.use(restify.queryParser()); server.use(restify.bodyParser()); server.get('/echo/:name', function (req, res, next) { res.send(req.params); return next(); }); server.listen(8080, function () { console.log('%s listening at %s', server.name, server.url); }); Wednesday, March 26, 14
  38. 38. Node.js - Example Continued /* restifyclient.js */ var assert = require('assert'); var restify = require('restify'); var client = restify.createJsonClient({ url: 'http://localhost:8080', version: '~1.0' }); client.get('/echo/mark', function (err, req, res, obj) { assert.ifError(err); console.log('Server returned: %j', obj); }); Wednesday, March 26, 14
  39. 39. Node.js Example - Continued $ node restifyserver.js myapp listening at http://0.0.0.0:8080 $ sudo dtrace -q -l -n 'restify*:::{}' ID PROVIDER MODULE FUNCTION NAME 214546 restify28832 mod-0x100f00420 route-start route-start 214547 restify28832 mod-0x100f00420 handler-start handler-start 214548 restify28832 mod-0x100f00420 handler-done handler-done 214549 restify28832 mod-0x100f00420 route-done route-done 214550 restify28832 mod-0x100f00420 client-request client- request 214551 restify28832 mod-0x100f00420 client-response client- response 217839 restify28832 mod-0x100f00420 client-error client-error Wednesday, March 26, 14
  40. 40. Node.js Example - Continued $ sudo dtrace -l -n 'node*:::{}' ID PROVIDER MODULE FUNCTION NAME 214536 node28832 node _ZN4nodeL14dtrace_gc_doneEN2v86GCTypeENS0_15GCCallbackFlagsE [node::dtrace_gc_done(v8::GCType, v8::GCCallbackFlags)] gc-done 214537 node28832 node _ZN4nodeL15dtrace_gc_startEN2v86GCTypeENS0_15GCCallbackFlagsE [node::dtrace_gc_start(v8::GCType, v8::GCCallbackFlags)] gc-start 214538 node28832 node _ZN4node26DTRACE_HTTP_CLIENT_REQUESTERKN2v89ArgumentsE [node::DTRACE_HTTP_CLIENT_REQUEST(v8::Arguments const&)] http-client-request 214539 node28832 node _ZN4node27DTRACE_HTTP_CLIENT_RESPONSEERKN2v89ArgumentsE [node::DTRACE_HTTP_CLIENT_RESPONSE(v8::Arguments const&)] http-client-response 214540 node28832 node _ZN4node26DTRACE_HTTP_SERVER_REQUESTERKN2v89ArgumentsE [node::DTRACE_HTTP_SERVER_REQUEST(v8::Arguments const&)] http-server-request ... Wednesday, March 26, 14
  41. 41. Node.js Example - Continued $ node ./restifyclient.js Server returned: {"name":"mark"}$ sudo dtrace -qn 'restify*:::{printf("%s: %s: %sn", probeprov, probefunc, probename);}' restify28832: route-start: route-start restify28832: handler-start: handler-start restify28832: handler-done: handler-done restify28832: handler-start: handler-start restify28832: handler-done: handler-done restify28832: handler-start: handler-start restify28832: handler-done: handler-done restify28832: handler-start: handler-start restify28832: handler-done: handler-done restify28832: handler-start: handler-start restify28832: handler-done: handler-done restify28832: route-done: route-done restify28844: client-response: client-response restify28844: client-request: client-request Wednesday, March 26, 14
  42. 42. What’s Missing? • ustack helpers - Allows jstack() action (stack backtrace showing function calls in higher level languages) • CPU performance counters? • Some source Wednesday, March 26, 14
  43. 43. References • http://www.dtrace.org • http://dtrace.org/blogs/dap/category/nodejs/ • http://dtrace.org/blogs/brendan/2011/10/10/top-10-dtrace-scripts-for- mac-os-x/ @mrbruning, max@joyent.com Wednesday, March 26, 14

×