Slides of my talk about the DashProfiler perl module, which enables lightweight always-on performance monitoring for critical sections of code. See http://search.cpan.org/perldoc?DashProfiler

1. DashProfiler
Lightweight Code Instrumentation
Tim.Bunce@pobox.com - July 2008

2. A Problem
A web application ~100K lines of code
Using many external services
If response time goes up... what’s causing it?

3. A Problem
A web application ~100K lines of code
Using many external services
If response time goes up... what’s causing it?
Continuous monitoring in production
Must have very low CPU and I/O cost
Minimal code changes

4. A Typical Approach
package MyNetIO;
sub send_request {
my ($hostname, $request) = @_
...send to $hostname...
}
How much time was spent sending the request?

5. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}

6. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
• Doesn’t record count so can’t produce averages.

7. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
• Doesn’t record count so can’t produce averages.
• Two lines of code. Worse if multiple return statements.

8. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
• Doesn’t record count so can’t produce averages.
• Two lines of code. Worse if multiple return statements.
• Doesn’t record time if function exits via an exception.

9. A Solution: DashProfiler
Simple
Flexible
Lightweight

10. DashProfiler
• Can group samples into granular time units
• Can measure exclusive time in a period
• Can flush to disk at intervals
• Just needs one line of code per sample

11. DashProfiler Internals
Built on DBI::Profile, part of the DBI
Aggregates measurements into a data tree
Two-level tree by default:
$root->{ $key1 }->{ $key2 }->[ ...leaf node... ]
$root->{ ‘MyNetIO’ }->{ $hostname }->[ ...leaf node... ]

12. DashProfiler Data
Each leaf node in the tree is a reference to an array:
$root->{ $key1 }->{ $key2 } = [
106, # 0: count of samples at this node
0.0312958955764771, # 1: total duration
0.000490069389343262, # 2: first duration
0.000176072120666504, # 3: shortest duration
0.00140702724456787, # 4: longest duration
1023115819.83019, # 5: time of first sample
1023115819.86576, # 6: time of last sample
]

13. DashProfiler By-Time
Optional extra time level in the data tree
$time = int(time() / $granularity) * $granularity;
$root->{ $time }->{ ‘MyNetIO’ }->{ $hostname }->[ ... ]
So a new sub-tree is grown each granularity seconds

14. DashProfiler Config
use DashProfiler;
DashProfiler->add_profile( foo => { } );
DashProfiler->add_profile( foo => {
granularity => 10,
flush_interval => 600,
flush_hook => sub { ... },
sample_class => ‘DashProfiler::Sample’,
dbi_profile_class => ‘DBI::Profile’,
period_exclusive => ...,
period_summary => ...,
...
});

15. Without DashProfiler
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}

16. Without DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}

17. With DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope

18. With DashProfiler
Name of profile created with add_profile()
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope

19. With DashProfiler
Name of profile created with add_profile()
Value to use for ‘key1’
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope

20. With DashProfiler
Name of profile created with add_profile()
Value to use for ‘key1’
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
Value to use for ‘key2’
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope

21. With DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname )
if foo_profiler_enabled();
...send to $hostname...
}

22. With DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname )
if foo_profiler_enabled();
...send to $hostname...
Automatically imported
compile-time constant
} reduces cost to zero
if profile is disabled

23. DashProfiler Flush
Data is written to STDERR on exit, by default
Regular flushing is enabled by specifying a flush_interval
The dbi_profile_class handles the flush. Choices include:
DBI::Profile
DBI::ProfileData
DBI::ProfileData::Apache
DashProfiler->add_profile( foo => {
...,
flush_interval => 600,
dbi_profile_class => ‘DBI::ProfileData’,
flush_hook => sub { ... },
...
});

24. DashProfiler Periods
• Group samples into periods
- e.g. http request to response
- start_sample_period() and end_sample_period()
- counted, to enable averages and totals per period
- can output period counts instead of sample counts
• Measure ‘exclusive’ time
- time from period start to end that’s not been
accounted for by other samples
- enabled via period_exclusive option

25. Example Data
Average response times over 24 hours
DashProfiler doesn’t generate graphs itself, but the
data can be used to create graphs like these

26. Example Data
Worst case response times over 24 hours

28. DashProfiler Perspectives
• Each DashProfiler can have multiple DBI
Profile objects attached
• Samples accumulate in all attached profiles
• Each profile can have a different Path
• giving different ‘perspectives’ or level of detail
- key1 + key2
- key1 + country + browser type
- key2 + browser type
- ... etc.

30. DashProfiler Per-Period
• Optional extra ‘per-period’ DBI profile
• Enabled via period_summary option
• Automatically attached and reset by
start_sample_period()
• Gives current totals for this period
• Great for ‘debug footers’ on web page showing
how much time was spent generating this page

31. DashProfiler Cost

32. DashProfiler Cost
Time cost of taking a sample:
0.000022s

34. Questions?