0
DashProfiler
Lightweight Code Instrumentation




Tim.Bunce@pobox.com - July 2008
A Problem

A web application ~100K lines of code
Using many external services
If response time goes up... what’s causing i...
A Problem

A web application ~100K lines of code
Using many external services
If response time goes up... what’s causing i...
A Typical Approach
 package MyNetIO;



 sub send_request {
    my ($hostname, $request) = @_


     ...send to $hostname....
A Typical Approach
package MyNetIO;

use Time::Hires qw(time);

sub send_request {
   my ($hostname, $request) = @_
   my ...
A Typical Approach
   package MyNetIO;

   use Time::Hires qw(time);

   sub send_request {
      my ($hostname, $request)...
A Typical Approach
   package MyNetIO;

   use Time::Hires qw(time);

   sub send_request {
      my ($hostname, $request)...
A Typical Approach
   package MyNetIO;

   use Time::Hires qw(time);

   sub send_request {
      my ($hostname, $request)...
A Solution: DashProfiler

        Simple
        Flexible
      Lightweight
DashProfiler

• Can group samples into granular time units

• Can measure exclusive time in a period

• Can flush to disk at...
DashProfiler Internals

Built on DBI::Profile, part of the DBI

Aggregates measurements into a data tree

Two-level tree by ...
DashProfiler Data
Each leaf node in the tree is a reference to an array:

$root->{ $key1 }->{ $key2     }   = [
    106,   ...
DashProfiler By-Time

Optional extra time level in the data tree

  $time = int(time() / $granularity) * $granularity;

  $...
DashProfiler Config
use DashProfiler;

DashProfiler->add_profile( foo => { } );

DashProfiler->add_profile( foo => {
   gran...
Without DashProfiler

package MyNetIO;

use Time::Hires qw(time);

sub send_request {
   my ($hostname, $request) = @_
   m...
Without DashProfiler

package MyNetIO;

use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];

sub send_request {
   my (...
With DashProfiler

package MyNetIO;

use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];

sub send_request {
   my ($ho...
With DashProfiler
                   Name of profile created with add_profile()


package MyNetIO;

use DashProfiler::Import...
With DashProfiler
                   Name of profile created with add_profile()

                                      Value...
With DashProfiler
                   Name of profile created with add_profile()

                                      Value...
With DashProfiler

package MyNetIO;

use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];

sub send_request {
   my ($ho...
With DashProfiler

package MyNetIO;

use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];

sub send_request {
   my ($ho...
DashProfiler Flush
Data is written to STDERR on exit, by default
Regular flushing is enabled by specifying a flush_interval
...
DashProfiler Periods
• Group samples into periods
  - e.g. http request to response
  -   start_sample_period() and end_sam...
Example Data
Average response times over 24 hours




           DashProfiler doesn’t generate graphs itself, but the
     ...
Example Data
Worst case response times over 24 hours
DashProfiler Perspectives
• Each DashProfiler can have multiple DBI
    Profile objects attached
•   Samples accumulate in al...
DashProfiler Per-Period
• Optional extra ‘per-period’ DBI profile
• Enabled via period_summary option
• Automatically attach...
DashProfiler Cost
DashProfiler Cost

Time cost of taking a sample:

     0.000022s
Questions?
DashProfiler 200807
DashProfiler 200807
DashProfiler 200807
Upcoming SlideShare
Loading in...5
×

DashProfiler 200807

1,025

Published on

Slides of my talk about the DashProfiler perl module, which enables lightweight always-on performance monitoring for critical sections of code. See
http://search.cpan.org/perldoc?DashProfiler

Published in: Technology, Art & Photos
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,025
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Still need to write code to flush
  • Still need to write code to flush
  • Still need to write code to flush
  • Still need to write code to flush
  • First sample populates all
    Later samples always update 0, 1, and 6
    and may update 3 or 4
  • Create named profiles

    Lots of features
  • DashProfiler::Import imports a pre-curried profiler code ref
    Profilers return bless object containing timestamp
    Object destruction triggers accumulation of sample
  • Time to create sample object, destroy it, accumulate the counts
    In hot code can be 0.000017s
    C version of sampler class should more than half the cost
  • Time to create sample object, destroy it, accumulate the counts
    In hot code can be 0.000017s
    C version of sampler class should more than half the cost
  • Transcript of "DashProfiler 200807"

    1. 1. DashProfiler Lightweight Code Instrumentation Tim.Bunce@pobox.com - July 2008
    2. 2. A Problem A web application ~100K lines of code Using many external services If response time goes up... what’s causing it?
    3. 3. A Problem A web application ~100K lines of code Using many external services If response time goes up... what’s causing it? Continuous monitoring in production Must have very low CPU and I/O cost Minimal code changes
    4. 4. A Typical Approach package MyNetIO; sub send_request { my ($hostname, $request) = @_ ...send to $hostname... } How much time was spent sending the request?
    5. 5. A Typical Approach package MyNetIO; use Time::Hires qw(time); sub send_request { my ($hostname, $request) = @_ my $start = time(); ...send to $hostname... $durations->{MyNetIO}{$hostname} = time() - $start; }
    6. 6. A Typical Approach package MyNetIO; use Time::Hires qw(time); sub send_request { my ($hostname, $request) = @_ my $start = time(); ...send to $hostname... $durations->{MyNetIO}{$hostname} = time() - $start; } • Doesn’t record count so can’t produce averages.
    7. 7. A Typical Approach package MyNetIO; use Time::Hires qw(time); sub send_request { my ($hostname, $request) = @_ my $start = time(); ...send to $hostname... $durations->{MyNetIO}{$hostname} = time() - $start; } • Doesn’t record count so can’t produce averages. • Two lines of code. Worse if multiple return statements.
    8. 8. A Typical Approach package MyNetIO; use Time::Hires qw(time); sub send_request { my ($hostname, $request) = @_ my $start = time(); ...send to $hostname... $durations->{MyNetIO}{$hostname} = time() - $start; } • Doesn’t record count so can’t produce averages. • Two lines of code. Worse if multiple return statements. • Doesn’t record time if function exits via an exception.
    9. 9. A Solution: DashProfiler Simple Flexible Lightweight
    10. 10. DashProfiler • Can group samples into granular time units • Can measure exclusive time in a period • Can flush to disk at intervals • Just needs one line of code per sample
    11. 11. DashProfiler Internals Built on DBI::Profile, part of the DBI Aggregates measurements into a data tree Two-level tree by default: $root->{ $key1 }->{ $key2 }->[ ...leaf node... ] $root->{ ‘MyNetIO’ }->{ $hostname }->[ ...leaf node... ]
    12. 12. DashProfiler Data Each leaf node in the tree is a reference to an array: $root->{ $key1 }->{ $key2 } = [ 106, # 0: count of samples at this node 0.0312958955764771, # 1: total duration 0.000490069389343262, # 2: first duration 0.000176072120666504, # 3: shortest duration 0.00140702724456787, # 4: longest duration 1023115819.83019, # 5: time of first sample 1023115819.86576, # 6: time of last sample ]
    13. 13. DashProfiler By-Time Optional extra time level in the data tree $time = int(time() / $granularity) * $granularity; $root->{ $time }->{ ‘MyNetIO’ }->{ $hostname }->[ ... ] So a new sub-tree is grown each granularity seconds
    14. 14. DashProfiler Config use DashProfiler; DashProfiler->add_profile( foo => { } ); DashProfiler->add_profile( foo => { granularity => 10, flush_interval => 600, flush_hook => sub { ... }, sample_class => ‘DashProfiler::Sample’, dbi_profile_class => ‘DBI::Profile’, period_exclusive => ..., period_summary => ..., ... });
    15. 15. Without DashProfiler package MyNetIO; use Time::Hires qw(time); sub send_request { my ($hostname, $request) = @_ my $start = time(); ...send to $hostname... $durations->{MyNetIO}{$hostname} = time() - $start; }
    16. 16. Without DashProfiler package MyNetIO; use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ]; sub send_request { my ($hostname, $request) = @_ my $sample = foo_profiler( $hostname ); ...send to $hostname... }
    17. 17. With DashProfiler package MyNetIO; use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ]; sub send_request { my ($hostname, $request) = @_ my $sample = foo_profiler( $hostname ); ...send to $hostname... } Duration is measured when $sample goes out of scope
    18. 18. With DashProfiler Name of profile created with add_profile() package MyNetIO; use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ]; sub send_request { my ($hostname, $request) = @_ my $sample = foo_profiler( $hostname ); ...send to $hostname... } Duration is measured when $sample goes out of scope
    19. 19. With DashProfiler Name of profile created with add_profile() Value to use for ‘key1’ package MyNetIO; use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ]; sub send_request { my ($hostname, $request) = @_ my $sample = foo_profiler( $hostname ); ...send to $hostname... } Duration is measured when $sample goes out of scope
    20. 20. With DashProfiler Name of profile created with add_profile() Value to use for ‘key1’ package MyNetIO; use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ]; sub send_request { Value to use for ‘key2’ my ($hostname, $request) = @_ my $sample = foo_profiler( $hostname ); ...send to $hostname... } Duration is measured when $sample goes out of scope
    21. 21. With DashProfiler package MyNetIO; use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ]; sub send_request { my ($hostname, $request) = @_ my $sample = foo_profiler( $hostname ) if foo_profiler_enabled(); ...send to $hostname... }
    22. 22. With DashProfiler package MyNetIO; use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ]; sub send_request { my ($hostname, $request) = @_ my $sample = foo_profiler( $hostname ) if foo_profiler_enabled(); ...send to $hostname... Automatically imported compile-time constant } reduces cost to zero if profile is disabled
    23. 23. DashProfiler Flush Data is written to STDERR on exit, by default Regular flushing is enabled by specifying a flush_interval The dbi_profile_class handles the flush. Choices include: DBI::Profile DBI::ProfileData DBI::ProfileData::Apache DashProfiler->add_profile( foo => { ..., flush_interval => 600, dbi_profile_class => ‘DBI::ProfileData’, flush_hook => sub { ... }, ... });
    24. 24. DashProfiler Periods • Group samples into periods - e.g. http request to response - start_sample_period() and end_sample_period() - counted, to enable averages and totals per period - can output period counts instead of sample counts • Measure ‘exclusive’ time - time from period start to end that’s not been accounted for by other samples - enabled via period_exclusive option
    25. 25. Example Data Average response times over 24 hours DashProfiler doesn’t generate graphs itself, but the data can be used to create graphs like these
    26. 26. Example Data Worst case response times over 24 hours
    27. 27. DashProfiler Perspectives • Each DashProfiler can have multiple DBI Profile objects attached • Samples accumulate in all attached profiles • Each profile can have a different Path • giving different ‘perspectives’ or level of detail - key1 + key2 - key1 + country + browser type - key2 + browser type - ... etc.
    28. 28. DashProfiler Per-Period • Optional extra ‘per-period’ DBI profile • Enabled via period_summary option • Automatically attached and reset by start_sample_period() • Gives current totals for this period • Great for ‘debug footers’ on web page showing how much time was spent generating this page
    29. 29. DashProfiler Cost
    30. 30. DashProfiler Cost Time cost of taking a sample: 0.000022s
    31. 31. Questions?
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×