Benchmarking Perl Lightning Talk (NPW 2007)

Slide 1: Benchmarking Perl brian d foy <brian@stonehenge.com> Stonehenge Consulting Ser vices Nordic Perl Workshop 2007 Lightning Talk Edition

Slide 2: Benchmarks are reference points

Slide 3: All Things Being Equal... There are lies, damned lies, and benchmarks Everyone has an agenda You don’t run testbeds as production Skepticism wins the day

Slide 4: Language Shootout Benchmarking programming languages? How can we benchmark a programming language? We can't - we benchmark programming language implementations. How can we benchmark language implementations? We can't - we measure particular programs. http:/ /shootout.alioth.debian.org/

Slide 5: Availability Disk Use Concurrent Users CPU Time Completion Time Memory Use Uptime Bandwidth Use Network Lag Responsiveness Binary Size Programmer Time

Slide 6: Programmer Ti me

Slide 7: Theory of Measurement Obser vation changes the universe Nothing is objective Tools have inherent uncertainities

Slide 8: Modules Devel::Size - size(), total_size() Devel::Peek - mstat() DBI::Profile Benchmark Devel::DProf, Devel::Smallprof

Slide 9: Common Benchmark misuse use Benchmark 'cmpthese'; my @long = ('a' .. 'z', ''); my $iter = shift || -1; cmpthese( $iter,{ long_block_ne => q{grep {$_ ne ''} @long}, long_block_len => q{grep {length} @long}, long_bare_ne => q{grep $_ ne '', @long}, long_bare_len => q{grep length, @long}, } ); http:/ /www.perlmonks.org/index.pl?node_id=536503

Slide 10: What’s wrong with this picture? Rate bare_ne block_len block_ne bare_len long_bare_ne 3635361/s -- -6% -6% -8% long_block_len 3869054/s 6% -- -0% -2% long_block_ne 3872708/s 7% 0% -- -2% long_bare_len 3963159/s 9% 2% 2% --

Slide 11: Do something useful use Benchmark 'cmpthese'; our @long = ('a' .. 'z', ''); my $iter = shift || -1; cmpthese( $iter,{ long_block_ne => q{my @array = grep {$_ ne ''} @long}, long_block_len => q{my @array = grep {length} @long}, long_bare_ne => q{my @array = grep $_ ne '', @long}, long_bare_len => q{my @array = grep length, @long}, } );

Slide 12: These numbers make sense 15” Powerbook G5, Mac OS X.4.5, perl5.8.4 Rate block_ne block_len bare_ne bare_len long_block_ne 31210/s -- -3% -3% -5% long_block_len 32119/s 3% -- -0% -2% long_bare_ne 32237/s 3% 0% -- -2% long_bare_len 32755/s 5% 2% 2% --

Slide 13: Conclusion Decide what is important to you Realize you have bias Report the situation Don’t turn off your brain Make predictions that you can verify