Top 10 Perl Performance Tips


Published on

This talk was presented at YAPC::NA 2010 and OSCON 2010.

Published in: Technology
1 Comment
  • I'd like to see numbers on the SQL stuff.

    Also, when I use DBIx::Class, I develop with DBIC_TRACE=1 so I know how insane the code is and I can EXPLAIN it to make sure my database it set up correctly.

    Nice summary!
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Top 10 Perl Performance Tips

  1. 1. Top 10 Perl Performance Tips Perrin Harkins We Also Walk Dogs
  2. 2. Devel::NYTProf
  3. 3. Ground Rules● Make a repeatable test to measure progress with ○ Sometimes turns up surprises● Use a profiler (Devel::NYTProf) to find where the time is going ○ Dont flail and waste time optimizing the wrong things!● Try to weigh the cost of developer time vs buying more hardware ○ Optimization is crack for developers, hard to know when to stop
  4. 4. 1. The Big Picture● The biggest gains usually come from changing your high- level approach ○ Is there a more efficient algorithm? ○ Can you restructure to reduce duplicated effort?● Sometimes you just need to tune your SQL● A boatload of RAM hides a multitude of sins● The bottleneck is usually I/O ○ Files ○ Database ○ Network ○ Batch I/O often makes a huge difference
  5. 5. 2. Use DBI Efficiently● Can make a huge difference in tight loops with many small queries● connect_cached() avoids connection overhead ○ Or use your favorite connection cache, but beware overuse of ping()● prepare_cached() avoids object creation and server-side prepare overhead● Use bind parameters to reuse SQL statements instead of creating new ones
  6. 6. 2. Use DBI Efficiently● Use bind_cols() in a fetch() loop for most efficient retrieval. ○ Less copying is faster. ○ Alternatively, fetchrow_arrayref()● prepare() and then many execute() calls is faster than do()
  7. 7. 2. Use DBI Efficiently● Turn off AutoCommit for batch changes ○ Commit every thousand rows or so saves work for your database● Use your databases bulk loader when possible ○ Writing rows to CSV and using MySQLs LOAD DATA INFILE crushes the fastest DBI code ○ 10X speedup is not unusual
  8. 8. 2. Use DBI Efficiently● Use ORMs Wisely ○ Consider using straight DBI for the most performance sensitive sections ■ Removing a layer means fewer method calls and faster code ○ Write report queries by hand if they seem slow ■ Optimizer hints and choices about SQL variations are beyond the scope of ORMs but make a huge difference for this kind of query
  9. 9. 3. Choose the Fastest Hash Storage● memcached is not the fastest option for a local cache ○ BerkeleyDB (not DB_File!) and Cache::FastMmap are about twice as fast● CHI abstracts the storage layer ○ Useful if you think network strategy may change later
  10. 10. 3. Choose the Fastest Hash StorageCache Get time Set time Run timeCHI::Driver::Memory 0.03ms 0.05ms 0.35sBerkeleyDb 0.05ms 0.17ms 0.57sCache::FastMmap 0.06ms 0.09ms 0.62sCHI::Driver::File 0.10ms 0.26ms 1.11sCache::Memcached::Fast 0.12ms 0.15ms 1.23sMemcached::libmemcached 0.14ms 0.16ms 1.40sCHI::Driver::DBI Sqlite 0.11ms 1.94ms 2.05sCache::Memcached 0.29ms 0.21ms 2.88sCHI::Driver::DBI MySQL 0.45ms 0.33ms 4.41s
  11. 11. 4. Generate Code and Compile to aSubroutine ● This is how most templating tools work. ● Remove the cost of things that wont change for a while ○ Skip re-parsing templates ○ Skip large groups of conditionals ○ Choose architecture-specific codemy %subs;my $code = qq{print "Hello $thingn";};$subs{hello} = eval "sub { $code }";$subs{hello}->();
  12. 12. 5. Sling Text Efficiently ● Slurp files when $text = do { local $/; <$fh>; } ● Seems obvious, but I still see people doing this:my @lines = <$fh>;my $text = join(, @lines); ● Consider memory with huge files.
  13. 13. 5. Sling Text Efficiently ● Use a "sliding window" to search very large files. ○ Too big to slurp, but line-by-line is slow. ○ Chunks of 8K or 16K are much faster, but require book- keeping code. ○ ● Use the cheapest string tests you can get away with. ○ index() beats a regex when you just want to know if a string contains another string ● Use a fast CSV parser ○ Text::CSV_XS is much faster than the regexes you copied from that web page.
  14. 14. 6. Replace LWP With SomethingFaster● LWP is amazing, but modules built on C libraries tend to be faster. ○ LWP::Curl ○ HTTP::Lite ○ Maybe HTTP::Async for parallel LWP 32.8/s HTTP::Async 64.5/s HTTP::Lite 200/s LWP::Curl 1000/s
  15. 15. 7. Use a Fast Serializer ● Data::Dumper is great for debugging, but slow for serialization. ● JSON::XS is the new speed king, and is human-readable and cross-language. ● Storable handles more and is second-best in speed.
  16. 16. 7. Use a Fast Serializer YAML 84.7/s XML::Simple 800/s Data::Dumper 2143/s FreezeThaw 2635/s YAML::Syck 4307/s JSON::Syck 4654/s Storable 9774/s JSON::XS 41473/s
  17. 17. 8. Avoid Startup Costs● Use a daemon to run code persistently ○ Skip the costs of compiling ○ Cache data ○ Open connections ahead of time● mod_perl, FastCGI, Plack, etc. for web● PPerl for command-line ○ Or hit your web server with lwp-get
  18. 18. 9. Sometimes You Have to Get Crazy ● Use the @_ array directly to avoid copyingsub add_to_sql { my $sqlbase = shift; # hashref my ($name, $value) = @_; if ($value) { push(@{ $sqlbase->{names} }, $name); push(@{ $sqlbase->{values} }, $value); } return $sqlbase;}
  19. 19. 9. Sometimes You Have to Get Crazysub add_to_sql { # takes 3 params: hashref, name, and value return if not $_[2]; push(@{ $_[0]->{names} }, $_[1]); push(@{ $_[0]->{values} }, $_[2]);} ● 40% faster than original ● More than 40% harder to read
  20. 20. 10. Consider Compiling Your Own Perl● Compiling without threads can be good for a free 15% or so.● No code changes needed!● Has maintenance costs.
  21. 21. ResourcesTim Bunces Advanced DBI slides: see Tims NYTProf slides: perlperfProgramming Perl appendix on performance
  22. 22. Thank you!Slides will be available on the conference website
  23. 23. Avoid tie() ● Slower than method calls! ● PITA to debug too.
  24. 24. Use a Fast Sort● For sorting on derived keys, consider a GRT sort. ○ Faster than Schwartzian Transform ○ Use Sort::Maker to build it.