LPW 2007 - Perl Plumbing

Perl Plumbing

Mike Astle

LPW – 1 Dec 2007

Perl Plumbing Toolbox

Pipe::Flush

Pipe::Leak

Pipe::Crack

Bundle::Pipe

Crack::Expose

HTML::Spanner

Crypt::Drip

Data::Bucket

Talking Pointz

Test::Harness

Devel::Cover

HTTPD::Bench::ApacheBench?

Devel::DProf/Apache::DProf

Plumbing == Infrastructure

Why spend the time?

More effective unit tests

Better benchmarking

Faster optimization

Ever so unsexy

Because you should

Automated Testing Test::Harness And His Super Friends

Testing

We've got both kinds of testing - unit >and< blackbox

Test::Harness, Test::Simple, Test::More, Test::WWW::Mechanize, ...

Mature family of modules with >lots< of features

A simple example...

Test Example – The Test

astle@nerdcore:~$ cat test.t

#!/usr/bin/perl -w

use Test::More tests => 3;

BEGIN { use_ok( ’Acme::Test’ ); }

my $obj = new Acme::Test();

ok $obj, 'new object';

ok $obj->doSomething(), 'doSomething';

Test Example – The Output

astle@nerdcore:~$ perl test.t

1..3

ok 1 - use Acme::Test;

ok 2 - new object

ok 3 - doSomething

Test Example - TAP

Test output is in the Test Anything Protocol (TAP) format

Easily digested by Test::Harness by way of the prove utility

Test Example - prove

astle@nerdcore:~$ prove test.t

test....ok

All tests successful.

Files=1, Tests=3, 0 wallclock secs

( 0.02 cusr + 0.00 csys = 0.02 CPU)

t/base/config/basics...................................................ok

t/base/config/dirs.....................................................ok

t/base/config/merging..................................................ok

t/base/config/warnings.................................................ok

t/base/crypt...........................................................ok

t/base/datadumper/basics...............................................ok

t/base/dates...........................................................ok

t/base/grid_search.....................................................ok

t/base/instructions....................................................ok

t/base/langutils.......................................................ok

t/base/logging/diag....................................................ok

t/base/logging.........................................................ok

t/base/obfuscate.......................................................ok

t/base/ranges..........................................................ok

t/base/report..........................................................ok

t/base/utils/basics....................................................ok

t/blackbox/api/es_realestate/create_url................................ok

t/blackbox/api/es_realestate/echo_es...................................ok

t/blackbox/api/es_realestate/keywords..................................ok

t/blackbox/api/es_realestate/metadata..................................ok

t/blackbox/api/es_realestate/search_listings...........................ok

t/blackbox/api/uk_realestate/create_url................................ok

t/blackbox/api/uk_realestate/echo_uk...................................ok

t/blackbox/api/uk_realestate/keywords..................................ok

t/blackbox/api/uk_realestate/metadata..................................ok

t/blackbox/api/uk_realestate/metadata_coord...........................ok

t/blackbox/api/uk_realestate/search_listings...........................ok

t/blackbox/frontend/es_realestate/302_or_html..........................ok

t/blackbox/frontend/es_realestate/apache...............................ok

t/blackbox/frontend/es_realestate/apache...............................ok

t/blackbox/frontend/es_realestate/autosuggest..........................ok

t/blackbox/frontend/es_realestate/cb...................................ok

t/blackbox/frontend/es_realestate/check_404_not_looping................ok

t/blackbox/frontend/es_realestate/cookies..............................ok

t/blackbox/frontend/es_realestate/demo.................................ok

t/blackbox/frontend/es_realestate/dimg.................................ok

t/blackbox/frontend/es_realestate/fbml.................................ok

t/blackbox/frontend/es_realestate/keep_query_for_redirect..............ok

t/blackbox/frontend/es_realestate/kml..................................ok

t/blackbox/frontend/es_realestate/map..................................ok

t/blackbox/frontend/es_realestate/metrics..............................ok

t/blackbox/frontend/es_realestate/redirect_to_www......................ok

t/blackbox/frontend/es_realestate/rss..................................ok

t/blackbox/frontend/es_realestate/share................................ok

t/blackbox/frontend/es_realestate/static_file..........................ok

t/blackbox/frontend/es_realestate/timer................................ok

t/blackbox/frontend/es_realestate/widget...............................ok

t/blackbox/frontend/es_realestate/www_cgi..............................ok

t/blackbox/frontend/uk_realestate/302_or_html..........................ok

t/blackbox/frontend/uk_realestate/acookie..............................ok

t/blackbox/frontend/uk_realestate/apache...............................ok

t/blackbox/frontend/uk_realestate/autosuggest..........................ok

t/blackbox/frontend/uk_realestate/cb...................................ok

t/blackbox/frontend/uk_realestate/check_404_not_looping................ok

t/blackbox/frontend/uk_realestate/cluster_image/end_to_end.............ok

t/blackbox/frontend/uk_realestate/cookies..............................ok

t/blackbox/frontend/uk_realestate/correct_let_to_rent..................ok

t/blackbox/frontend/uk_realestate/correct_location.....................ok

t/blackbox/frontend/uk_realestate/demo.................................ok

t/blackbox/frontend/uk_realestate/dimg.................................ok

t/blackbox/frontend/uk_realestate/botnet....ok

t/blackbox/frontend/uk_realestate/fbml.................................ok

t/blackbox/frontend/uk_realestate/keep_query_for_redirect..............ok

t/blackbox/frontend/uk_realestate/kml..................................ok

t/blackbox/frontend/uk_realestate/map..................................ok

t/blackbox/frontend/uk_realestate/metrics..............................ok

t/blackbox/frontend/uk_realestate/read_from_apache_log.................ok

t/blackbox/frontend/uk_realestate/redirect_to_www......................ok

t/blackbox/frontend/uk_realestate/rss..................................ok

t/blackbox/frontend/uk_realestate/share................................ok

t/blackbox/frontend/uk_realestate/static_file..........................ok

t/blackbox/frontend/uk_realestate/timer................................ok

t/blackbox/frontend/uk_realestate/widget...............................ok

t/blackbox/frontend/uk_realestate/www_cgi..............................ok

t/blackbox/search-api/es_realestate....................................ok

t/blackbox/search-api/uk_realestate....................................ok

t/cache/basics.........................................................ok

t/cache/list_of_all_caches.............................................ok

t/compiles_and_use_strict..............................................ok

t/database/base/arcgrid_datapoint......................................ok

t/database/base/geoid_datapoint........................................ok

t/database/base/geoid_summary..........................................ok

t/database/base........................................................ok

t/database/census......................................................ok

t/database/coverage....................................................ok

t/database/ctax........................................................ok

t/database/get_random_listings.........................................ok

t/database/listingdb...................................................ok

t/database/localdb.....................................................ok

t/database/metricsdb...................................................ok

t/database/searchindex/base............................................ok

t/database/searchindex/set_comment.....................................ok

t/database/users.......................................................ok

t/dependencies.........................................................ok

t/etl/attrhash.........................................................ok

t/etl/convert..........................................................ok

t/etl/dropbox..........................................................ok

t/etl/get_summaries_since..............................................ok

t/etl/images...........................................................ok

t/etl/import...........................................................ok

t/etl/instance/cleanup.................................................ok

t/etl/instance/convert.................................................ok

t/etl/instance/import..................................................ok

t/etl/instance/process.................................................ok

t/etl/instance/receive.................................................ok

t/etl/instance.........................................................ok

t/etl/object_cache.....................................................ok

t/listings/scorer/quality_score_v2.....................................ok

t/listings/scorer/random_scorer........................................ok

t/listings/scorer/simple_scorer........................................ok

t/listings/searchindex.................................................ok

t/listings/slice.......................................................ok

t/listings/stripe......................................................ok

t/listings/striper.....................................................ok

t/listings/transformer.................................................ok

t/localization/basics..................................................ok

t/etl/weather/control.............................ok

t/etl/process/realestate/es............................................ok

t/etl/process/realestate/uk............................................ok

t/etl/process..........................................................ok

t/etl/product/realestate/es............................................ok

t/etl/product/realestate/realestate....................................ok

t/etl/product/realestate/uk............................................ok

t/etl/receive..........................................................ok

t/etl/servers..........................................................ok

t/etl/summary..........................................................ok

t/etl/thumbs...........................................................ok

t/external_apps/lockrun................................................ok

t/geo/cleangeoid.......................................................ok

t/geo/clusterutils.....................................................ok

t/geo/coord_location...................................................ok

t/geo/creategeoid......................................................ok

t/geo/find_locations/base..............................................ok

t/geo/find_locations/coordinates.......................................ok

t/geo/find_locations/es/postcodes......................................ok

t/geo/find_locations/es/specials.......................................ok

t/geo/find_locations/es/spellcheck.....................................ok

t/geo/find_locations/geoids............................................ok

t/geo/find_locations/uk/postcodes......................................ok

t/geo/find_locations/uk/specials.......................................ok

t/geo/find_locations/uk/spellcheck.....................................ok

t/geo/find_locations/wordindex.........................................ok

t/geo/geocoder/common..................................................ok

t/geo/geocoder/coordinates.............................................ok

t/geo/geocoder/es/addresses............................................ok

t/geo/geocoder/es/cleansing............................................ok

t/geo/geocoder/es/hierarchy............................................ok

t/geo/geocoder/es/streets..............................................ok

t/geo/geocoder/uk/addresses............................................ok

t/geo/geocoder/uk/cleansing............................................ok

t/geo/geocoder/uk/postcode_attributes..................................ok

t/geo/geogrid..........................................................ok

t/geo/geoidindex.......................................................ok

t/geo/geometry.........................................................ok

t/geo/inputformat......................................................ok

t/geo/lookupfiles/es...................................................ok

t/geo/lookupfiles/uk...................................................ok

t/geo/postcodes/es.....................................................ok

t/geo/postcodes/uk.....................................................ok

t/geo/thirdparty/arcgrid/base..........................................ok

t/geo/thirdparty/arcgrid/create........................................ok

t/geo/thirdparty/arcgrid/dbfile........................................ok

t/geo/thirdparty/arcgrid/parse.........................................ok

t/geo/thirdparty/arcgrid/serialize.....................................ok

t/geo/thirdparty/arcgrid/testfile......................................ok

t/geo/tileoverlay......................................................ok

t/geo/utils............................................................ok

t/geo/utils_es.........................................................ok

t/geo/wordindex........................................................ok

t/io/writer/multi/file.................................................ok

t/io/writer/multi/memory...............................................ok

t/io/writer/single/file................................................ok

t/io/writer/single/memory..............................................ok

t/link_checker/test_plugins............................................ok

And so on...

Failed Test Stat Wstat Total Fail List of Failed

------------------------------------------------------------------------------

t/blackbox/api/es_realestate/echo_es.t 255 65280 1 1 1

t/blackbox/api/uk_realestate/echo_uk.t 255 65280 1 1 1

t/listings/scorer/quality_score_v2.t 2 512 57 2 28-29

t/localization/check_po_files.t 4 1024 506 4 3-4 425-426

t/pod-names.t 1 256 294 1 163

t/test-template.t 255 65280 1 2 1

51 tests and 5 subtests skipped.

Failed 6/348 test scripts. 10/9515 subtests failed.

Files=348, Tests=9515, 845 wallclock secs (624.34 cusr + 18.95 csys = 643.29 CPU)

Failed 6/348 test programs. 10/9515 subtests failed.

Testing/Staging Server/Cluster

This is >not< a development server – no local work

As similar as possible to development environment – could be on production hardware running on non-conflicting ports

Enough oomph to run test suite in a reasonable amount of time

Nightly Build

Get the HEAD

We use svn up via IPC::RUN3

You may prefer SVN::Agent or SVN::Client

Check for local changes – no tomfoolery

Clean build

Update dependencies

Rebuild configuration

Restart services (e.g. Apache, MySQL)

What about test data?

Nightly Test Run

Only test each revision once – nobody likes to hear the same thing twice

We use prove -r (look forward to prove -r -j 4)

Send output in a useful format – we do some simple filtering of the prove output

Fix problems right away – squeaky wheels eventually get ignored

Congratulate yourself when things are good

Code Coverage Devel::Cover Your Ass

Code Coverage Tools

How do you know if you have enough tests?

Coverage tools tell you how much of your code is actually executed

Devel::Cover is pretty bad ass

Still alpha after all these years?

Can also be used in mod_perl – we don't do this...yet

Run For Cover

Call cover on every .t file – output into a single database

Ignore list can be a bit trying – trial and error

Generate summary stats from a single database

Running for Cover

Running command: /usr/bin/perl -MDevel::Cover=-ignore,^t/,+ignore,^ext/,+ignore,^/usr/ /home/lokku/code/perl/t/etl/images.t

Running command: /usr/bin/perl -MDevel::Cover=-ignore,^t/,+ignore,^ext/,+ignore,^/usr/ /home/lokku/code/perl/t/database/users.t

Running command: /usr/bin/perl -MDevel::Cover=-ignore,^t/,+ignore,^ext/,+ignore,^/usr/ /home/lokku/code/perl/t/geo/coord_location.t

Ran for Cover

---------------------------- ------ ------ ------ ------ ------ ------ ------

File stmt bran cond sub pod time total

---------------------------- ------ ------ ------ ------ ------ ------ ------

lib/Lokku/Base/Config.pm 92.9 85.0 66.7 94.7 n/a 5.4 90.8

lib/Lokku/Base/Crypt.pm 96.6 85.3 47.8 100.0 n/a 4.7 87.3

lib/Lokku/Base/DataDumper.pm 80.0 68.8 66.7 83.3 40.0 1.4 75.0

lib/Lokku/Base/Dates.pm 80.6 70.1 63.2 91.3 n/a 10.9 75.2

lib/Lokku/Base/GridSearch.pm 100.0 83.3 66.7 100.0 n/a 0.7 95.6

...okku/Base/Instructions.pm 100.0 n/a n/a 100.0 n/a 0.0 100.0

lib/Lokku/Base/LangUtils.pm 100.0 n/a n/a 100.0 n/a 0.0 100.0

lib/Lokku/Base/Logging.pm 86.6 37.5 38.1 100.0 0.0 0.3 68.9

...se/Logging/DiagLogging.pm 100.0 n/a n/a 100.0 20.0 0.1 89.2

...Base/Logging/NoLogging.pm 66.7 n/a n/a 45.5 12.5 0.0 48.6

.../Logging/SimpleLogging.pm 39.3 n/a n/a 36.4 12.5 0.0 34.0

lib/Lokku/Base/Obfuscate.pm 98.7 92.3 50.0 100.0 n/a 3.6 96.5

lib/Lokku/Base/Ranges.pm 74.4 50.0 40.7 91.7 100.0 1.1 62.4

lib/Lokku/Base/Report.pm 82.7 69.4 69.6 94.4 100.0 2.7 80.1

lib/Lokku/Base/Utils.pm 68.2 50.0 40.0 85.2 100.0 58.4 63.2

lib/Lokku/Cache.pm 26.1 0.0 0.0 33.3 n/a 0.0 23.5

lib/Lokku/DB/Base.pm 8.5 0.0 0.0 21.2 24.3 0.1 7.5

lib/Lokku/DB/UsersDB.pm 11.6 0.0 0.0 31.9 0.0 0.1 7.4

...okku/Geo/CoordLocation.pm 21.1 3.3 4.3 50.0 n/a 0.0 16.5

lib/Lokku/Geo/CreateGeoId.pm 16.6 3.2 0.0 33.3 37.0 0.2 16.3

lib/Lokku/Geo/GeoCoder.pm 20.8 0.0 0.0 48.4 87.5 0.1 17.7

lib/Lokku/Geo/GeoIdIndex.pm 34.1 9.7 5.9 57.7 90.9 0.2 29.5

...

Benchmarking How much do ya HTTPD::Bench::ApacheBench?

Benchmarking

Running tests against your own code (or services) to measure performance

Emphasize comparability

Name and version benchmarks

Generate benchmarks from real query logs or from your staging server for new features

Record context (name, version, hardware, number of clients) in output

Benchmarking - Tools

We use HTTPD::Bench::ApacheBench

A few shortcomings:

No per-request headers (e.g. cookies)

Could be easier to pick out the slow pokes

Benchmarking - Output

---------- Variables ----------

Start Time: 1195660491

Stop Time: 1195662141

Benchmark: frontend-profile

Version: version1

---------- Statistics ----------

Total Correct Responses: 2931

Average: 0.2432

Standard Deviation: 0.3950

Percentiles:

p95: 0.913

p98: 1.451

p99: 2.075

---------- Errors ----------

Benchmarking - Analysis

Ignore the average

Look at percentiles

All SLAs are in nines

Good average can be bad

Bad average can be good

Profiling Not Just For Java Anymore

Profiling

Profilers are tools for analyzing where your code is spending all of its time

My personal fetish

Sometimes complicated to set up – but definitely worth it

Profiling - Tools

We use Devel::DProf

Best of Breed?

There are others in CPAN

Hook into mod_perl with Apache::DProf

Profiling - Output

DProf File: /home/lokku/common/conf/apache/logs/dprof/10284/tmon.out

Total Elapsed Time = 792.3461 Seconds

User+System Time = 13.22611 Seconds

Exclusive Times

%Time ExclSec CumulS #Calls sec/call Csec/c Name

24.0 3.178 3.178 804 0.0040 0.0040 Lokku::URL::SERP::process

17.9 2.371 13.116 172 0.0138 0.0763 Lokku::Website::Base::handler

9.78 1.294 9.353 2506 0.0005 0.0037 HTML::Mason::Request::comp

8.95 1.184 1.292 5806 0.0002 0.0002 Lokku::Template::loc

7.75 1.025 1.129 3783 0.0003 0.0003 Lokku::Website::SERP::__ANON__

3.15 0.417 9.645 4090 0.0001 0.0024 HTML::Mason::Commands::__ANON__

2.80 0.370 5.025 1468 0.0003 0.0034 HTML::Mason::Request::scomp

2.56 0.339 0.339 220 0.0015 0.0015 Lokku::URL::SERP::as_path

1.71 0.226 0.226 1743 0.0001 0.0001 Lokku::Template::url_for

1.70 0.225 0.225 29819 0.0000 0.0000 BerkeleyDB::_tiedHash::NEXTKEY

1.29 0.170 0.170 4 0.0425 0.0425 Lokku::NLP::UK::EN::RealEstate::An

alyse::new

1.14 0.151 0.143 260 0.0006 0.0005 Text::Balanced::_match_codeblock

1.00 0.132 0.149 1652 0.0001 0.0001 Text::Balanced::_match_variable

0.73 0.096 0.096 1691 0.0001 0.0001 CGI::Util::rearrange

0.69 0.091 0.091 7748 0.0000 0.0000 URI::_query::query

0.68 0.090 0.090 2 0.0450 0.0450 Lokku::NLP::new

0.65 0.086 0.086 1493 0.0001 0.0001 Lokku::URL::SERP::set

0.60 0.079 0.079 8434 0.0000 0.0000 HTML::Mason::Interp::apply_escapes

0.57 0.075 0.075 5806 0.0000 0.0000 Locale::Maketext::Lexicon::EXISTS

0.57 0.075 0.081 497 0.0002 0.0002 Lokku::NLP::sort_and_convert

0.49 0.065 0.065 1812 0.0000 0.0000 Text::Balanced::_match_quotelike

Profiling - Analysis

One output file per apache process

Code out there to combine, looks dicey

Just run one child

Look for targets of opportunity

Inclusive and Exclusive display modes

Some reason to believe that output is not always accurate

Is it all worth it?

Hells yes

Small teams need good tools – take advantage of the fine work done by others

Low one-time cost, low maintenance, big long-term gain

Run tests daily – review other data monthly

Many ways to do things, but the lazy way is often more efficient – avoid rolling your own

Thanks! [email_address]

LPW 2007 - Perl Plumbing

0 comments

Notes on slide 1

2 Groups