Your SlideShare is downloading. ×
  • Like
Wringing Performance out of Perl
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Wringing Performance out of Perl

  • 760 views
Published

I gave this lightning talk at Yapc 2011. My company uses Perl in a variety of products, some of which have serious performance implications. Here I give a quick overview of some of the tricks we use …

I gave this lightning talk at Yapc 2011. My company uses Perl in a variety of products, some of which have serious performance implications. Here I give a quick overview of some of the tricks we use to squeeze extra performance out of Perl.

Published in Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
760
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1.
  • 2. Wringing Performance out of Perl
  • 3. Grant Street Group
    Began as a financial advisor group
  • 4. Grant Street Group
    Discovered the Internet in 1997
  • 5. Grant Street Group
    Online Auctions of Property Tax Liens
    Web-Based billing system for tax collectors
    Conversion of legacy tax-collector databases
    Online license / vehicle tag renewals
    Online payment processing
    Auctions of all types of bonds
    And lots, lots more!
  • 6. Tax Lien Auctions
  • 7. Tax Lien Auctions
    Absolute feeding frenzy
    Our bidders threatened to exhaust TIN numbers
    20 million bidders in 2011
    More than 30 billion bids altogether
    Average was a 500,000-way tie
    About 2,000 auctions closing simultaneously
  • 8. Tax Lien Auctions
    How do we award auctions performantly?
    Random tie-breaking with Crypt::Random
    Random row-ID plus MySQL = S L O W
    Turns out we can do it much faster in Perl
  • 9. Tax Lien Auctions
    Net result: auction closing takes 20 seconds
    Breaking 2,000 ties, each 500,000-way
    Stress-testing indicates can scale by 4x
    The IRS definitely cannot scale by 4x
  • 10. Property Tax Online Payments
  • 11. Property Tax Online Payments
    Florida residents can pay the property tax
    Hosted, customized sites per county
    Largest counties have ~1,000,000 parcels
    Users are typical Florida residents
  • 12. Property Tax Online Payments
  • 13. Property Tax Online Payments
  • 14. Property Tax Online Payments
  • 15. Property Tax Online Payments
  • 16.
  • 17. Property Tax Online Payments
    Backend is MySQL and Sphinx
    Lightning-fast searches with Perl
    Mapping IDs to table, column, PK
    Parsing SHOW STATUS LIKE ‘sphinx%’
    Lots of useful metadata!
  • 18. Property Tax Online Payments
    Net results:
    Sub-second turnaround times
    9 minute average time on site by payers
    4 minute average time on site overall
  • 19. Customer Data Conversion
  • 20. Customer Data Conversion
    Largest county in FL is a customer
    Population ~2.4M people
    Tax roll of ~900K parcels
    History of ~5.6M bills across 6 years
    Full database is large (by our standards)
    Data files are ~30-50GB
    Full conversion is ~160 hours, using Perl
    Might be ~8 hours using pure SQL
  • 21. Customer Data Conversion
    Problem is we can’t use pure SQL
    Ridiculous amounts of business logic
    Utterly different data models
    We’re a Perl shop; Perl is our hammer
  • 22. Customer Data Conversion
    Hugely parallel data conversion
    Subdivide conversion into smaller steps
    Build hash of dependencies between steps
    Construct DAG of work units in MongoDB
    Distribute the actual work
    Run lots of Perl worker processes
    Workers grab ready work units
    Perform the work unit sequentially
  • 23. Customer Data Conversion
    The end result
    Total conversion time ~3 hours with 80 workers
    Nightly reloads now very practical
    Able to resume incomplete loads
  • 24. We’re Hiring Telecommuters