The document discusses how Grant Street Group uses Perl to power their online tax lien auctions and property tax payment systems. Key points: - Tax lien auctions see 20 million bidders in 2011 placing over 30 billion bids, with auctions closing in under 20 seconds using random tie-breaking in Perl. - Their property tax payment system handles up to 1 million parcels for large Florida counties, with sub-second search times and average user session times of only 4-9 minutes powered by Perl and MySQL. - Converting a large county's tax roll database of 900,000 parcels and 5.6 million bills over 6 years to their system takes only 3 hours using a highly parallel Perl-

2. Wringing Performance out of Perl

3. Grant Street Group Began as a financial advisor group

4. Grant Street Group Discovered the Internet in 1997

5. Grant Street Group Online Auctions of Property Tax Liens Web-Based billing system for tax collectors Conversion of legacy tax-collector databases Online license / vehicle tag renewals Online payment processing Auctions of all types of bonds And lots, lots more!

6. Tax Lien Auctions

7. Tax Lien Auctions Absolute feeding frenzy Our bidders threatened to exhaust TIN numbers 20 million bidders in 2011 More than 30 billion bids altogether Average was a 500,000-way tie About 2,000 auctions closing simultaneously

8. Tax Lien Auctions How do we award auctions performantly? Random tie-breaking with Crypt::Random Random row-ID plus MySQL = S L O W Turns out we can do it much faster in Perl

9. Tax Lien Auctions Net result: auction closing takes 20 seconds Breaking 2,000 ties, each 500,000-way Stress-testing indicates can scale by 4x The IRS definitely cannot scale by 4x

10. Property Tax Online Payments

11. Property Tax Online Payments Florida residents can pay the property tax Hosted, customized sites per county Largest counties have ~1,000,000 parcels Users are typical Florida residents

12. Property Tax Online Payments

13. Property Tax Online Payments

14. Property Tax Online Payments

15. Property Tax Online Payments

17. Property Tax Online Payments Backend is MySQL and Sphinx Lightning-fast searches with Perl Mapping IDs to table, column, PK Parsing SHOW STATUS LIKE ‘sphinx%’ Lots of useful metadata!

18. Property Tax Online Payments Net results: Sub-second turnaround times 9 minute average time on site by payers 4 minute average time on site overall

19. Customer Data Conversion

20. Customer Data Conversion Largest county in FL is a customer Population ~2.4M people Tax roll of ~900K parcels History of ~5.6M bills across 6 years Full database is large (by our standards) Data files are ~30-50GB Full conversion is ~160 hours, using Perl Might be ~8 hours using pure SQL

21. Customer Data Conversion Problem is we can’t use pure SQL Ridiculous amounts of business logic Utterly different data models We’re a Perl shop; Perl is our hammer

22. Customer Data Conversion Hugely parallel data conversion Subdivide conversion into smaller steps Build hash of dependencies between steps Construct DAG of work units in MongoDB Distribute the actual work Run lots of Perl worker processes Workers grab ready work units Perform the work unit sequentially

23. Customer Data Conversion The end result Total conversion time ~3 hours with 80 workers Nightly reloads now very practical Able to resume incomplete loads

24. We’re Hiring Telecommuters