Care and Feeding
of Large Web Applications




     Perrin Harkins
Arcos, a project at Plus Three
About 2.5 years of continuous development
2 - 5 developers on the team during that time
~79...
Arcos, cont'd
CMS with modern AJAX UI
E-commerce
Data warehouse with AJAX query builder GUI
E-mail campaign management
Asy...
Arcos, cont'd
Object-oriented
MVC-ish
  Controller classes
  Database classes
  Templates
Core modules:
  CGI::Application...
Deployment
Hard to generalize
.tar.gz files
Always release full builds, not individual files
  Consistent state for produc...
The CPAN problem
“Just use the CPAN shell”
  Fails too often
  Installs whatever the latest version is
CPAN installer requirements:
Install specific versions
Install from local media
Allow locally modified versions
Fully auto...
Our solution
Finds all modules in src/ directory of package
  (Including ones we hacked)
Uses Expect to answer questions
B...
Why bundle dependencies?
Avoid troubleshooting local problems
If possible, specify Perl, MySQL, OS, etc.
Sometimes reality...
Next up: binary distributions
Compile takes too long
  Still just tar?
  RPMs?
  PAR?
See build system in Krang:
  http://...
OH HAI!




I HELP WIF UR
INDENTATION!
Upgrades
Need full automation
Got the database part
  Schema version number in database
  Run all upgrade scripts with app...
Configuration
    Highly configurable systems must be highly
●


    configured
    Started with httpd.conf style
●


    ...
Observations on Configuration
    People ignore options they don't understand
●


        If the server starts, it must be...
I CAN HAS
MAINTENANCE
  BRANCH?
Version Control
Many tool choices now
  svn, svk, git...
  Fight it out and let me know who wins
Most projects need at lea...
Very simple version control
Main branch is development
  Must build and pass tests
When making a release from a branch, ta...
Very simple version control, cntd
Maintenance branch from that tag for bug fixes
  “2.x” branch
On maintenance release, me...
Too simple?
Not everyone is done at the same time
Hard to keep dev branch stable for major
 changes
  Feature branches
  P...
HALP!
TESTS R
FAYLIN!
Testing
    You all know the drill
●



    Local library for common testing tasks
●


        Log in mech object
    –

 ...
Testing tools
    Simple Test::More scripts are easy
●


        But scoping becomes an issue
    –

        Test::Class
 ...
Testing tools, ctd
                   ●



    Test::WWW::Mechanize
●


        Great for testing SSL, Apache config, mod_...
KTHXBYE!
Upcoming SlideShare
Loading in …5
×

Os Harkins

2,514 views
2,435 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,514
On SlideShare
0
From Embeds
0
Number of Embeds
32
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Os Harkins

  1. 1. Care and Feeding of Large Web Applications Perrin Harkins
  2. 2. Arcos, a project at Plus Three About 2.5 years of continuous development 2 - 5 developers on the team during that time ~79,000 lines of Perl code ~4900 lines of SQL (Computed with David Wheeler's SLOCCount program) Plenty of HTML, CSS, and JavaScript too ~14000 automated tests in 269 files 169 CPAN modules
  3. 3. Arcos, cont'd CMS with modern AJAX UI E-commerce Data warehouse with AJAX query builder GUI E-mail campaign management Asynchronous job queue system Comprehensive reporting
  4. 4. Arcos, cont'd Object-oriented MVC-ish Controller classes Database classes Templates Core modules: CGI::Application Class::DBI HTML::Template
  5. 5. Deployment Hard to generalize .tar.gz files Always release full builds, not individual files Consistent state for production system QA installs are accurate copies
  6. 6. The CPAN problem “Just use the CPAN shell” Fails too often Installs whatever the latest version is
  7. 7. CPAN installer requirements: Install specific versions Install from local media Allow locally modified versions Fully automated “Erase your hard drive? [n]” Install in a local directory Skip the tests Cluster install Some will never work
  8. 8. Our solution Finds all modules in src/ directory of package (Including ones we hacked) Uses Expect to answer questions Builds non-CPAN stuff too Apache mod_perl SWISH-E search engine
  9. 9. Why bundle dependencies? Avoid troubleshooting local problems If possible, specify Perl, MySQL, OS, etc. Sometimes reality intervenes Hardware support Office politics
  10. 10. Next up: binary distributions Compile takes too long Still just tar? RPMs? PAR? See build system in Krang: http://krangcms.com/
  11. 11. OH HAI! I HELP WIF UR INDENTATION!
  12. 12. Upgrades Need full automation Got the database part Schema version number in database Run all upgrade scripts with appropriate names e.g. 2.0 --> 3.0 means run upgrade/V2_1.pm and upgrade/V3_0.pm Hand-written SQL scripts Test mode that upgrades from old schema first LVM for rollback in QA
  13. 13. Configuration Highly configurable systems must be highly ● configured Started with httpd.conf style ● Config::ApacheFormat – Basic scoping and inheritance –
  14. 14. Observations on Configuration People ignore options they don't understand ● If the server starts, it must be ok! – Some comments in a config file == weak ● documentation Things you rarely change shouldn't be in the ● shipped config file
  15. 15. I CAN HAS MAINTENANCE BRANCH?
  16. 16. Version Control Many tool choices now svn, svk, git... Fight it out and let me know who wins Most projects need at least two branches at all times development stable
  17. 17. Very simple version control Main branch is development Must build and pass tests When making a release from a branch, tag it Make a “2.0” tag when you release it
  18. 18. Very simple version control, cntd Maintenance branch from that tag for bug fixes “2.x” branch On maintenance release, merge all changes on maintenance branch to main branch Tag “2.1” on 2.x branch Merge 2.0 --> 2.1 onto main branch
  19. 19. Too simple? Not everyone is done at the same time Hard to keep dev branch stable for major changes Feature branches Possible integration problems Beware of complex merges This is not the Linux kernel Could be a people problem An alternate point of view http://utsl.gen.nz/talks/git-svn/intro.html
  20. 20. HALP! TESTS R FAYLIN!
  21. 21. Testing You all know the drill ● Local library for common testing tasks ● Log in mech object – Run test SMTP server – Test data setup and teardown ● Keep stack, delete in reverse order – Probably easier to just wipe the db –
  22. 22. Testing tools Simple Test::More scripts are easy ● But scoping becomes an issue – Test::Class – Test::Builder – Improves test failure messages ●
  23. 23. Testing tools, ctd ● Test::WWW::Mechanize ● Great for testing SSL, Apache config, mod_rewrite – Works fine on JavaScript pages – But doesn't test the JavaScript ● Selenium does ● Smolder ● http://sourceforge.net/projects/smolder –
  24. 24. KTHXBYE!

×