Proﬁling vs Benchmarking• Proﬁling applies to sub- systems (component design)• Benchmarking applies to entire systems (architecture and interaction)• Proﬁle is to unit test, as benchmark is to system test
Benchmarking Subjects• CPU • Programmer productivity• RAM • Time to initial• Disk access implementation• Database access • Time to add new• Network access major feature• Requests/second • Time to ﬁx bugs -- numeric measurement -- -- control for variables --
Why Bother With Benchmarking?• New site will be linked by Slashdot, Digg, etc.• “Can we handle the trafﬁc, Dodgers?”• “Sure thing, boss.”
Why Bother With Benchmarking?• “...(time to load-test.)”• How many req/sec expected?• Load-tested, optimized in a hurry, balanced
Know Your Limitations• What limits are imposed, and where?• Where to spend effort when optimizing?• Proﬁling your app will not help with non-app limits• Proﬁling does not tell you req/sec
The Stack• Web server• Framework (foundation)• Application
Combined Tooling• Each tool (ab, http_load, siege) has different options, params, and target speciﬁcations• http://github.com/pmjones/php-framework- benchmarks./bench/(ab|http_load|siege) target/all.ini
Benchmark Subject• Run against a test installation• As similar to production as possible• Hardware, OS, server, PHP installation• No database, no application code
Control for Known Variables• Network latency (use localhost)• Server processes (turn everything off)
HTML and PHP Graph 2814.62 ab 2170.31 3195.91http_load 2496.59 4262.12 siege 3148.12 0 1250 2500 3750 5000 html php (10 concurrent users, 60 seconds)
HTML and PHP Percentage Change html php pctab 2814.62 2170.31 77.11%httpload 3195.91 2496.59 78.12%siege 4262.12 3148.12 73.86%
Control for Unknown Variables• Run multiple times• No two runs will be identical; look for trends• File-based sessions will cause exponentially declining returns• session_id(‘constant_value’)
Other Problem Indicators• PHP faster than HTML?• Byte counts?• Socket availability?
Benchmarking as Resource Prediction• About resource planning, not speed (per se)• With supply of “R” requests/ second, you need “S” servers for demand of “T” trafﬁc• What stack components can you modify to increase “R” so you can handle more “T” with same or fewer “S”?
Single-Framework Methodology• Minimal, not full, application• Sane conﬁg• No action code• No view layout or helpers• Static view text• No caching• No database
Framework (Foundation) Benchmarking Web Server + PHP Framework Boot Front Page Action View• Dynamic dispatch cycle as the limiting factor• Improvements off critical path will not increase responsiveness• Page cache?
Single-Framework Analysis• Now you know how much you need to improve• Work on the critical path• Iterative benchmarking
Benchmarking as Engineering Aid• Every architecture decision has a cost• Compare benchmark stats between versions• Are added features are worth req/sec cost?• (Thanks, Laura Thomson.)
Multiple-Framework Comparison• Examine responsiveness in context of other systems• Compare architecture costs between systems• Use the same minimal app to remove variability in application code
Compare Like With Like• Frameworks to be compared should be reasonably similar in “style” or “class”• PHP5, design patterns, front & page controllers, controller & view separation, plugins or extensions, no globals• Cake, Lithium, Solar, Symfony, Yii, Zend• CI, Kohana
Multiple-Framework Methodology (1/2)• Minimal application• Reduced or optimized conﬁg• Production mode• No debugging or logging• No database connection• No layouts or view helpers
Multiple-Framework Methodology (2/2)• APC to reduce ﬁlesystem access• Restart web server between runs (APC)• 5 runs of 10 concurrent users for 60 seconds, averaged• Calculate req/sec as percent of PHP (thanks, Richard “Cyberlot” Thomas)
What Does This Mean?• Both more and less than you think it might• Speed is not the only consideration • Features, intangibles • Unlikely to change minds of devotees• Even the slowest is still pretty fast• Each AJAX call is one request
Failure Modes• Apache/Lighty, APC/XCache, mod_php/ fcgid• EC2 load ﬂuctuation between/during runs• Order of testing• Different concurrency or duration• Added features and app code will reduce dynamic response ... but curve is unknown.
Objections• “Highly proﬁled.”• “‘Hello world’ is unrealistic.”• “Not optimized for ‘hello world.’”• “Faster for ‘real world’ applications.”
Avoid Frameworks?• If framework code is only at 4-30% of max PHP, should we not use frameworks?• “Slow” relative to what?• Compare like with like, and proposed systems to existing systems.• You will use a framework, even if you write it yourself.• The good comes not from the code or the features, but process standardization.