3. Read These
• “Free to Choose”, Friedman
• “Pleasure of Finding Things Out”, Feynman
• “Fooled By Randomness”, Taleb
4. About Me
• Dev, Sr Dev, Team Lead,
Architect, VP Eng
• PHP since 1999
• Savant template system
• PEAR Group member
• http://paul-m-jones.com
• @pmjones
5. Full Disclosure
• Solar framework
• Zend framework
• Zend_Db*,
Zend_View
• “Rigor and integrity”
• Google Code, now
Github
7. Profiling vs
Benchmarking
• Profiling applies to sub-
systems (component design)
• Benchmarking applies to
entire systems (architecture
and interaction)
• Profile is to unit test, as
benchmark is to system test
8. Benchmarking Subjects
• CPU • Programmer
productivity
• RAM
• Time to initial
• Disk access implementation
• Database access • Time to add new
• Network access major feature
• Requests/second • Time to fix bugs
-- numeric measurement --
-- control for variables --
9. Why Bother With
Benchmarking?
• New site will be linked
by Slashdot, Digg, etc.
• “Can we handle the
traffic, Dodgers?”
• “Sure thing, boss.”
10. Why Bother With
Benchmarking?
• “...(time to load-test.)”
• How many req/sec
expected?
• Load-tested, optimized
in a hurry, balanced
11. Know Your Limitations
• What limits are imposed,
and where?
• Where to spend effort
when optimizing?
• Profiling your app will not
help with non-app limits
• Profiling does not tell you
req/sec
12. The Stack
• Web server
• Framework (foundation)
• Application
14. Methodology
• Refining and reporting since 2007
• Benchmark the basic system
• Add a component, benchmark again
• Track responsiveness at each point
15. Tools
• ab, http_load,
siege, ...
• Emulate concurrent
users hitting a URI
• Reports req/sec,
latency, min/max
times, etc
ab -c 10 -t 60 http://example.com/
16. ab
Document Path: /baseline-html/index.html
Document Length: 13 bytes
Concurrency Level: 10
Time taken for tests: 0.19088 seconds
Complete requests: 50000
Failed requests: 0
Total transferred: 15950000 bytes
HTML transferred: 650000 bytes
Requests per second: 2619.45
Transfer rate: 835.60 kb/s received
17. http_load
4278 fetches, 325 max parallel, 25668 bytes
6 mean bytes/connection
855 fetches/sec, 5130 bytes/sec
msecs/connect: 20.0881 mean, 3006.54 max, 0.099 min
msecs/first-response: 51.3568 mean, 342.488 max, 1.423 min
HTTP response codes:
code 200 -- 4278
19. Combined Tooling
• Each tool (ab, http_load, siege) has different
options, params, and target specifications
• http://github.com/pmjones/php-framework-
benchmarks
./bench/(ab|http_load|siege) target/all.ini
20. Benchmark Subject
• Run against a test installation
• As similar to production as possible
• Hardware, OS, server, PHP installation
• No database, no application code
21. Control for
Known Variables
• Network latency (use localhost)
• Server processes (turn everything off)
23. HTML and PHP Graph
2814.62
ab
2170.31
3195.91
http_load
2496.59
4262.12
siege
3148.12
0 1250 2500 3750 5000
html php
(10 concurrent users, 60 seconds)
24. HTML and PHP
Percentage Change
html php pct
ab 2814.62 2170.31 77.11%
httpload 3195.91 2496.59 78.12%
siege 4262.12 3148.12 73.86%
25. Control for
Unknown Variables
• Run multiple times
• No two runs will be
identical; look for trends
• File-based sessions will
cause exponentially
declining returns
• session_id(‘constant_value’)
26. Other Problem
Indicators
• PHP faster than
HTML?
• Byte counts?
• Socket availability?
27. Benchmarking as
Resource Prediction
• About resource planning, not
speed (per se)
• With supply of “R” requests/
second, you need “S” servers
for demand of “T” traffic
• What stack components can
you modify to increase “R”
so you can handle more “T”
with same or fewer “S”?
29. Single-Framework
Methodology
• Minimal, not full, application
• Sane config
• No action code
• No view layout or helpers
• Static view text
• No caching
• No database
30. Framework (Foundation)
Benchmarking
Web Server + PHP
Framework
Boot Front Page Action View
• Dynamic dispatch cycle as the limiting
factor
• Improvements off critical path will not
increase responsiveness
• Page cache?
31. Single-Framework
Analysis
• Now you know how
much you need to
improve
• Work on the critical path
• Iterative benchmarking
32. Benchmarking as
Engineering Aid
• Every architecture
decision has a cost
• Compare benchmark
stats between versions
• Are added features
are worth req/sec
cost?
• (Thanks, Laura
Thomson.)
34. Multiple-Framework
Comparison
• Examine responsiveness
in context of other systems
• Compare architecture
costs between systems
• Use the same minimal app
to remove variability in
application code
35. Compare Like With Like
• Frameworks to be compared
should be reasonably similar
in “style” or “class”
• PHP5, design patterns, front
& page controllers, controller
& view separation, plugins or
extensions, no globals
• Cake, Lithium, Solar,
Symfony, Yii, Zend
• CI, Kohana
36. Multiple-Framework
Methodology (1/2)
• Minimal application
• Reduced or optimized config
• Production mode
• No debugging or logging
• No database connection
• No layouts or view helpers
37. Multiple-Framework
Methodology (2/2)
• APC to reduce filesystem access
• Restart web server between runs (APC)
• 5 runs of 10 concurrent users for 60
seconds, averaged
• Calculate req/sec as percent of PHP
(thanks, Richard “Cyberlot” Thomas)
41. What Does This Mean?
• Both more and less than you think it
might
• Speed is not the only consideration
• Features, intangibles
• Unlikely to change minds of devotees
• Even the slowest is still pretty fast
• Each AJAX call is one request
42. Failure Modes
• Apache/Lighty, APC/XCache, mod_php/
fcgid
• EC2 load fluctuation between/during runs
• Order of testing
• Different concurrency or duration
• Added features and app code will reduce
dynamic response ... but curve is
unknown.
45. Objections
• “Highly profiled.”
• “‘Hello world’ is unrealistic.”
• “Not optimized for ‘hello world.’”
• “Faster for ‘real world’ applications.”
46. Avoid Frameworks?
• If framework code is only at 4-30% of max
PHP, should we not use frameworks?
• “Slow” relative to what?
• Compare like with like, and proposed systems
to existing systems.
• You will use a framework, even if you write it
yourself.
• The good comes not from the code or the
features, but process standardization.
