This document summarizes the results of benchmarking tests run on various PHP frameworks including CakePHP, CodeIgniter, Kohana, Lithium, Solar, Symfony, Yii, and Zend. The tests measured each framework's performance in requests per second using tools like ApacheBench. While some frameworks were faster than others, the document notes that speed is not the only consideration and that even the slowest frameworks are still relatively fast. It concludes that architectural decisions can impact performance and benchmarks are useful for engineering analysis and resource planning.

1. Framework and
Application Benchmarking
Paul M. Jones
PhpBenelux 2011
@pmjones
http://joind.in/2498

2. Overview
• Benchmarking methodology, results,
and interpretation
• Server/application
• Single frameworks
• Framework comparisons
• Resource usage prediction
• Engineering analysis

3. Read These
• “Free to Choose”, Friedman
• “Pleasure of Finding Things Out”, Feynman
• “Fooled By Randomness”, Taleb

4. About Me
• Dev, Sr Dev, Team Lead,
Architect, VP Eng
• PHP since 1999
• Savant template system
• PEAR Group member
• http://paul-m-jones.com
• @pmjones

5. Full Disclosure
• Solar framework
• Zend framework
• Zend_Db*,
Zend_View
• “Rigor and integrity”
• Google Code, now
Github

6. Part 1:
In General

7. Profiling vs
Benchmarking
• Profiling applies to sub-
systems (component design)
• Benchmarking applies to
entire systems (architecture
and interaction)
• Profile is to unit test, as
benchmark is to system test

8. Benchmarking Subjects
• CPU • Programmer
productivity
• RAM
• Time to initial
• Disk access implementation
• Database access • Time to add new
• Network access major feature
• Requests/second • Time to fix bugs
-- numeric measurement --
-- control for variables --

9. Why Bother With
Benchmarking?
• New site will be linked
by Slashdot, Digg, etc.
• “Can we handle the
traffic, Dodgers?”
• “Sure thing, boss.”

10. Why Bother With
Benchmarking?
• “...(time to load-test.)”
• How many req/sec
expected?
• Load-tested, optimized
in a hurry, balanced

11. Know Your Limitations
• What limits are imposed,
and where?
• Where to spend effort
when optimizing?
• Profiling your app will not
help with non-app limits
• Profiling does not tell you
req/sec

12. The Stack
• Web server
• Framework (foundation)
• Application

13. Part 2:
Getting Started

14. Methodology
• Refining and reporting since 2007
• Benchmark the basic system
• Add a component, benchmark again
• Track responsiveness at each point

15. Tools
• ab, http_load,
siege, ...
• Emulate concurrent
users hitting a URI
• Reports req/sec,
latency, min/max
times, etc
ab -c 10 -t 60 http://example.com/

16. ab
Document Path: /baseline-html/index.html
Document Length: 13 bytes
Concurrency Level: 10
Time taken for tests: 0.19088 seconds
Complete requests: 50000
Failed requests: 0
Total transferred: 15950000 bytes
HTML transferred: 650000 bytes
Requests per second: 2619.45
Transfer rate: 835.60 kb/s received

17. http_load
4278 fetches, 325 max parallel, 25668 bytes
6 mean bytes/connection
855 fetches/sec, 5130 bytes/sec
msecs/connect: 20.0881 mean, 3006.54 max, 0.099 min
msecs/first-response: 51.3568 mean, 342.488 max, 1.423 min
HTTP response codes:
code 200 -- 4278

18. siege
2009-02-26 16:35:22, 256869, 60.28, 3, 0.00, 4261.26, ...

19. Combined Tooling
• Each tool (ab, http_load, siege) has different
options, params, and target specifications
• http://github.com/pmjones/php-framework-
benchmarks
./bench/(ab|http_load|siege) target/all.ini

20. Benchmark Subject
• Run against a test installation
• As similar to production as possible
• Hardware, OS, server, PHP installation
• No database, no application code

21. Control for
Known Variables
• Network latency (use localhost)
• Server processes (turn everything off)

22. Outer Limits Of
Responsiveness
• Amazon EC2 “Large”
instance, 64-Bit Ubuntu
10.04 (Alestic)
• Apache 2.2 (stock install)
• PHP 5.3.3, APC, Suhosin
(stock install)
• “index.html” and
“index.php”

23. HTML and PHP Graph
2814.62
ab
2170.31
3195.91
http_load
2496.59
4262.12
siege
3148.12
0 1250 2500 3750 5000
html php
(10 concurrent users, 60 seconds)

24. HTML and PHP
Percentage Change
html php pct
ab 2814.62 2170.31 77.11%
httpload 3195.91 2496.59 78.12%
siege 4262.12 3148.12 73.86%

25. Control for
Unknown Variables
• Run multiple times
• No two runs will be
identical; look for trends
• File-based sessions will
cause exponentially
declining returns
• session_id(‘constant_value’)

26. Other Problem
Indicators
• PHP faster than
HTML?
• Byte counts?
• Socket availability?

27. Benchmarking as
Resource Prediction
• About resource planning, not
speed (per se)
• With supply of “R” requests/
second, you need “S” servers
for demand of “T” traffic
• What stack components can
you modify to increase “R”
so you can handle more “T”
with same or fewer “S”?

28. Part 3:
Single Frameworks

29. Single-Framework
Methodology
• Minimal, not full, application
• Sane config
• No action code
• No view layout or helpers
• Static view text
• No caching
• No database

30. Framework (Foundation)
Benchmarking
Web Server + PHP
Framework
Boot Front Page Action View
• Dynamic dispatch cycle as the limiting
factor
• Improvements off critical path will not
increase responsiveness
• Page cache?

31. Single-Framework
Analysis
• Now you know how
much you need to
improve
• Work on the critical path
• Iterative benchmarking

32. Benchmarking as
Engineering Aid
• Every architecture
decision has a cost
• Compare benchmark
stats between versions
• Are added features
are worth req/sec
cost?
• (Thanks, Laura
Thomson.)

33. Part 3:
Multiple
Frameworks

34. Multiple-Framework
Comparison
• Examine responsiveness
in context of other systems
• Compare architecture
costs between systems
• Use the same minimal app
to remove variability in
application code

35. Compare Like With Like
• Frameworks to be compared
should be reasonably similar
in “style” or “class”
• PHP5, design patterns, front
& page controllers, controller
& view separation, plugins or
extensions, no globals
• Cake, Lithium, Solar,
Symfony, Yii, Zend
• CI, Kohana

36. Multiple-Framework
Methodology (1/2)
• Minimal application
• Reduced or optimized config
• Production mode
• No debugging or logging
• No database connection
• No layouts or view helpers

37. Multiple-Framework
Methodology (2/2)
• APC to reduce filesystem access
• Restart web server between runs (APC)
• 5 runs of 10 concurrent users for 60
seconds, averaged
• Calculate req/sec as percent of PHP
(thanks, Richard “Cyberlot” Thomas)

38. Framework Numbers
Target relative average 1 2 3 4 5
baseline/html 1.2823 3220.56 3227.18 3226.03 3207.75 3222.63 3219.20
baseline/php 1.0000 2511.54 2521.14 2509.00 2506.45 2508.48 2512.63
cake/1.3.7 0.0412 103.46 103.37 103.73 103.93 102.55 103.70
ci/1.7.3 0.2946 739.95 722.08 725.95 722.96 768.37 760.40
kohana/3.0.9 0.2912 731.37 696.83 757.46 743.72 720.10 738.76
lithium/0.9.9 0.0827 207.62 252.87 249.94 35.19 249.53 250.58
solar/1.1.1 0.1609 404.17 405.08 404.33 403.12 399.63 408.68
symfony/1.4.8 0.0784 196.91 197.00 197.77 199.00 194.93 195.87
symfony/2.0.0alpha1 0.1675 420.73 422.02 424.85 422.48 411.20 423.08
symfony/2.0.0pr4 0.0897 225.22 224.59 221.18 225.88 227.73 226.71
yii/1.1.5 0.1891 474.82 500.21 439.74 456.77 494.82 482.57
zend/1.10.2-minapp 0.1136 285.28 279.83 291.61 280.61 287.00 287.37
zend/1.10.2-project 0.0832 208.84 215.74 206.20 209.26 207.75 205.25

39. Framework Graph (100%)
baseline/html
baseline/php
cake/1.3.7
ci/1.7.3
kohana/3.0.9
lithium/0.9.9
solar/1.1.1
symfony/1.4.8
symfony/2.0.0alpha1
symfony/2.0.0pr4
yii/1.1.5
zend/1.10.2-minapp
zend/1.10.2-project
0 0.375 0.750 1.125 1.500

40. Framework Graph (30%)
cake/1.3.7 0.04
ci/1.7.3 0.29
kohana/3.0.9 0.29
lithium/0.9.9 0.08
solar/1.1.1 0.16
symfony/1.4.8 0.08
symfony/2.0.0alpha1 0.17
symfony/2.0.0pr4 0.09
yii/1.1.5 0.19
zend/1.10.2-minapp 0.11
zend/1.10.2-project 0.08
0 0.10 0.20 0.30

41. What Does This Mean?
• Both more and less than you think it
might
• Speed is not the only consideration
• Features, intangibles
• Unlikely to change minds of devotees
• Even the slowest is still pretty fast
• Each AJAX call is one request

42. Failure Modes
• Apache/Lighty, APC/XCache, mod_php/
fcgid
• EC2 load fluctuation between/during runs
• Order of testing
• Different concurrency or duration
• Added features and app code will reduce
dynamic response ... but curve is
unknown.

43. Links
• http://github.com/pmjones/php-framework-benchmarks
• http://code.google.com/p/web-framework-benchmarks
• “New Year’s Benchmarks”:
http://paul-m-jones/archives/238
• “Do All Frameworks Suck?”:
http://www.laurathomson.com/2007/07/do-all-
framewor.html
• Google: "cargo cult science"

44. • Questions?
• Comments?
• Criticism?
• http://joind.in/2498

45. Objections
• “Highly profiled.”
• “‘Hello world’ is unrealistic.”
• “Not optimized for ‘hello world.’”
• “Faster for ‘real world’ applications.”

46. Avoid Frameworks?
• If framework code is only at 4-30% of max
PHP, should we not use frameworks?
• “Slow” relative to what?
• Compare like with like, and proposed systems
to existing systems.
• You will use a framework, even if you write it
yourself.
• The good comes not from the code or the
features, but process standardization.

47. Thanks!