Framework andApplication Benchmarking         Paul M. Jones        PhpBenelux 2011            @pmjones       http://joind....
Overview• Benchmarking methodology, results,  and interpretation • Server/application • Single frameworks • Framework comp...
Read These• “Free to Choose”, Friedman• “Pleasure of Finding Things Out”, Feynman• “Fooled By Randomness”, Taleb
About Me• Dev, Sr Dev, Team Lead,  Architect, VP Eng• PHP since 1999• Savant template system• PEAR Group member• http://pa...
Full Disclosure• Solar framework• Zend framework  • Zend_Db*,    Zend_View• “Rigor and integrity”• Google Code, now  Github
Part 1:In General
Profiling vs            Benchmarking• Profiling applies to sub-  systems (component design)• Benchmarking applies to  entire...
Benchmarking Subjects• CPU                      • Programmer                             productivity• RAM                ...
Why Bother With          Benchmarking?• New site will be linked  by Slashdot, Digg, etc.• “Can we handle the  traffic, Dodg...
Why Bother With            Benchmarking?• “...(time to load-test.)”• How many req/sec  expected?• Load-tested, optimized  ...
Know Your Limitations• What limits are imposed,  and where?• Where to spend effort  when optimizing?• Profiling your app wi...
The Stack• Web server• Framework (foundation)• Application
Part 2:Getting Started
Methodology• Refining and reporting since 2007• Benchmark the basic system• Add a component, benchmark again• Track respons...
Tools• ab, http_load,  siege, ...• Emulate concurrent  users hitting a URI• Reports req/sec,  latency, min/max  times, etc...
abDocument Path: /baseline-html/index.htmlDocument Length: 13 bytesConcurrency Level: 10Time taken for tests: 0.19088 seco...
http_load4278 fetches, 325 max parallel, 25668 bytes6 mean bytes/connection855 fetches/sec, 5130 bytes/secmsecs/connect: 2...
siege2009-02-26 16:35:22, 256869, 60.28, 3, 0.00, 4261.26, ...
Combined Tooling• Each tool (ab, http_load, siege) has different  options, params, and target specifications• http://github...
Benchmark Subject• Run against a test installation• As similar to production as possible• Hardware, OS, server, PHP instal...
Control for     Known Variables• Network latency (use localhost)• Server processes (turn everything off)
Outer Limits Of          Responsiveness• Amazon EC2 “Large”  instance, 64-Bit Ubuntu  10.04 (Alestic)• Apache 2.2 (stock i...
HTML and PHP Graph                           2814.62      ab                       2170.31                              31...
HTML and PHP     Percentage Change           html      php       pctab         2814.62   2170.31   77.11%httpload   3195.9...
Control for        Unknown Variables• Run multiple times• No two runs will be  identical; look for trends• File-based sess...
Other Problem          Indicators• PHP faster than  HTML?• Byte counts?• Socket availability?
Benchmarking as  Resource Prediction• About resource planning, not  speed (per se)• With supply of “R” requests/  second, ...
Part 3:Single Frameworks
Single-Framework      Methodology• Minimal, not full, application• Sane config• No action code• No view layout or helpers• ...
Framework (Foundation)    Benchmarking             Web Server + PHP               Framework      Boot   Front   Page   Act...
Single-Framework             Analysis• Now you know how  much you need to  improve• Work on the critical path• Iterative b...
Benchmarking as       Engineering Aid• Every architecture  decision has a cost• Compare benchmark  stats between versions•...
Part 3: MultipleFrameworks
Multiple-Framework          Comparison• Examine responsiveness  in context of other systems• Compare architecture  costs b...
Compare Like With Like• Frameworks to be compared  should be reasonably similar  in “style” or “class”• PHP5, design patte...
Multiple-Framework   Methodology (1/2)• Minimal application• Reduced or optimized config• Production mode• No debugging or ...
Multiple-Framework   Methodology (2/2)• APC to reduce filesystem access• Restart web server between runs (APC)• 5 runs of 1...
Framework NumbersTarget                relative   average   1         2         3         4         5baseline/html        ...
Framework Graph (100%)        baseline/html        baseline/php          cake/1.3.7              ci/1.7.3        kohana/3....
Framework Graph (30%)          cake/1.3.7        0.04             ci/1.7.3                                                ...
What Does This Mean?• Both more and less than you think it  might• Speed is not the only consideration  • Features, intang...
Failure Modes• Apache/Lighty, APC/XCache, mod_php/  fcgid• EC2 load fluctuation between/during runs• Order of testing• Diff...
Links• http://github.com/pmjones/php-framework-benchmarks• http://code.google.com/p/web-framework-benchmarks• “New Year’s ...
• Questions?• Comments?• Criticism?• http://joind.in/2498
Objections• “Highly profiled.”• “‘Hello world’ is unrealistic.”• “Not optimized for ‘hello world.’”• “Faster for ‘real worl...
Avoid Frameworks?• If framework code is only at 4-30% of max  PHP, should we not use frameworks?• “Slow” relative to what?...
Thanks!
Upcoming SlideShare
Loading in...5
×

Framework and Application Benchmarking

9,089

Published on

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,089
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
132
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Framework and Application Benchmarking

  1. 1. Framework andApplication Benchmarking Paul M. Jones PhpBenelux 2011 @pmjones http://joind.in/2498
  2. 2. Overview• Benchmarking methodology, results, and interpretation • Server/application • Single frameworks • Framework comparisons• Resource usage prediction• Engineering analysis
  3. 3. Read These• “Free to Choose”, Friedman• “Pleasure of Finding Things Out”, Feynman• “Fooled By Randomness”, Taleb
  4. 4. About Me• Dev, Sr Dev, Team Lead, Architect, VP Eng• PHP since 1999• Savant template system• PEAR Group member• http://paul-m-jones.com• @pmjones
  5. 5. Full Disclosure• Solar framework• Zend framework • Zend_Db*, Zend_View• “Rigor and integrity”• Google Code, now Github
  6. 6. Part 1:In General
  7. 7. Profiling vs Benchmarking• Profiling applies to sub- systems (component design)• Benchmarking applies to entire systems (architecture and interaction)• Profile is to unit test, as benchmark is to system test
  8. 8. Benchmarking Subjects• CPU • Programmer productivity• RAM • Time to initial• Disk access implementation• Database access • Time to add new• Network access major feature• Requests/second • Time to fix bugs -- numeric measurement -- -- control for variables --
  9. 9. Why Bother With Benchmarking?• New site will be linked by Slashdot, Digg, etc.• “Can we handle the traffic, Dodgers?”• “Sure thing, boss.”
  10. 10. Why Bother With Benchmarking?• “...(time to load-test.)”• How many req/sec expected?• Load-tested, optimized in a hurry, balanced
  11. 11. Know Your Limitations• What limits are imposed, and where?• Where to spend effort when optimizing?• Profiling your app will not help with non-app limits• Profiling does not tell you req/sec
  12. 12. The Stack• Web server• Framework (foundation)• Application
  13. 13. Part 2:Getting Started
  14. 14. Methodology• Refining and reporting since 2007• Benchmark the basic system• Add a component, benchmark again• Track responsiveness at each point
  15. 15. Tools• ab, http_load, siege, ...• Emulate concurrent users hitting a URI• Reports req/sec, latency, min/max times, etcab -c 10 -t 60 http://example.com/
  16. 16. abDocument Path: /baseline-html/index.htmlDocument Length: 13 bytesConcurrency Level: 10Time taken for tests: 0.19088 secondsComplete requests: 50000Failed requests: 0Total transferred: 15950000 bytesHTML transferred: 650000 bytesRequests per second: 2619.45Transfer rate: 835.60 kb/s received
  17. 17. http_load4278 fetches, 325 max parallel, 25668 bytes6 mean bytes/connection855 fetches/sec, 5130 bytes/secmsecs/connect: 20.0881 mean, 3006.54 max, 0.099 minmsecs/first-response: 51.3568 mean, 342.488 max, 1.423 minHTTP response codes: code 200 -- 4278
  18. 18. siege2009-02-26 16:35:22, 256869, 60.28, 3, 0.00, 4261.26, ...
  19. 19. Combined Tooling• Each tool (ab, http_load, siege) has different options, params, and target specifications• http://github.com/pmjones/php-framework- benchmarks./bench/(ab|http_load|siege) target/all.ini
  20. 20. Benchmark Subject• Run against a test installation• As similar to production as possible• Hardware, OS, server, PHP installation• No database, no application code
  21. 21. Control for Known Variables• Network latency (use localhost)• Server processes (turn everything off)
  22. 22. Outer Limits Of Responsiveness• Amazon EC2 “Large” instance, 64-Bit Ubuntu 10.04 (Alestic)• Apache 2.2 (stock install)• PHP 5.3.3, APC, Suhosin (stock install)• “index.html” and “index.php”
  23. 23. HTML and PHP Graph 2814.62 ab 2170.31 3195.91http_load 2496.59 4262.12 siege 3148.12 0 1250 2500 3750 5000 html php (10 concurrent users, 60 seconds)
  24. 24. HTML and PHP Percentage Change html php pctab 2814.62 2170.31 77.11%httpload 3195.91 2496.59 78.12%siege 4262.12 3148.12 73.86%
  25. 25. Control for Unknown Variables• Run multiple times• No two runs will be identical; look for trends• File-based sessions will cause exponentially declining returns• session_id(‘constant_value’)
  26. 26. Other Problem Indicators• PHP faster than HTML?• Byte counts?• Socket availability?
  27. 27. Benchmarking as Resource Prediction• About resource planning, not speed (per se)• With supply of “R” requests/ second, you need “S” servers for demand of “T” traffic• What stack components can you modify to increase “R” so you can handle more “T” with same or fewer “S”?
  28. 28. Part 3:Single Frameworks
  29. 29. Single-Framework Methodology• Minimal, not full, application• Sane config• No action code• No view layout or helpers• Static view text• No caching• No database
  30. 30. Framework (Foundation) Benchmarking Web Server + PHP Framework Boot Front Page Action View• Dynamic dispatch cycle as the limiting factor• Improvements off critical path will not increase responsiveness• Page cache?
  31. 31. Single-Framework Analysis• Now you know how much you need to improve• Work on the critical path• Iterative benchmarking
  32. 32. Benchmarking as Engineering Aid• Every architecture decision has a cost• Compare benchmark stats between versions• Are added features are worth req/sec cost?• (Thanks, Laura Thomson.)
  33. 33. Part 3: MultipleFrameworks
  34. 34. Multiple-Framework Comparison• Examine responsiveness in context of other systems• Compare architecture costs between systems• Use the same minimal app to remove variability in application code
  35. 35. Compare Like With Like• Frameworks to be compared should be reasonably similar in “style” or “class”• PHP5, design patterns, front & page controllers, controller & view separation, plugins or extensions, no globals• Cake, Lithium, Solar, Symfony, Yii, Zend• CI, Kohana
  36. 36. Multiple-Framework Methodology (1/2)• Minimal application• Reduced or optimized config• Production mode• No debugging or logging• No database connection• No layouts or view helpers
  37. 37. Multiple-Framework Methodology (2/2)• APC to reduce filesystem access• Restart web server between runs (APC)• 5 runs of 10 concurrent users for 60 seconds, averaged• Calculate req/sec as percent of PHP (thanks, Richard “Cyberlot” Thomas)
  38. 38. Framework NumbersTarget relative average 1 2 3 4 5baseline/html 1.2823 3220.56 3227.18 3226.03 3207.75 3222.63 3219.20baseline/php 1.0000 2511.54 2521.14 2509.00 2506.45 2508.48 2512.63cake/1.3.7 0.0412 103.46 103.37 103.73 103.93 102.55 103.70ci/1.7.3 0.2946 739.95 722.08 725.95 722.96 768.37 760.40kohana/3.0.9 0.2912 731.37 696.83 757.46 743.72 720.10 738.76lithium/0.9.9 0.0827 207.62 252.87 249.94 35.19 249.53 250.58solar/1.1.1 0.1609 404.17 405.08 404.33 403.12 399.63 408.68symfony/1.4.8 0.0784 196.91 197.00 197.77 199.00 194.93 195.87symfony/2.0.0alpha1 0.1675 420.73 422.02 424.85 422.48 411.20 423.08symfony/2.0.0pr4 0.0897 225.22 224.59 221.18 225.88 227.73 226.71yii/1.1.5 0.1891 474.82 500.21 439.74 456.77 494.82 482.57zend/1.10.2-minapp 0.1136 285.28 279.83 291.61 280.61 287.00 287.37zend/1.10.2-project 0.0832 208.84 215.74 206.20 209.26 207.75 205.25
  39. 39. Framework Graph (100%) baseline/html baseline/php cake/1.3.7 ci/1.7.3 kohana/3.0.9 lithium/0.9.9 solar/1.1.1 symfony/1.4.8 symfony/2.0.0alpha1 symfony/2.0.0pr4 yii/1.1.5 zend/1.10.2-minapp zend/1.10.2-project 0 0.375 0.750 1.125 1.500
  40. 40. Framework Graph (30%) cake/1.3.7 0.04 ci/1.7.3 0.29 kohana/3.0.9 0.29 lithium/0.9.9 0.08 solar/1.1.1 0.16 symfony/1.4.8 0.08symfony/2.0.0alpha1 0.17 symfony/2.0.0pr4 0.09 yii/1.1.5 0.19zend/1.10.2-minapp 0.11 zend/1.10.2-project 0.08 0 0.10 0.20 0.30
  41. 41. What Does This Mean?• Both more and less than you think it might• Speed is not the only consideration • Features, intangibles • Unlikely to change minds of devotees• Even the slowest is still pretty fast• Each AJAX call is one request
  42. 42. Failure Modes• Apache/Lighty, APC/XCache, mod_php/ fcgid• EC2 load fluctuation between/during runs• Order of testing• Different concurrency or duration• Added features and app code will reduce dynamic response ... but curve is unknown.
  43. 43. Links• http://github.com/pmjones/php-framework-benchmarks• http://code.google.com/p/web-framework-benchmarks• “New Year’s Benchmarks”: http://paul-m-jones/archives/238• “Do All Frameworks Suck?”: http://www.laurathomson.com/2007/07/do-all- framewor.html• Google: "cargo cult science"
  44. 44. • Questions?• Comments?• Criticism?• http://joind.in/2498
  45. 45. Objections• “Highly profiled.”• “‘Hello world’ is unrealistic.”• “Not optimized for ‘hello world.’”• “Faster for ‘real world’ applications.”
  46. 46. Avoid Frameworks?• If framework code is only at 4-30% of max PHP, should we not use frameworks?• “Slow” relative to what?• Compare like with like, and proposed systems to existing systems.• You will use a framework, even if you write it yourself.• The good comes not from the code or the features, but process standardization.
  47. 47. Thanks!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×