Framework and Application Benchmarking
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Framework and Application Benchmarking

on

  • 9,295 views

 

Statistics

Views

Total Views
9,295
Views on SlideShare
6,761
Embed Views
2,534

Actions

Likes
6
Downloads
126
Comments
0

13 Embeds 2,534

http://allframeworks.ru 1247
http://paul-m-jones.com 1024
http://www.allframeworks.ru 241
http://www.sfexception.com 4
http://us-w1.rockmelt.com 4
http://www.techgig.com 3
http://www.bear-project.net 3
http://translate.googleusercontent.com 2
http://xss.yandex.net 2
http://webcache.googleusercontent.com 1
https://si0.twimg.com 1
http://hghltd.yandex.net 1
http://physicsfreaks.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Framework and Application Benchmarking Presentation Transcript

  • 1. Framework andApplication Benchmarking Paul M. Jones PhpBenelux 2011 @pmjones http://joind.in/2498
  • 2. Overview• Benchmarking methodology, results, and interpretation • Server/application • Single frameworks • Framework comparisons• Resource usage prediction• Engineering analysis
  • 3. Read These• “Free to Choose”, Friedman• “Pleasure of Finding Things Out”, Feynman• “Fooled By Randomness”, Taleb
  • 4. About Me• Dev, Sr Dev, Team Lead, Architect, VP Eng• PHP since 1999• Savant template system• PEAR Group member• http://paul-m-jones.com• @pmjones
  • 5. Full Disclosure• Solar framework• Zend framework • Zend_Db*, Zend_View• “Rigor and integrity”• Google Code, now Github
  • 6. Part 1:In General
  • 7. Profiling vs Benchmarking• Profiling applies to sub- systems (component design)• Benchmarking applies to entire systems (architecture and interaction)• Profile is to unit test, as benchmark is to system test
  • 8. Benchmarking Subjects• CPU • Programmer productivity• RAM • Time to initial• Disk access implementation• Database access • Time to add new• Network access major feature• Requests/second • Time to fix bugs -- numeric measurement -- -- control for variables --
  • 9. Why Bother With Benchmarking?• New site will be linked by Slashdot, Digg, etc.• “Can we handle the traffic, Dodgers?”• “Sure thing, boss.”
  • 10. Why Bother With Benchmarking?• “...(time to load-test.)”• How many req/sec expected?• Load-tested, optimized in a hurry, balanced
  • 11. Know Your Limitations• What limits are imposed, and where?• Where to spend effort when optimizing?• Profiling your app will not help with non-app limits• Profiling does not tell you req/sec
  • 12. The Stack• Web server• Framework (foundation)• Application
  • 13. Part 2:Getting Started
  • 14. Methodology• Refining and reporting since 2007• Benchmark the basic system• Add a component, benchmark again• Track responsiveness at each point
  • 15. Tools• ab, http_load, siege, ...• Emulate concurrent users hitting a URI• Reports req/sec, latency, min/max times, etcab -c 10 -t 60 http://example.com/
  • 16. abDocument Path: /baseline-html/index.htmlDocument Length: 13 bytesConcurrency Level: 10Time taken for tests: 0.19088 secondsComplete requests: 50000Failed requests: 0Total transferred: 15950000 bytesHTML transferred: 650000 bytesRequests per second: 2619.45Transfer rate: 835.60 kb/s received
  • 17. http_load4278 fetches, 325 max parallel, 25668 bytes6 mean bytes/connection855 fetches/sec, 5130 bytes/secmsecs/connect: 20.0881 mean, 3006.54 max, 0.099 minmsecs/first-response: 51.3568 mean, 342.488 max, 1.423 minHTTP response codes: code 200 -- 4278
  • 18. siege2009-02-26 16:35:22, 256869, 60.28, 3, 0.00, 4261.26, ...
  • 19. Combined Tooling• Each tool (ab, http_load, siege) has different options, params, and target specifications• http://github.com/pmjones/php-framework- benchmarks./bench/(ab|http_load|siege) target/all.ini
  • 20. Benchmark Subject• Run against a test installation• As similar to production as possible• Hardware, OS, server, PHP installation• No database, no application code
  • 21. Control for Known Variables• Network latency (use localhost)• Server processes (turn everything off)
  • 22. Outer Limits Of Responsiveness• Amazon EC2 “Large” instance, 64-Bit Ubuntu 10.04 (Alestic)• Apache 2.2 (stock install)• PHP 5.3.3, APC, Suhosin (stock install)• “index.html” and “index.php”
  • 23. HTML and PHP Graph 2814.62 ab 2170.31 3195.91http_load 2496.59 4262.12 siege 3148.12 0 1250 2500 3750 5000 html php (10 concurrent users, 60 seconds)
  • 24. HTML and PHP Percentage Change html php pctab 2814.62 2170.31 77.11%httpload 3195.91 2496.59 78.12%siege 4262.12 3148.12 73.86%
  • 25. Control for Unknown Variables• Run multiple times• No two runs will be identical; look for trends• File-based sessions will cause exponentially declining returns• session_id(‘constant_value’)
  • 26. Other Problem Indicators• PHP faster than HTML?• Byte counts?• Socket availability?
  • 27. Benchmarking as Resource Prediction• About resource planning, not speed (per se)• With supply of “R” requests/ second, you need “S” servers for demand of “T” traffic• What stack components can you modify to increase “R” so you can handle more “T” with same or fewer “S”?
  • 28. Part 3:Single Frameworks
  • 29. Single-Framework Methodology• Minimal, not full, application• Sane config• No action code• No view layout or helpers• Static view text• No caching• No database
  • 30. Framework (Foundation) Benchmarking Web Server + PHP Framework Boot Front Page Action View• Dynamic dispatch cycle as the limiting factor• Improvements off critical path will not increase responsiveness• Page cache?
  • 31. Single-Framework Analysis• Now you know how much you need to improve• Work on the critical path• Iterative benchmarking
  • 32. Benchmarking as Engineering Aid• Every architecture decision has a cost• Compare benchmark stats between versions• Are added features are worth req/sec cost?• (Thanks, Laura Thomson.)
  • 33. Part 3: MultipleFrameworks
  • 34. Multiple-Framework Comparison• Examine responsiveness in context of other systems• Compare architecture costs between systems• Use the same minimal app to remove variability in application code
  • 35. Compare Like With Like• Frameworks to be compared should be reasonably similar in “style” or “class”• PHP5, design patterns, front & page controllers, controller & view separation, plugins or extensions, no globals• Cake, Lithium, Solar, Symfony, Yii, Zend• CI, Kohana
  • 36. Multiple-Framework Methodology (1/2)• Minimal application• Reduced or optimized config• Production mode• No debugging or logging• No database connection• No layouts or view helpers
  • 37. Multiple-Framework Methodology (2/2)• APC to reduce filesystem access• Restart web server between runs (APC)• 5 runs of 10 concurrent users for 60 seconds, averaged• Calculate req/sec as percent of PHP (thanks, Richard “Cyberlot” Thomas)
  • 38. Framework NumbersTarget relative average 1 2 3 4 5baseline/html 1.2823 3220.56 3227.18 3226.03 3207.75 3222.63 3219.20baseline/php 1.0000 2511.54 2521.14 2509.00 2506.45 2508.48 2512.63cake/1.3.7 0.0412 103.46 103.37 103.73 103.93 102.55 103.70ci/1.7.3 0.2946 739.95 722.08 725.95 722.96 768.37 760.40kohana/3.0.9 0.2912 731.37 696.83 757.46 743.72 720.10 738.76lithium/0.9.9 0.0827 207.62 252.87 249.94 35.19 249.53 250.58solar/1.1.1 0.1609 404.17 405.08 404.33 403.12 399.63 408.68symfony/1.4.8 0.0784 196.91 197.00 197.77 199.00 194.93 195.87symfony/2.0.0alpha1 0.1675 420.73 422.02 424.85 422.48 411.20 423.08symfony/2.0.0pr4 0.0897 225.22 224.59 221.18 225.88 227.73 226.71yii/1.1.5 0.1891 474.82 500.21 439.74 456.77 494.82 482.57zend/1.10.2-minapp 0.1136 285.28 279.83 291.61 280.61 287.00 287.37zend/1.10.2-project 0.0832 208.84 215.74 206.20 209.26 207.75 205.25
  • 39. Framework Graph (100%) baseline/html baseline/php cake/1.3.7 ci/1.7.3 kohana/3.0.9 lithium/0.9.9 solar/1.1.1 symfony/1.4.8 symfony/2.0.0alpha1 symfony/2.0.0pr4 yii/1.1.5 zend/1.10.2-minapp zend/1.10.2-project 0 0.375 0.750 1.125 1.500
  • 40. Framework Graph (30%) cake/1.3.7 0.04 ci/1.7.3 0.29 kohana/3.0.9 0.29 lithium/0.9.9 0.08 solar/1.1.1 0.16 symfony/1.4.8 0.08symfony/2.0.0alpha1 0.17 symfony/2.0.0pr4 0.09 yii/1.1.5 0.19zend/1.10.2-minapp 0.11 zend/1.10.2-project 0.08 0 0.10 0.20 0.30
  • 41. What Does This Mean?• Both more and less than you think it might• Speed is not the only consideration • Features, intangibles • Unlikely to change minds of devotees• Even the slowest is still pretty fast• Each AJAX call is one request
  • 42. Failure Modes• Apache/Lighty, APC/XCache, mod_php/ fcgid• EC2 load fluctuation between/during runs• Order of testing• Different concurrency or duration• Added features and app code will reduce dynamic response ... but curve is unknown.
  • 43. Links• http://github.com/pmjones/php-framework-benchmarks• http://code.google.com/p/web-framework-benchmarks• “New Year’s Benchmarks”: http://paul-m-jones/archives/238• “Do All Frameworks Suck?”: http://www.laurathomson.com/2007/07/do-all- framewor.html• Google: "cargo cult science"
  • 44. • Questions?• Comments?• Criticism?• http://joind.in/2498
  • 45. Objections• “Highly profiled.”• “‘Hello world’ is unrealistic.”• “Not optimized for ‘hello world.’”• “Faster for ‘real world’ applications.”
  • 46. Avoid Frameworks?• If framework code is only at 4-30% of max PHP, should we not use frameworks?• “Slow” relative to what?• Compare like with like, and proposed systems to existing systems.• You will use a framework, even if you write it yourself.• The good comes not from the code or the features, but process standardization.
  • 47. Thanks!