Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Profiling for Grown-Ups


Published on

If you profile your applications, symfonys builtin profiler, the web debug bar and the xhprof bundle provide a lot of service to introspect your application. But what happens if you can't optimize your php code anymore? If there is no clear bottleneck inside the database or in the php code? If the system only slows down under certain, "phase of the moon" circumstances? This talk shows profiling solutions beyond sql-logs, xdebug and xhprof. It demonstrates how to get an overview of everything that happens inside your webserver or database nodes. Tools like cachegrind, callgrind, sysprof, mytop and perf are shown.

Published in: Technology

Profiling for Grown-Ups

  1. 1. Profiling for Gr!w"#p$ (107 slides to go ... )
  2. 2. H%!Hi Everybody! Nice to be here in berlin!
  3. 3. L&‘$ w(% #"%) &v&r*b!+* %$ $&(&+ - !r )&, :-)Let‘s wait until everybody is here. Actually i haven‘t enough slides, so i try to spend sometime useless slides.
  4. 4. H%, -r&( ! $&& Y!# .&r&!A, so that‘s why we waited. Great you came, too.
  5. 5. H&* -!!+ )!!/%"‘!I am only in software business for all the pretty people around
  6. 6. G!!+ 0!r"%"-!Who is tired?
  7. 7. Y&(., % (0 %r&+, !!Yep, me too. That‘s the bad thing about berlin. In munich - where i am from - you getenough sleep because there is no
  8. 8. O/, )&‘$ $(rBut let‘s start anyway
  9. 9. W%. ( $.!r $!r*.I am old. And you know what old people do. They tell you war stories
  10. 10. A1#())* w! $.!r $!r*$.two war stories, actually.
  11. 11. War Story 1The first one, already some years ago.
  12. 12. Big On-Demand V%+&! Sr&(0%"- PlatformIt was one of the first video on demand platforms around. Large scale, with some uniquelicensed live events that were only streamed using this platform.
  13. 13. B($&+ !" (" &"&rpr%$& &1!00&r1& $!)#%!"To build a solid solution they decided to use an enterprise ecommerce solution as the core.So basic performance problems shouldn‘t happen, since it was already enterprise.
  14. 14. W& (r& pr!f&$$%!"()$!The other reason it should scale was us. they wanted to have mayflower in the team to careabout performance und scaling issues.
  15. 15. +!"& r%-. Server SetupSo we cared about the server setup. We implemented a small testing environment,benchmarked the resource usage and calculated that we‘d need 12 4-core machines for astart.
  16. 16. Performance Testing +!"& r%-., !!When the cluster was ready. We did a classic check to see if it does scale.
  17. 17. root@local:/# ab2 -n 50000 -c 500 http://lb.ours.tld/start This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http:// Licensed to The Apache Software Foundation, http:// ... Requests per second: A_LOT [#/sec] (mean) Time per request: REALLY_FAST [ms] (mean) Time per request: EVEN_FASTER [ms] (mean, across all concurrent requests)We first run Apache-Bench. Everything looked pretty good.
  18. 18. Then we simulated user traffic using jmeter to create realistic traffic.
  19. 19. Pr!2)%"- %"1)#+&+!So we tough: hey, how professional we are
  20. 20. Everything was 2"&.Until 10 minutes before the life event. Then everything was $#1/.
  21. 21. O!p$.That wasn‘t this brilliant
  22. 22. War Story 2Second war story, not this long ago.
  23. 23. C)!#+ Pr!3&1 100% HipsterIt was a classic hipster cloud project. Yes, there is a memcache, a rabbitmq,mogilefs,ejabberd, varnish, nginx and cloud! Uh, and a deployment pipeline, of course, all managed bypuppet.
  24. 24. I"&r"() C)!#+We started on our internal cloud - based on eucalyptus 1.6. Anybody else worked witheucalyptus? Hate it?However, we developed it there. It was fast and snappy. We were happy, and so was ourcustomer.
  25. 25. At some day it went live, and it worked great. Just like we expected it to work. Just a bit lesssnappy. Responsivity wasn‘t as good.
  26. 26. F$1/%"- S)!wActually, it was very slow, more than 2 seconds for the start page, more than a second forevery logged in page.
  27. 27. P&rf!r0("1& Op%04(%!" FTW! ORM-Level-Caching APC Template-Caching MemCacheSo we did a lot of work. There was a caching-layer. And another one to support it. And one inthe ORM. And one for the database data. And so on.
  28. 28. S%)) $)!!!!!!w.But it still was slow. We tried it again at our local cloud, where it was fast.
  29. 29. T&$: Put it on 1 LaptopSo we took an really old Lenovo R61 Laptop, and installed everything on it. MySQL, MogileFS,RabbitMQ, eJabberD, and the whole set of production data.
  30. 30. B)5%"-)* f($(-(%".And it was blazingly fast again. On a laptop, with 4G ram, not even a SSD was installed.
  31. 31. And we were like „What the fuck?“ This was a solid medium sized amazon ec2 installation,mostly based on large vms. and it was slower than an old laptop?
  32. 32. S!, .(‘$ w.( .%$ ()/ %$ (b!#. Stuff about profiling we never wanted to know.That were the stories i wanted to tell. It‘s about the things i never wanted to know aboutprofiling
  33. 33. J!.(""-P&&r H(r0("": H(1/&r-#r"&+-M("(-&r H(1/&r ( .&(r. C!0p("*: M(*6!w&r G0bH (f!#"+&+ S&/%!"E%"$ G0bH, !!)I am actually an old school php hacker, i even did a talk at the first php conference ever in2000. Now i am the CTO of mayflower gmbh, we do php development. a number of phprelease manager has been working for our company. we love php.I founded sektioneins together with stefan esser, too. Ask me about security.
  34. 34. A"+ .%$ %$ !#r )!-! +!"& w%. b(1!".And this is the mayflower logo, done with bacon! We do agile, devops, slacktime withnodecopter projects.
  35. 35. Ev&r* "&w %0p!r(" pr!3&1 %$ +!"& w%. $*0f!"* 2.We do every new, important project with symfony 2.
  36. 36. S*0f!"* 2: I‘v& -! !")* b($%1 /"!w)&+-& S*0f!"* CMF $("+b7 w($ .& !ffi1%() )(b r( f!r .%$ ()/But i don‘t know a lot about it. My colleagues do, like paul, who is going to do a short talkabout fancy reactive programming symfony stutt tomorrow.
  37. 37. Z&"+ Fr(0&w!r/ 1: ( )! !f /"!w)&+-&i did a lot of stuff with zend framework, though, maybe that‘s the reason i end up with aprofiling talk here :-)For sure it‘s one of the reasons we do symfony now :-)
  38. 38. Pr!2)%"-Who in here uses the zend Profiler?Who uses XDebug?Who xhprof?Symfony web debug toolbar?Web Debug toolbar with xhprof?Server site profiling like Valgrind? Sysprof? oprofile?
  39. 39. W.* pr!2)%"- ("*w(*?Why are you doing it? Imho there are two major reasons:
  40. 40. Reason No 1 T! 2-#r& !# .!w ! $#rv%v& .& )(#"1. / #p+(& / "&w p!p#)(r%*First, you really want to survive the launch. or a new version update. or a tv spot for yourwebsite tonight.
  41. 41. Reason No 2 T! 2-#r& !# w.* *!# +%+ "! $#rv%v& .& )(#"1. / #p+(& / "&w p!p#)(r%*Second reason: you want to know why it did not work out.
  42. 42. 9& #-)* r#. (b!# pr!2)%"-: The first moment You are certain about your applications performance is one day after the launchThe bad thing about profiling is that you really, really want to know if your website staysonline, all of your customers are happy and everything is up and running. Even with atelevision campaign. But you won‘t get a guarantee anyway, and in fact everybody knows this.
  43. 43. 9($ w.* *!#r b:$ .%"/$ %‘$ .& r&$p!"$%b%)%* !f .& $!,w(r& -#*$.And that‘s why your boss thinks you are the one to blame. Maybe some very stupid featurejust needs a lot of computation work to be done, the reason for a bad responsivity of theserver is always slow code, not a slow feature.
  44. 44. 9($ w.* *!# .%"/ %‘$ .& r&$p!"$%b%)%* !f .& !p&r(%!" -#*$.And, of course, there is the operations team and the operations infrastructure. If the systemslows down, it‘s their job to make it fast again. So if their hardware is not fast enough, theyshould get some faster iron.
  45. 45. O" .& !.&r .("+ $%+&, .& !p&r(%!" -#*$ .%"/ .& +&v&)!p&r$ +!"‘ 1(r& (b!# p&rf!r0("1& ("*w(*.On the other hand site the ops guys thinks that the software developers simply write slowcode. which, in a lot of cases, they actually do.
  46. 46. Strategy No 1 M(/& % .(r+ ! b)(0& Y!#.
  47. 47. Cr&(& (" %0pr&$$%v& pr&$&"(%!"!So we use a common, well established strategy: CYA.
  48. 48. W& +! 300 R&q/$!And we do a little benchmark based on apachebench. And we prove that we are able toservice 300 requests in one second. How great we are!
  49. 49. N!, *!# +!"‘Actually, we did not. You measured something that does not exist.
  50. 50. U")&$$ $!0&b!+* ...... .(1/&+ *!#r "&w!r/... $(r&+ 50 w30 %" p(r())&) ("+... pr&$$&+ „r&)!(+“ 6 %0&$ ( $&1!"+
  51. 51. W.( &v&r*"b!+*r&())* w("$ ! /"!w
  52. 52. 1W.( (r& .& !++$ .( &v&r*.%"- %$#p ("+ r#""%"- (,&r .& )(#"1.?
  53. 53. 2I$ .&r& ("*.%"- w& $.!#)+; b&f!r& w& )(#"1.?
  54. 54. 3D! w& /"!w .!w ! ; ("*p&rf!r0("1& %$$#& f($?
  55. 55. 1 Real Test ScenarioB&1(#$&1 r&q#&$ != 1 !# !f 1000 r&q#&$$First you need a test scenario that is close to your current or expected reality.
  56. 56. 1 JMeter Virtual Users (Personas)First: use a proper Load generation tool like jmeter, webload, silk performer or similar. Createdifferent thread groups . Personas: In marketing and user-centered design, personas are fictional characters created torepresent the different user types within a targeted demographic, attitude and/or behavior set that might use a site, brand or product in asimilar way.
  57. 57. That‘s a simple jmeter setup. Who already uses jmeter? 1
  58. 58. Ar& 0* &$$ r&()%$%1?
  59. 59. If *!#‘r& ()r&(+* 1 !")%"&If you already launched it‘s easy to answer. Simply take a look at your existing web traffic.
  60. 60. Google Analytics Webalizer W&bA"()*%1$ f!r V()%+(%!" Analog AWStatsIf you already know what is happening on your website: great.Just do a webalizer or analoguerun on the access_log of a normal hour, and another one done with your benchmark. If theresult looks similar, you are generating the right kind of traffic.
  61. 61. C!rr&)(%!": 99,5%In a few cases we did some validation using proper statistics, where we did analyzed thecorrelation.
  62. 62. If *!# .(v&"‘ 1 )(#"1.&+ *&If you haven‘t got any data, and we are not talking about an intranet application for exactly50 people who start working at 9 am, than you have to guess was is going to happen
  63. 63. 1 Create Scenarios based on Personas: 1 Worst Case Users doing all the expensive stuff 2 Expected Average „Normal“ user behaviorSince you don‘t know where you are going to end up, simply create 3 scenarios that try toshow the range of your applications performance.
  64. 64. Monitoring 1 M#"%" C(1% JM&&r-P&rf0!"On the other hand side you need to figure out what happens on the server side. Maybe youalready got a monitoring solution installed, like cacti or munin. Cacti anybody? Planning tochange? Jmeter provides an agent based monitoring solution, it‘s part of the jmeter pluginpackage.
  65. 65. Most of the time we use munin, since it‘s imply already there M#"%"
  66. 66. source: can configure perfmon to show graphs like „CPU load based on number of users“ and thelike
  67. 67. Interesting StuffCPU User, System, IOWait, Interrupts,Context switches, ForksM&0!r* PageFaults, Swap, Free MemoryIO Network,Connections, Harddisk, SSD
  68. 68. A lot of consoles.And of course a lot of consoles. To monitor everything. I am not talking about gameconsoles, btw. :_
  69. 69. T!!)$ ! +! $!0& ("()*$%$: top iotop vmstat sysprofTop - everybody should know this one. iotop: a small python script, already part of all mayordistributionsvmstat display was originally about virtual memory statistics (hence the name), but doesprocesses, interrupts, paging, block io and a lot of device statistics as well. Sysprof is astatistical, systemwide profiler i‘ll show later.
  70. 70. Ramp up? app/console does the cache warming!Now test all your scenarios. you don‘t need any ramp up since your favourite frameworkalready does the cache warmup :-)
  71. 71. P:%%v& !#1!0& Application still works System is stable Good service qualityMaybe there is some high load at the beginning, but maybe the systems get stable again. Ifeverything is alright, we probably bought too much hardware.
  72. 72. N!w w& .(v& ( $&#p f!r r&()%$%1 pr!2)%"-But that‘s cool, that‘s the perfect starting point for realistic profiling.
  73. 73. N&-(%v& !#1!0& Application still works System is stable Good service qualityBut what happens if it goes wrong? if we are not able to provide the scalability orresponsitivity needed?
  74. 74. N!w w& .(v& ( "&&+ f!r r&()%$%1 pr!2)%"-Now we have to do some profiling anyway :-)
  75. 75. Default H&*, %‘$ ()r&(+* %" 0* IDE! A"+ !" 0* Server!By default you do the profiling while development in your IDE. Who has a profiler enabled? Onyour local or on a close-to-production system? Lets have a closer look.
  76. 76. S%"-)& R&q#&$ %" 30&&r: 2230$
  77. 77. S%"-)& R&q#&$ %" X+&b#-: 14080$
  78. 78. Ev&r*.%"- %$ #p ! 12 %0&$ $)!w&r A )&($ .(‘$ w.( !#r b&"1.0(r/$ &)). B# .&*, %‘$ r&)(%v& p&rf!r0("1&!We did some benchmarking in different environments, and the slowdown caused by xdebugwas up to 12 times. That‘s quite a big difference. Now you could say: yeah, but that is justrelative.
  79. 79. 20 0$ +((b($& q#&r* %" ( 100 0$ r&q#&$ v$ 20 0$ +((b($& q#&r* %" ( 1.200 0$ r&q#&$ 20% vs 1.6 %The problem is, that this makes it close to impossible to find race conditions. or to measurethe influence of external inout. See the example: a slow query my result in 20% of the requesttime, or in below 2% in yur profiling environment.
  80. 80. Y!#r PHP pr!2)&r )%&$.
  81. 81. What does really happen in your application? H&*, %‘$ .& A#!)!(+&r!So, what does really happen in your application. You can see it here - xdebugs callgrind-format export.Who knows this one? Obviously most of the time is spend inside the autoloader. That‘swellknown, i know.So, do we know everything that is happening there?
  82. 82. Wall Clock Time W& /"!w .( %0& w($ $p&", b# w& +!"‘ /"!w .!w .& %0& w($ $p&"xdebug measures wall clock time, and that is fine. but that means you only know that timewas spent, not how it was spent.
  83. 83. O"& )&v&) +&&p&r
  84. 84. I"$()) v()-r%"+ ("+ ()) "&&+&+ +&b#--%"--$*0b!)$ $(r .& 30&&r &$ r#" !"& (p(1.& 1.%)+ $&p(r(&)* valgrind --tool=callgrind /usr/sbin/apache2 -f /etc/apache2/apache2_single.conf -XSo let‘s have a look at valgrind / callgrind /cachegrind.It‘s nice to have all debugging symbols installed to see what‘s happening. Start a singleapache child in valgrind while the jmeter test is running. Attention: this single request is a lotslower, just like xdebug.
  85. 85. What does really happen in your application? H&*, %‘$ .& P(r$&r!Again we see what really happens - now most of the (self) time is spend inside the parser.
  86. 86. APC FTW!Luckily both issues can be fixed with APC and the ApcUniversalClassLoader
  87. 87. O"& )&v&) +&&p&rBut anyway, there is more way to go.
  88. 88. SysprofPr!2)%"- .& &"%r& )%"< $*$&0,#$&r- ("+ /&r"&)$p(1& f($ pr!2)&r w%. )!w 2"-&rpr%"Sysprof tells you about everything that is happening inside your cpu. and, the performanceimpact is okeyish
  89. 89. root@local:/# sysprof-cli myoutfile ... wait some seconds to gather enough data ... ... stop with ctrl-c when you are done ... root@local:/# sysprof myoutfileAnd that‘s how it works: when the jmeter test is running, sysprof-cli is started for some time.It‘s stopped with ctrl-c and writes the data to the filesystem. this file can be viewed withsysprof itself, which is a X11 software.
  90. 90. T&$ r#" w%. =+&b#- )!(+&+ 18% of the time spent to look at the watch?!Most of the time is spent within PHP, and that‘s expected to happe. but half of the time isspent within gettimeofday, a system call that is rather slow, especially when used in 32bitenvironments. so 18% of the whole cpu time is used to look at the wallclock.
  91. 91. perfS(0p)%"- 1#rr&" 1())$ !r &v&"$ &v&r* f&w$&1!"+$ f($ pr!2)&r w%. )!w 2"-&rpr%"
  92. 92. root@local:/# perf record -a -F 10000 sleep 60 ... creates 600.000 samples ... ... saving result in ... root@local:/# perf reportBecause the memory bus is slow ...
  93. 93. War Story 2Now we have got the toolset to look at our war stories.
  94. 94. L&"!v! R61: f($E"&rpr%$& C)!#+ C)#$&r: $)!w
  95. 95. D&f(#) pr!2)%"- Xdebug profiling xhprof profiling iotop, vmstat, ... Ev&r*.%"- %$ 2"&, b# $%)) $)!w?
  96. 96. $*$pr!f page_fault is the reason?! R&($!": XEN 0&0!r* $.(r%"- !" (05!" &12We saw a lot of page_faults. but there was no swapping. So this should not happen?!_very_ expensive page_faults due to memory sharing in XENSolution: memory sharing, pages marked as read_only -> page_faultand this very page_fault was slow in 32bit guests in 64bit hosts.
  97. 97. War Story 1The first one, already some years ago.
  98. 98. Everything was 2"&. Until 10 minutes before the life event. Then everything was $#1/.You remember? Where we went not only live, but dead.
  99. 99. D&f(#) pr!2)%"- Xdebug profiling xhprof profiling iotop, vmstat, ... cacti monitoring ... Ev&r*.%"- %$ 2"&, b# $%)) $#1/ w.&" % 0(&r$?And we have had the same problem - everything looked fine in normal profiling.
  100. 100. W(% f!r r!#b)& 50 terminals running top, atop, htop, iotop, mytop etcAnd we waited, and when the next wave hit us we could see it in mytop: next slide
  101. 101. source: a lot of mysql queries waiting for a lock
  102. 102. $0(r 0*$q)-pr7*-)!--%"- See root@local:/# mysql-proxy -P --admin-username=root --admin-password=mypw --admin-lua-script=/usr/share/mysql-proxy/admin.lua -s /usr/share/mysql-proxy/debug-blind.luaab2 -n 50000 -c 500 http://lb.ours.tld/start 12,345678 ms select * from mytable ... 2,123456 ms insert into mytable ...This creates a mysql proxy listener on port 4400, that forwards all request to the local server.Logging all queries with execution time, removing all unneeded whitespaces. for the _whole_system, nod for a single process.
  103. 103. > 500 r&-%$&r%"- ("+ > 1000 )!--%"- %" #$&r$ %" .& $(0& 0%"#& == T(b)& )!1/#p(Yep, solution was simple: MyISAM -> InnoDB)
  104. 104. O.&r w(r $!r%&$: IBM /&r"&) 0!+#)& $1r&w%"- IO %" 1!"1#rr&"1*, b# .%+%"- ("* %!w(% I"-0&0!r*-C(1.%"- )(*&r$ +!&$ 0!r& b(+ .(" -!!+We‘ve got more war stories of this kind, mostly fixed with tools like oprofile, sysprof, vmstatand the like.
  105. 105. W.* $.!#)+ % 1(r&? B&1(#$&: T!+(*$ $*$&0 (r1.%&1#r& %$ 1r&(&+ b* +&v&)!p&r$ D&vOp$: Y!# (r& r&$p!"$%b)& f!r pr!+#1%!", !! Y!# 1("‘ ; ( !p&r(%!"() b#- 1(#$&+ b* (pp)%1(%!" 1!+& !.&rw%$&Now you might ask: why should i care? i am a developer, not a system administrator?Because:
  106. 106. W.(‘$ %" f!r *!#? A recommended setup (but not very surprising) Setup: Symfony CMF sandbox 1400 requests with concurrency 20
  107. 107. Pure PHP With APC With Xdebug 6 APC With APC, with New Relic With APC,XHProf 5000 2698 3750 2500 1250 0 Low Average Highnew relic is too slow for production right now.
  108. 108. Pr!2)%"-? T(/& .& r&+ p%)) Thanks!Sysprof Perf Valgrind/CallgrindMytop Mysql-proxy Xdebug 9("/$!