Downloaded 16 times
Big server-is-watching-you
- 1. Big server iswatching you Maurice Kherlakian Professional services consultant maurice.k@zend.com twitter.com/zend http://ca.linkedin.com/in/mkherlakian
- 2. About me • Originally from Lebanon, based in Montreal since 2003 • BS in Computer Science from the American University of Beirut • Speak english, french and arabic • Been using PHP for the past 10 years • Worked on some large and very heavy traffic websites (4M+ page views/day) • Zend Consultant for the past 2 years • Had the opportunity to work on some very large accounts like NYSE, Guthy- Renker, Best Buy, Levolor, Harvard, UBM, Shaklee… • Deliver consulting (architecture and performance) and training (all of Zend’s courses) • Married, have a 14 months old daughter and 2 persian cats • During little free time I fly (but mostly crash) R/C planes and helicopters 2
- 3. About the session • Covers Zend Server’s monitoring and profiling features • Monitoring and code tracing through application examples • Session is demo-based 3
- 4. Problem Statement • Applications get developed and deployed on disparate environments • Focus typically on speed of development (time crunches), which can leave sub-optimal code performance-wise • Test data typically different than production data Production data sets are typically much larger, leading to different scenarios Production users are notorious for entering data that makes the application break • Failure, and especially performance problem reports are vague • They are also very difficult to reproduce 4
- 5. So how dowe capture those problems as soon as they occur? 5
- 6. The application • Needed an application to demo • Customer applications and examples would have been ideal but… • Created BeerIOU • DISCLAIMER: The application is intentionally written not to follow best practices, and has bad code 6
- 7. The problems Slow page • Captured a slow request execution event • The Monitor and its rules • How do we diagnose it? By using code tracing • Problem was that API randomly took long times to load 7
- 8. The problems PHP errors • Captured a PHP error event • How do we diagnose it? By using code tracing • Problem was that API changed return format 8
- 9. The problems High memory usage • Captured a High memory usage • How do we diagnose it? By using code tracing • Problem was that favorite beers DD has too many entries instantiated into objects • Caching is not always the best option 9
- 10. The problems Slow web service • Maintaining highly scalable infrastructure means being able to predict response times • The less time an Apache process spends serving a request the better • When dealing with unpredictable loads, offload the request processing (using Job Queue for example) 10
- 11. Performance impact ofmonitoring and code tracing • Monitoring is not free (in terms of resources, not like free beer) Stress/load test with monitoring and CT enabled/disabled Results may vary ;) • Code tracing buffer Larger means more memory Can run in stand-by mode 11
- 12. Monitoring and codetracing on/off benchmark • Test-get-beer 10 seconds, 6 concurrent requests cache enabed cache disabled Monitoring and CT off 3.65 rps / 298Mb 1.40 rps / 226Mb Monitoring ON, CT off 3.39 rps / 315 Mb 1.37 rps / 280Mb Monitoring and CT on 3.37 rps / 327 Mb 1.28 rps / 275Mb rps % on/off 108.31% 109.38% Memory % on/off 109.73% 121.68% 12
- 13. Home page • Home 10 seconds, 6 concurrent requests cache enabed Monitoring and CT off 79.59 rps / 52.7Mb Monitoring ON, CT off 79.59 rps / 63.4 Mb Monitoring and CT on 79.20 rps / 68.4 Mb rps % on/off +/-0% Memory % 129.79% 13
- 14. Additional features • Events can be set up to send emails Default for all events Individually per event • Events management (open, closed, re-open, ignore) • Monitor and Code tracing’s main advantages They can run un production (Optimizer + is always active) Easy interface, easy to use • Monitoring can be disabled on a per-page basis Using zend_monitor_event_reporting API 14
- 15. Conclusion • Monitoring is an essential part of application lifecycle Applies both for development and production • Not enough to rely on inaccurate reports • Some sysadmin tools (Gomez, Splunk, Xymon, Cacti) are great for server monitoring but none monitor PHP itself internally 15
- 16. Next steps • Download Zend Server (http://www.zend.com/en/products/server/) • Generate a 30 day trial license (http://www.zend.com/en/products/server/license) • Test it out with your own application 16
- 17. Questions? Questions? 17
- 18. Thank you Thank you Email: maurice.k@zend.com Twitter: twitter/mkherlakian Linekdin: http://ca.linkedin.com/in/mkherlakian The full source code for this demo can be found at: git@github.com:mkherlakian/BeerIOU 18
Editor's Notes
- #4 About the session:- Zend server session, with focus on Zend Server monitoring/code tracing- Could have talked about Zend Server, but thought that would be more interesting to show how I use Zend Server on an almost daily basis as a Zend consultant, especially for perf. audits- session is largely demos
- #7 Introduce Beeriou: In order to introduce one of the possible solutions, needed an application. Since cannot use customers applications, I built one - That's what we do in our spare time anyway, right? Since Zend is a ver global company, we use Skype a lot. And when we help each other out (we're nice like that :)), the last thing we say to one another is usually: "I owe you a beer!", usually redeemable at Zendcon since it's the time of year when we all get together. That makes a lot of beers to remember, so I introduce to you the Beer IOU project.The application is intentionally written to not follow best practices of development (otherwise it would preform well). It also features a killer design (sorry about that). It is built to illustrate the most common problems I've seen at customer's sites, and how I address them using Zend Server.So let's jump right in.(Show application – refresh a couple of times)
- #8 Explain, after showing the slow request execution, the functioning of rules management1- Home page -> notice that the home page is sometimes slow. Since it's the first time I see the application, I don't know what's going on.--- PAUSE go into Zend Server Monitor ---Introduce monitor -> but very briefly. Go over main events, then go to the rules management, and explain that Monitor is rules-based and allows for modifications of thresholds.Then go into the actual slow request execution event--- RESUME ---We have all the info on the event. We know that the page is slow, we caught it in monitor, we don't need user input to know. So big whoop, we know. what's the next step?ZS:CT: activeActivate code tracing.Re-run the page load a few times. get event, open code trace. It's the call to the external web service that allows us to get bar locations. Solution? cache the call.
- #9 2- Home page -> bar locations changed the APIbeerlocationsurl: http://barlocations.vm/index/index/break/1Show homepage. Breaks on the bottom, but only slightly. User might report it. Obviously the problem may be more serious, in case of a payment gateway API, or a product catalog which changes only one attribute.Show the event, click show file in Zend Studio -> show the line. So it seems that the attribute is not in the json feed. Let's check CT.After examining the code we figure out that the req/resp uses zend_http_client. CT shows the response, and shows the json string -> address has been concatenated into address fieldShow function error as well if possible? ---Let's disable API all togetherbeerlocations: disabled
- #10 3- MyAccount -> slowNotice that the myaccount page is slow - let's check the event--- PAUSE go into events details here ---In an event's detail, aggregation. All POST/GET/COOKIE/SERVER varsif within 5 mins, only the counter is increased to prevent over-logging--- RESUME ---we know it's slow, what do we do? well check the code trace of course!But wait, code tracing is being truncated... need to increase buffer.increase to 10Mb. It's not enough, increase to 50Mb. Now it's enough.--- PAUSE talk about buffer ---when you increase the buffer, you increase the memory utilization per Apache process. use wisely.--- RESUME ---Code trace clearly shows that we are instantiating 8000 objects getting them from DB, and that this has a perf impact. (use of memory)open: atop 1Try a load test:Explain that for purposes of testing simulate in test controller to avoid dealing with authentication. But this could be set up in JMeter as well to do full test.# ab -t 20 -c 4 http://beeriou.vm/test/test-get-all-beersresults: 1 req/secincrease -c to 10# ab -t 20 -c 10 http://beeriou.vm/test/test-get-all-beersSame results 1.5 req/sec, avg response time 6.0secTry enabling caching: (uncomment line in test controller)# ab -t 20 -c 10 http://beeriou.vm/test/test-get-all-beers3 req/sec. avg response time: 2.6 secNotice CPU usage sky-rocketing because of serialization/deserialization into memoryMoral of the story: caching is not a silver bullet. Do not abuse it.
- #11 4- Job queue async web service requestShow syncShow asyncShow job being queued. highlight passing variables betweeen script and job. Show disabling events monitoring.
- #12 The performance impacts of monitoring:- monitoring doesn't come for free in terms of resources, neither does code tracing. Do stress tests with Monitoring enabled/disabled to get the true measures. Results will vary ;)- code tracing's buffer size - remember? code tracing engine never stops running on the server. The difference is when an event is caught as defined by rules, it is persisted to disk.- Code tracing can run in stand-by mode. It will not catch the first event error, but it will catch the second event -> why? b/c user's reflex is to hit Refresh! in this mode CT is awoken and will stay awake for duration specified
- #15 - Can send emails to teams/admins. can have default email and specific email address per event- Event management and cleanup - events are automatically purged after specified time. you can manage events by closing them, ignoring them- Zend Monitoring's main advantage is that it can run as a production OR dev solution. Because of the fact that optimizar+ runs regardless it is ver fast and doesn't cause a lot of overhead
- #16 conclusion:- Monitoring is an essential part of an application's lifecycle- It is not enough to rely on vague reports, and user reports when something happensSome tools are great to monitor servers like Gomez, Xymon, Cacti, Splunk! etc. but none monitor PHP internally