Object.wait()                                             SOAP Exception            OutOfMemoryException()                ...
ABOUT ME•   Developer•   Product Manager•   Tech Evangelist•   Part-time Superhero
AGENDA                                     A L     Change                               T I                       Drowning...
CHANGE   “To improve is tochange; to be perfect is to change often.”   Winston Churchill     1874 - 1965
HOW OFTEN ARE YOUR          RELEASE CYCLES?                            AppDynamics 2011 Survey: 250+                      ...
ARCHITECTURES CHANGE                   JDBC      Tomcat                MYSQL               Monolithic
ARCHITECTURES CHANGE                                                                      SOAP                            ...
A “simple”login transaction
SLIPPERY SOAP• Media   Customer• Few   Million Users• History   of Performance Issues in Production• Angry   calls from Us...
ARCHITECTURE
SOME WEB SERVICES PERFORM      OTHERS DON’T
EXAMPLE: PAYPALIN 6 WEEKS PERFORMANCE IMPROVED BY 15%
Monitor externalservices (avg, max)  Only Tune if  you have to!
DEATH BY JAVA BEANS• Huge    E-Commerce Customer• 24   Million Business Transaction a Day• Outage    around 6pm in the eve...
ARCHITECTURE
System Metrics look Good
JVM Metrics not so Good
JVM had huge GC time
What caused the GC?Search Transaction uses most CPU, is invoked often          and has the most errors & stalls
Search Concurrency & Response Time    Correlates with GC & CPU Spike
Let’s look at a Search
EJB Call for every Search Result
Every EJB Call makes 500+ SQL Calls12,000+ Queries & associated objects exhausted heap
GC can kill your App!What consumes CPU?Transactions, Objects, DataMinimize DB HitsTweak your Heap Size
DROWNING IN A POOL• EMEA Television   Broadcaster• Reported   slowdowns between 7pm and 10pm• Thousands    of requests tim...
ARCHITECTURE
NEW SERVICE LAUNCHED Traffic on service was doubling every week
CONNECTION POOL> 2,000 trx/min = DB Connections gone.
Developer        Operations   “I did tell      “Wow, thatsOperations this    interesting, letabout a month       me speak ...
Monitor your traffic growthConnection Pools are sacredTweak over-timeTest on one node & verifyTune SQL > 500ms
TUNING A CAR• Amateurs •   Loud Exhaust, Induction Kits •   Stupid Rear Wings, Bling Wheels •   Go Faster Strips, neon lig...
Focus on what really    makes a difference5% of things are responsible  for 95% of performanceBaseline, measure, compare E...
DEADLY LOCK• Huge   US Retailer• $1   billion revenue• Random     Outages & Slowdowns in Webstore• Revenue    would be up/...
ARCHITECTURE
DEADLOCK DETECTEDSYS ADMINS WOULD JUST RESTART SERVERS
THREAD DUMPS
CACHE WASN’T THREAD SAFE
BUSINESS IMPACT?46,463 Checkouts in the Day        Avg. $75 per Checkout 2,492 were impacted          Cost of Deadlock: $1...
Restarting Servers isn’t         enough!Thread Dumps can be a pain in the ass. Exploit Tools.   Make sure caches are      ...
WHAT ABOUT NEW TECHNOLOGIES?   Like Apache Cassandra?
IS NOSQL SLOW?If so, what and how slow?
SEE THE QUERY RESPONSE TIME
CASSANDRA IS A JVM TOO
JMX KPI METRICS
Make Sure You Monitor all Application ComponentsAll software can have bugs      & Bottlenecks!
SO, DO YOU KNOWHOW FAST AND RELIABLE   YOUR CODE IS   IN PRODUCTION?   <1% of Developers actually do
THE PROBLEM           DEV                          QA/TEST                           PRODUCTIONConcurrency Data Volume   R...
TIPS FOR DEV & OPS• Monitor   and Test your Applications in Production• Leverage Application   Performance Management (APM...
WANT TO LEARN MORE?    Visit AppDynamics Booth “Kiss my App” T-shirtDownload Free Monitoring Solution  www.appdynamics.com...
Secret Diary of a Performance Tuning Superhero
Secret Diary of a Performance Tuning Superhero
Secret Diary of a Performance Tuning Superhero
Upcoming SlideShare
Loading in...5
×

Secret Diary of a Performance Tuning Superhero

4,115

Published on

The year is 2012, and many organizations have lost their grip on application performance and scalability as they innovate using agile release cycles and open source technologies. Architectures are at cloud bursting point with logic and data stores becoming more distributed, virtual and elastic than ever before. This session takes a look back at the major performance and scalability bottlenecks of the past year, described using real-life customer case studies. It will cover common architecture pain points, the troubleshooting process and the end root cause many customers faced in production environments. It will also discuss the key challenges and best practices for Dev and Ops teams to manage application performance and scalability in live environments.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,115
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
2
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Secret Diary of a Performance Tuning Superhero

  1. 1. Object.wait() SOAP Exception OutOfMemoryException() Connection Pool LeakThread Synchronization Debug Logging of a Performance TuningThread BLOCKED Superhero Stephen Burton AppDynamics @BurtonSays Slow executeQuery()
  2. 2. ABOUT ME• Developer• Product Manager• Tech Evangelist• Part-time Superhero
  3. 3. AGENDA A L Change T I Drowning in a Pool E N I D Deadly Lock Slippery SOAP N F C O DevOps TipsDeath by Java Beans
  4. 4. CHANGE “To improve is tochange; to be perfect is to change often.” Winston Churchill 1874 - 1965
  5. 5. HOW OFTEN ARE YOUR RELEASE CYCLES? AppDynamics 2011 Survey: 250+ Monthly 17% 2+ Months 1/3 Experienced 58% 21% Weekly Severity 1 Incident 3% Each Month DailySource: http://www.appdynamics.com/blog/2011/12/14/storm-clouds-in-2012-summary-of-appdynamics-apm-customer-survey/
  6. 6. ARCHITECTURES CHANGE JDBC Tomcat MYSQL Monolithic
  7. 7. ARCHITECTURES CHANGE SOAP HTTP JBoss ASP.NET Tomcat ADO.NET SOAP JDBC SQL 3rd Party Server SOAP Web Service MySQL Tomcat JMS Tibco BW HTTP LDAP THRIFT Cassandra JBoss Active Directory JMS THRIFT SOAPDistributed 3rd Party Web Service 3rd Party Java App
  8. 8. A “simple”login transaction
  9. 9. SLIPPERY SOAP• Media Customer• Few Million Users• History of Performance Issues in Production• Angry calls from Users• Tomcat, MySQL & Web Services
  10. 10. ARCHITECTURE
  11. 11. SOME WEB SERVICES PERFORM OTHERS DON’T
  12. 12. EXAMPLE: PAYPALIN 6 WEEKS PERFORMANCE IMPROVED BY 15%
  13. 13. Monitor externalservices (avg, max) Only Tune if you have to!
  14. 14. DEATH BY JAVA BEANS• Huge E-Commerce Customer• 24 Million Business Transaction a Day• Outage around 6pm in the evening• Lots of Alerts & Angry calls from Users• Tomcat, JBoss, MQ, MySQL, Oracle, Web Services
  15. 15. ARCHITECTURE
  16. 16. System Metrics look Good
  17. 17. JVM Metrics not so Good
  18. 18. JVM had huge GC time
  19. 19. What caused the GC?Search Transaction uses most CPU, is invoked often and has the most errors & stalls
  20. 20. Search Concurrency & Response Time Correlates with GC & CPU Spike
  21. 21. Let’s look at a Search
  22. 22. EJB Call for every Search Result
  23. 23. Every EJB Call makes 500+ SQL Calls12,000+ Queries & associated objects exhausted heap
  24. 24. GC can kill your App!What consumes CPU?Transactions, Objects, DataMinimize DB HitsTweak your Heap Size
  25. 25. DROWNING IN A POOL• EMEA Television Broadcaster• Reported slowdowns between 7pm and 10pm• Thousands of requests timing out• Users sending complaints via portal• Tomcat, Oracle, Web Services
  26. 26. ARCHITECTURE
  27. 27. NEW SERVICE LAUNCHED Traffic on service was doubling every week
  28. 28. CONNECTION POOL> 2,000 trx/min = DB Connections gone.
  29. 29. Developer Operations “I did tell “Wow, thatsOperations this interesting, letabout a month me speak to ago, but they Dev about it” didn’t listen”
  30. 30. Monitor your traffic growthConnection Pools are sacredTweak over-timeTest on one node & verifyTune SQL > 500ms
  31. 31. TUNING A CAR• Amateurs • Loud Exhaust, Induction Kits • Stupid Rear Wings, Bling Wheels • Go Faster Strips, neon lights• Pros • Service the Car (oil, filters, sparks) • Supercharger, Turbos, Cooling • Dyno Testing, ECU Remaps, Brakes, Suspension
  32. 32. Focus on what really makes a difference5% of things are responsible for 95% of performanceBaseline, measure, compare Evangelize your Success!
  33. 33. DEADLY LOCK• Huge US Retailer• $1 billion revenue• Random Outages & Slowdowns in Webstore• Revenue would be up/down• Tomcat, ATG, MQ, Oracle, Web Services
  34. 34. ARCHITECTURE
  35. 35. DEADLOCK DETECTEDSYS ADMINS WOULD JUST RESTART SERVERS
  36. 36. THREAD DUMPS
  37. 37. CACHE WASN’T THREAD SAFE
  38. 38. BUSINESS IMPACT?46,463 Checkouts in the Day Avg. $75 per Checkout 2,492 were impacted Cost of Deadlock: $186,900
  39. 39. Restarting Servers isn’t enough!Thread Dumps can be a pain in the ass. Exploit Tools. Make sure caches are thread-safe! (e.g. concurrentHashMap)
  40. 40. WHAT ABOUT NEW TECHNOLOGIES? Like Apache Cassandra?
  41. 41. IS NOSQL SLOW?If so, what and how slow?
  42. 42. SEE THE QUERY RESPONSE TIME
  43. 43. CASSANDRA IS A JVM TOO
  44. 44. JMX KPI METRICS
  45. 45. Make Sure You Monitor all Application ComponentsAll software can have bugs & Bottlenecks!
  46. 46. SO, DO YOU KNOWHOW FAST AND RELIABLE YOUR CODE IS IN PRODUCTION? <1% of Developers actually do
  47. 47. THE PROBLEM DEV QA/TEST PRODUCTIONConcurrency Data Volume Resource Concurrency Data Volume Resource Concurrency Data Volume Resource Very Fast Fast ?
  48. 48. TIPS FOR DEV & OPS• Monitor and Test your Applications in Production• Leverage Application Performance Management (APM)• Feedback Loop from Ops to Dev• Share Goals, Tools & Metrics Lessons Learned •Don’t write slow SQL Queries• Learn from Failure •Remember to use caching •Don’t rely on ORM •Assume everything will fail •Logging everything is not a good idea •Remember to do performance testing
  49. 49. WANT TO LEARN MORE? Visit AppDynamics Booth “Kiss my App” T-shirtDownload Free Monitoring Solution www.appdynamics.com/free Follow: @AppDynamics

×