Top Application Performance Landmines

645 views

Published on

Covering the top performance landmines that we see in live production systems and how to avoid them with proper testing

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
645
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://www.ca.com/Files/SupportingPieces/final_webstress_survey_report_229296.pdf
  • Top Application Performance Landmines

    1. 1. Top Performance Land MinesAnd how to address themAndreas Grabner, Lead of Center of Excellence @ dynaTrace
    2. 2. WHY do WE care?
    3. 3. OnlineGraduatePrograms.com “Instant America”
    4. 4. OnlineGraduatePrograms.com “Instant America”
    5. 5. WHAT do WE miss?
    6. 6. Performance Land Mines found in Production• Bloated Web Frontends • Deployment mistakes• Too many Database • Traffic estimates incorrect Statements • Poorly provisioned hardware• Too heavy logging and tracing • No consideration of end user• Synchronized to Death devices or latency• Too chatty on remoting • Misconfigured CDN channels • Load Balancing Issues• Wrong use of O/R Mappers • Networking/infrastructure• Memory Leaks issues • Problematic 3rd parties
    7. 7. Problem #1: Size
    8. 8. Watch your Weight (Content Size) Avg Bytes # of Objects Avg GPSI 400,000 50 45 350,000 40 300,000 # of Objects/Avg GPSI 35 250,000 30Bytes 200,000 25 20 150,000 15 100,000 10 50,000 5 0 0 May Jun Jul 11 Aug Sep Oct Nov Dec Jan Feb Mar Apr 11 11 11 11 11 11 11 12 12 12 12
    9. 9. Where did all of this stuff come from?(Application Complexity) Average number of hosts accessed by the browser across the Gomez US Sports Web Performance Benchmark 29
    10. 10. Where did all of this stuff come from?(Application Complexity)Average number of hosts accessed by the browser across the Gomez USSports Mobile Website Performance Benchmark 12
    11. 11. Cost of Complexity – 1 As the page gets larger, it gets slower and more complex
    12. 12. Cost of Complexity – 2 Response Time and Page Size by Hosts 16 3000000 14 2500000 12 2000000 10 As the page gets moreSeconds complex, it gets slower Bytes 8 1500000 6 and larger 1000000 4 500000 2 0 0 1 3 5 7 9 25 39 11 13 15 17 19 21 23 27 29 31 33 35 37 41 43 45 47 49 51 53 Hosts Geometric Mean BYTES
    13. 13. Tips for Testing: Analyzing Page KPIs • Analyze Key Performance Indicators (KPIs) for every tested pageTested Pages KPIs
    14. 14. Tips for Testing: Comparing Test Runs
    15. 15. Tips for Testing• Free Tooling – dynaTrace AJAX Edition for IE and FF – YSlow for FF – SpeedTracer for Chrome – ShowSlow as performance repository
    16. 16. Problem #2:Deployment Issues
    17. 17. Top Deployment Problems• Missing Files• Incorrect Access Settings• Slow Web Server Modules
    18. 18. (1) Missing Files: Impact User Experience Enter Here PopUp Here
    19. 19. (1) Missing Files: Lead to JavaScript Errors Errors caused by missing files
    20. 20. (1) Missing Files: How to identify• Watch out for HTTP 4xx
    21. 21. (1) Missing Files: Tips for Testing• Test with real browsers – HTTP-only testing might not download dynamically loaded JS files – Catch browser-specific implementations• Test from the In- and Out-side – Files might be deployed on AppServer but blocked on Web Server or Load Balancer – Certain files might come from a CDN. Testing the CDN settings is important
    22. 22. (2) Incorrect Access Settings: Impact User Experienceand Performance 14k Errors in 5 minutes after deploying in Production
    23. 23. (2) Incorrect Access Settings: Impact User Experienceand Performance Deployment causes extra roundtrips for each restricted resource
    24. 24. (2) Incorrect Access Settings: Tips for Testing• Test with real browsers – HTTP-only testing might not download dynamically loaded JS files – Catch browser-specific implementations• Test from the In- and Out-side – Files might be accessible on AppServer but blocked on Web Server – Watch out for missing JS, CSS and Image files
    25. 25. (3) Slow Web Server Modules: Performance Impact• 90% of ASP.NET Request Time spent in IIS 90%
    26. 26. (3) Slow Web Server Modules: Tips for Testing• Test with all Modules used in Production• Analyze Time spent in Modules vs. Application Code
    27. 27. Problem #3: Overhead through Logging/Tracing
    28. 28. Logging is important!!• As otherwise we are really blind – but – it comes with a cost
    29. 29. (1) Excessive use of Exceptions• Tomcat Connection Pool with logAbandoned=trueA debug flag causes many exception objects to be created
    30. 30. (1) Excessive use of Exceptions• HUNDREDS of exceptions per request
    31. 31. (1) Excessive use of Exceptions• Getting Stack Trace that many times is a performance risk
    32. 32. (2) Too granular logging• Too much logging doesn’t provide much value
    33. 33. Tips for Testing• What to watch out for during a test – Number of log entries written – Size and growth of log files – Usage of Stack Traces in Logs -> Usage of Exception Objects• Test Production Readiness by – Watch out for DEBUG, FINE, FINEST, … Log Messages -> should not be logged – Show produced log files to developers and ask whether they are useful
    34. 34. Other Problems:Excessive Database
    35. 35. Common Database Access Problem Patterns Application Connection getConnection() Application Connection locked select ... where id= 1 select ... where id= 1 closeConnection() Application Connection getConnection() Code requiring DB access Connection locked select ... where id= 100 closeConnection() Code NOT requiring DB access Database N+1 Query/Too many SQLs Unnecessary Resource Usage Application select ... from master select ... from detail select ... from detail select ... from detail Database Application select ... from master, detail where .. Database Lazy vs. Eager Loading Loading too much data
    36. 36. (1) N+1 Query Problem
    37. 37. (2) Loading too much data• Example from SharePoint for (int itemIx=0;itemIx< SPContext.Current.List.Items.Count;itemIx++) { SPListItem listItem = SPContext.Current.List.Items[itemIx]; // do something ... } Every access to Count and Items Property queries the whole SharePoint list We end up with 202 SQL Executions with a total exec time of > 1s
    38. 38. Tips for Testing• Watch out for – Slow SQL Statements – Number of SQL Statements per Request – Same SQL executed many times per request – Prepared vs. Unprepared Statements – Connection Pool Usage
    39. 39. Other Problems:Synchronization
    40. 40. Identifying Synchronization Problems Response Time BAD CPU GOOD
    41. 41. Tips for Testing• Use Performance Management Tool• Take Continuous Thread Dumps
    42. 42. Other Problems: 3rd Party Content
    43. 43. 3rd Party Content is important
    44. 44. Testing 3rd Party Content• Watch out for Size and Number of Resources # of Domains # of Resources Total Bytes DNS [ms] Connect [ms] With Third 26 176 2856 Kb 1286,82 1176,09 Party Content Without Third 2 59 897 Kb 0,91 22,25 Party Content• Remember: Watch this over time
    45. 45. Testing 3rd Party Content Performance Impact
    46. 46. Testing 3rd Party Content Performance Impact
    47. 47. Resources• Free Online Java Enterprise Performance Book – http://book.dynatrace.com• Our Performance Blog – http://blog.dynatrace.com• Contact – andreas.grabner@compuware.com – @grabnerandi
    48. 48. THANK YOU

    ×