Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

London WebPerf Meetup: End-To-End Performance Problems

6 Real Life Performance Problem Scenarios: Why They failed, How to Avoid it and the Metrics to spot these problems

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

London WebPerf Meetup: End-To-End Performance Problems

  1. 1. 1 @Dynatrace Tech Blogs on http://blog.dynatrace.com Free Tools on http://ajax.dynatrace.com Web/App Performance How to keep you out of the News Web Perf London, Feb 2nd 2015
  2. 2. 2 @Dynatrace Andreas Grabner Performance Advocate @ Dynatrace andreas.grabner@dynatrace.com @grabnerandi Darren Edwards IT Service Manager @ esure Darren.Edwards@esure.com Darren Edwards IT Service Manager @ esure Darren.Edwards@esure.com
  3. 3. 3 @Dynatrace Introduction – Darren Edwards (esure IT Service Manager) • 28 Years in IT (Financial Services) • Programmer • Tools (Dev, Test, Release/Config Mgt). • Architect • Release Management • Service Delivery Management – last 6 years • Passion for delivery of reliable and performant services to customers
  4. 4. 4 @Dynatrace Introduction - esure • The vision of renowned insurance entrepreneur, Peter Wood, esure Group plc is one of the UK’s leading providers of general insurance products. • esure has three offices in the UK, in Reigate, Surrey; Manchester and Glasgow. It employs over 1,400 staff and has over 1.5million customers. • Four key brands within the esure group:
  5. 5. 5 @Dynatrace That’s why I ended up talking about performance
  6. 6. 6 @Dynatrace
  7. 7. 7 @Dynatrace 99.9% Backend0.01% Web Server
  8. 8. 8 @Dynatrace That’s why we are here Today
  9. 9. 9 @Dynatrace Nobody wants this …
  10. 10. 10 @Dynatrace Unless you work for Google or Microsoft 
  11. 11. 11 @Dynatrace Nor this …
  12. 12. 12 @Dynatrace
  13. 13. 13 @Dynatrace As it leads to this …
  14. 14. 14 @Dynatrace The “War Room” Facebook – December 2012
  15. 15. 15 @Dynatrace And potentially to this …
  16. 16. 16 @Dynatrace
  17. 17. 17 @Dynatrace And this …
  18. 18. 18 @Dynatrace
  19. 19. 19 @Dynatrace And that’s why Business doesn’t like it either …
  20. 20. 20 @Dynatrace ~80% of problems caused by ~20% patterns YES we know this 80%Dev Time in Bug Fixing $60BDefect Costs BUT
  21. 21. 21 @Dynatrace 6 Situations on WHY this happened, HOW to avoid it
  22. 22. 22 @Dynatrace Darren Hidden Incompatibility
  23. 23. 23 @Dynatrace Hidden Incompatibility
  24. 24. 24 @Dynatrace • No obvious reasons why this should be a problem • Observed throughput through live systems and monitoring of resources looked good • Good to Go? Requirement – increased volume without issues
  25. 25. 25 @Dynatrace Performance testing identified increase in 501’s Resin Threads not being released
  26. 26. 26 @Dynatrace JSF/Resin incompatability • Increased number of long running JSF methods in long running PurePaths • Linked to backing up of Resin Threads • Traced to locking issue between JSF and Resin
  27. 27. 27 @Dynatrace • Short Term – Increased Resin instances • Longer Term – Moved across to TomCat • Peak volumes handled without issues that would have been encountered Remediation
  28. 28. 28 @Dynatrace Andreas
  29. 29. 29 @Dynatrace
  30. 30. 30 @Dynatrace #Push without a Plan
  31. 31. 31 @Dynatrace Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
  32. 32. 32 @Dynatrace m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections
  33. 33. 33 @Dynatrace Fifa.com during Worldcup http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/
  34. 34. 34 @Dynatrace # Images # Redirects Size of Resources
  35. 35. 35 @Dynatrace Darren It’s not just your systems !!!
  36. 36. 36 @Dynatrace It’s not just your systems !!!
  37. 37. 37 @Dynatrace Third Party impact Third Party response time Aggregator Quote time
  38. 38. 38 @Dynatrace Third Party impact • Long running PurePaths show timeout of 5s set for wait on Third Party exceeded. • Will continue with overall transaction with some data input but impact on time for overall service has happened.
  39. 39. 39 @Dynatrace • Need to ensure Third Party services can provide required performance levels • Need to monitor • Look to process Third Party feeds in parallel if within a key transaction flow to minimise added time It’s not just your systems !!!
  40. 40. 40 @Dynatrace Andreas
  41. 41. 41 @Dynatrace
  42. 42. 42 @Dynatrace #“Blindly” (Re)use Existing Components
  43. 43. 43 @Dynatrace Requirement: We need a report
  44. 44. 44 @Dynatrace Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s
  45. 45. 45 @Dynatrace # SQL Executions # of SAME SQLs
  46. 46. 46 @Dynatrace Darren Bad database queries
  47. 47. 47 @Dynatrace Bad database queries
  48. 48. 48 @Dynatrace Response time for key search action within a document repository system slowing down: • Avg Response • Invocations • Response - percentile
  49. 49. 49 @Dynatrace Trace to database query • Looked at slow PurePaths and identified time being taken up by single query • Turned on bind variable capture to assist with query testing and case identification
  50. 50. 50 @Dynatrace • Moved index cache into memory • Tune query Remediation
  51. 51. 51 @Dynatrace Andreas
  52. 52. 52 @Dynatrace
  53. 53. 53 @Dynatrace #No “Agile” Deployment
  54. 54. 54 @Dynatrace Load Spike resulted in Unavailability Adonair
  55. 55. 55 @Dynatrace Alternative: “GoDaddy goes DevOps” 1h before SuperBowl KickOff 1h after Game ended
  56. 56. 56 @Dynatrace # of Domains Total Size
  57. 57. 57 @Dynatrace •# Images •# Redirects •Size of Resources •# SQL Executions •# of SAME SQLs •# Items per Page •# AJAX per Page Consider these Metrics •Time Spent in API •# Calls into API •# Functional Errors •3rd Party calls •# of Domains •Total Size
  58. 58. 58 @Dynatrace Commit Stage • Compile • Execute Unit Test • Code Analysis • Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing • Key showcases • Exploratory testing Release Unit & Integration Tests Functional Tests Performance Tests Production Monitoring Functional Tests (R)Evolutionize Web Performance Optimization
  59. 59. 59 @Dynatrace
  60. 60. 60 @Dynatrace Example from Web Diagnostics 282! Objects on that page9.68MB Page Size 8.8s Page Load Time Most objects are images delivered from your main domain Very long Connect time (1.8s) to your CDN
  61. 61. 61 @Dynatrace Example from Server-Side Diagnostics 526s to render that report 1 SQL running 210s! Lots of time spent in logging to Log4J Lots of time spent in rendering
  62. 62. 62 @Dynatrace Online Performance Clinics Every Other week @ bit.ly/onlineperfclinic
  63. 63. 63 @Dynatrace Your Benefits • Free Performance Review • Extended Dynatrace License “Share Your PurePath” bit.ly/sharepurepath My Benefits • More blog material for next year  • Gratification that I could help you
  64. 64. 64 @Dynatrace Questions and/or Demo Get Tools: http://ajax.dynatrace.com Contact Me: agrabner@dynatrace.com Follow Me: @grabnerandi Read More: http://blog.dynatrace.com
  65. 65. 65 @Dynatrace Andreas Grabner Performance Advocate @ Dynatrace andreas.grabner@dynatrace.com @grabnerandi Darren Edwards IT Service Manager @ esure Darren.Edwards@esure.com Darren Edwards IT Service Manager @ esure Darren.Edwards@esure.com

    Be the first to comment

    Login to see the comments

  • PanosHapsides

    Feb. 4, 2015
  • AnthonyClarke3

    Feb. 5, 2015
  • dineshaltius

    Nov. 10, 2016

6 Real Life Performance Problem Scenarios: Why They failed, How to Avoid it and the Metrics to spot these problems

Views

Total views

1,438

On Slideshare

0

From embeds

0

Number of embeds

28

Actions

Downloads

50

Shares

0

Comments

0

Likes

3

×