Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Performance Metrics for your Build Pipeline - presented at Vienna WebPerf October Meetup

1,298 views

Published on

Software Performance Metrics that you should look at throughout your Build Pipeline and not just when your app crashes in productiong.

Find performance and scalability problems as soon as executing your first Unit Test. Simply focus on metrics such as #SQLs, #LogMessages, #Objects on Heap, ...

Published in: Software

Performance Metrics for your Build Pipeline - presented at Vienna WebPerf October Meetup

  1. 1. 1 Checking Performance Along your Build Pipeline @grabnerandi http://apmblog.compuware.com http://bit.ly/dttrial
  2. 2. 2 Commit Stage • Compile • Execute Unit Test • Code Analysis • Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing • Key showcases • Exploratory testing Release
  3. 3. 3 Who is doing it? How many successful deployments can they do? 300 Deployments / Year 10+ Deployments / Day 50-60 Deployments / Day Every 11.6 seconds
  4. 4. 4 More on Amazons Story Deploying every 11.6s 75% fewer outages since 2006 90% fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automatic rollback
  5. 5. Recommended Book https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/ 5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
  6. 6. Status Quo: Unreliable Software Impacts Business 6 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
  7. 7. The “War Room” – back then 'Houston, we have a problem‘ NASA Mission Control Center, Apollo 13, 1970 7
  8. 8. The “War Room” – NOW Facebook – December 2012 8
  9. 9. Status Quo: Bugfixing in Production happens YES we know this 80%Dev Time in Bug Fixing $60B Defect Costs BUT ~80% of problems caused by ~20% patterns 9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
  10. 10. 10 4 Situations on WHY this happens, HOW to avoid it
  11. 11. 11
  12. 12. “Blindly” (Re)use 12 Existing Components
  13. 13. 13 Requirement: We need a report
  14. 14. 14 Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s
  15. 15. 15 Requirement: We need a fancy UI
  16. 16. 16 Using Telerik Controls Results in 9s for Data-Binding of UI Controls #1: Slow Stored Procedure Depending on Request execution time of this SP varies between 1 and 7.5s #2: 240! Similar SQL Statements Most of these 240! Statements are not prepared and just differ in things like Column Names
  17. 17. 17 Metrics: # Total SQLs # SQLs / Web Request # Same SQLs / Request Transferred Rows Dev: “Learn” Frameworks Test: With realistic Data
  18. 18. 18
  19. 19. Implementation 19 Flaws
  20. 20. 20 Business Impact requires Action!
  21. 21. 21 Solution: Cache to the RESCUE!!
  22. 22. 22 Implementation and Rollout Implemented InMemory Cache Worked well in Load Testing
  23. 23. 23 Result: Out of Memory Crashes!! Still crashes Fixed Version Deployed Problem fixed!
  24. 24. 24 Metrics: Heap Size, # Objects Allocated, # Objects in Cache Cache Hit Ratio Test: With realistic Data
  25. 25. 25
  26. 26. 26 #No “Agile” Deployment
  27. 27. 27 Load Spike resulted in Unavailability Ad on air
  28. 28. 28 Alternative: “GoDaddy goes DevOps” 1h before SuperBowl KickOff 1h after Game ended
  29. 29. 29 Behind the Scenes
  30. 30. 30 Metrics: Availability Page Size, # Objects # Hosts, # Connections DevOps: “Feature” Switches
  31. 31. 31
  32. 32. 32 #Push without a Plan
  33. 33. 33 Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
  34. 34. 34 m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections
  35. 35. 35 Metrics: Load Time, # Resources (Images, …), # HTTP 3xx, 4xx, 5xx Dev: Build for Mobile Test: Test on Mobile Ops: Monitor Mobile
  36. 36. 36
  37. 37. # of Requests / User # of Log Messages # of Exceptions 37 Page Size # Objects In Cache # Objects Allocated Cache Hit Ratio # of Images # of SQLs Availability # SQLs per Request # HTTP 3xx, 4xx
  38. 38. 38 Commit Stage • Compile • Execute Unit Test • Code Analysis • Build installers Automated Acceptance Testing Performance Tests Automated Capacity Testing Manual testing • Key showcases • Exploratory testing Release Unit & Integration Tests Functional Tests Production Monitoring
  39. 39. How? Performance Focus in Test Automation Analyzing All Unit / Performance Tests Analyze Perf Metrics Identify Regressions
  40. 40. How? Performance Focus in Test Automation
  41. 41. How? Performance Focus in Test Automation Identify Regressions Cross Impact of KPIs
  42. 42. 4242
  43. 43. 43
  44. 44. 44 If we do all that
  45. 45. Want MORE of these and more details? http://apmblog.compuware.com 45
  46. 46. 46 FREE Products & More Info •dynaTrace Free Trial – 30 Days Free Trial on http://bit.ly/dttrial – After that totally free for local usage •Our Blog: http://apmblog.compuware.com •Follow Me: @grabnerandi

×