Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014
•Download as PPTX, PDF•
0 likes•990 views
5 Real Life Examples on why Mobile Web and Mobile Native Apps failed and Which Metrics would have shown the problem early on. Using these metrics along your delivery chain allows you go get closer to full automated deployment pipeline but also making sure performance criteria is met
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014
Similar to Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014 (20)
More from Andreas Grabner
More from Andreas Grabner (20)
Recently uploaded
Recently uploaded (20)
Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014
- 1. Performance Quality Metrics for Mobile Web and Mobile Native http://bit.ly/atd2014challenge @grabnerandi
- 2. Why? What?
- 3. It allows us to Evolve from …
- 4. Some Examples from Real Life 300 Deployments / Year 10+ Deployments / Day 50-60 Deployments / Day Every 11.6 seconds
- 5. More details on Amazon Deploying every 11.6s 75% fewer outages since 2006 90% fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automatic rollback
- 6. The “War Room” – back then 'Houston, we have a problem‘ NASA Mission Control Center, Apollo 13, 1970
- 7. The “War Room” – NOW Facebook – December 2012
- 8. 5 Situations on WHY this happens, HOW to avoid it, Metrics to look at
- 10. Don’t assume You know the environment
- 11. 1.3 Million iOS Apps $ 10 billion iOS purchases 2013 1.4 million Android Apps $ 5 billion Google Payout within 1 year
- 12. Distance Calculation Issues 480km biking in 1 hour! Solution: Unit Test in Live App reports Geo Calc Problems Finding: Only happens on certain Android versions
- 13. Bluetooth 4.0 Issue on Android OS Issue crashing App
- 14. 3rd Party Issues Impact of bad 3rd party calls
- 15. Slow Network – Bad Latency Impact of Latency and Bandwidth
- 16. Metrics: Crashes, Exceptions, # and Status of 3rd Party Calls, Payload of Web Service Calls Dev: Build for Mobile Test: Test on Mobile and Diff. Carriers Ops: Monitor Mobile
- 18. #Push without a Plan
- 19. Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
- 20. m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections
- 21. Metrics: Load Time, # Resources (Images, …), # HTTP 3xx, 4xx, 5xx Dev: Build for Mobile Test: Test on Mobile Ops: Monitor Mobile
- 23. Architectural Decisions gone Bad
- 24. We wanted Web 2.0 and Mobile Ready!
- 27. Metrics: # Visitors # Requests / User Dev: Follow Best Practices Test: Find realistic scenarios
- 29. “Blindly” (Re)use Existing Components
- 30. Requirement: We need a report
- 31. Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s
- 32. Requirement: We need a fancy UI
- 33. Using Telerik Controls Results in 9s for Data-Binding of UI Controls #1: Slow Stored Procedure Depending on Request execution time of this SP varies between 1 and 7.5s #2: 240! Similar SQL Statements Most of these 240! Statements are not prepared and just differ in things like Column Names
- 34. Metrics: # Total SQLs # SQLs / Web Request # Same SQLs / Request Transferred Rows Dev: “Learn” Frameworks Test: With realistic Data
- 37. Load Spike resulted in Unavailability Ad on air
- 38. Alternative: “GoDaddy goes DevOps” 1h before SuperBowl KickOff 1h after Game ended
- 40. Metrics: Availability Page Size, # Objects # Hosts, # Connections DevOps: “Feature” Switches
- 42. # of Requests / User # of Log Messages # of Crashes Page Size # Objects In Cache # Objects Allocated Cache Hit Ratio # of Images # HTTP 3xx, 4xx # of SQLs Availability # SQLs per Request
- 43. A final thought …
- 45. How about this idea? Test Framework Results Architectural Data Build # Test Case Status # SQL # Excep CPU 12 0 120ms 3 1 68ms Build 17 testPurchase OK testSearch OK Build 18 testPurchase FAILED testSearch OK Build 19 testPurchase OK testSearch OK Build 20 testPurchase OK testSearch OK 12 0 120ms 3 1 68ms 12 5 60ms 3 1 68ms 75 0 230ms 3 1 68ms We identified a regresesion Problem solved Let’s look behind the scenes Exceptions probably reason for failed tests Problem fixed but now we have an architectural regression Now we have the functional and architectural confidence
- 46. How? Performance Focus in Test Automation Analyzing All Unit / Performance Tests Analyze Perf Metrics Identify Regressions
- 49. If we do all that
- 50. Want MORE of these and more details? http://blog.dynatrace.com
- 51. FREE Products & More Info • Dynatrace Free Trial – http://bit.ly/dtrial – Full End-to-End Visibility in your Java, .NET, PHP Apps – Sign up for a 30 Days (option for Lifetime) Free Trial on http://bit.ly/atd2014challenge • Our Blog: http://blog.dynatrace.com • My contact: @grabnerandi, agrabner@dynatrace.com
- 52. Bonus Bite(s)
- 54. Don’t ruin your reputation
- 57. Outdated Library causing Issues
- 60. Business Impact requires Action!
- 61. Solution: Cache to the RESCUE!!
- 62. Implementation and Rollout Implemented InMemory Cache Worked well in Load Testing
- 63. Result: Out of Memory Crashes!! Still crashes Fixed Version Deployed Problem fixed!
- 64. Metrics: Heap Size, # Objects Allocated, # Objects in Cache Cache Hit Ratio Test: With realistic Data
Editor's Notes
- Why do we need metrics? What do they allow us to do?
- When we look at the results of your Testing Framework from Build over Build we can easily spot functional regressions. In our example we see that testPurchase fails in Build 18. We notify the developer, problem gets fixed and with Build 19 we are back to functional correctness. Looking behind the scenes The problem is that Functional Testing only verifies the functionality to the caller of the tested function. Using dynaTrace we are able to analyze the internals of the tested code. We analyze metrics such as Number of Executed SQL Statements, Number of Exceptions thrown, Time spent on CPU, Memory Consumption, Number of Remoting Calls, Transfered Bytes, … In Build 18 we can see a nice correlation of Exceptions to the failed functional test. We can assume that one of these exceptions caused the problem. For a developer it would be very helpful to get exception information which helps to quickly identify the root cause of the problem and solve it faster. In Build 19 the Testing Framework indicates ALL GREEN. When we look behind the scenes we see that we have a big jump in SQL Statements as well as CPU Usage. What just happened? The Developer fixed the functional problem but introduced an architectural regression. This needs to be looked into – otherwise this change will have negative impact on the application once tested under load In Build 20 all these problems are fixed. We are still meeting our functional goals and are back to acceptable number of SQL Statements, Exceptions, CPU Usage, …
- Web Architectural Metrics # of JS Files, # of CSS, # of redirects Size of Images