Best Practices for Performance Testing Mobile Apps

1,384 views
1,250 views

Published on

With an ever increasing amount of traditional, web-based content being accessed from mobile devices performance testing of mobile apps has become critical. According to the 2012-2013 World Quality Report, 64% of firms identified performance as a focus for mobile testing over both functional testing and security. It’s no wonder why firms are placing this degree of focus on mobile performance. Consumers expect their mobile content to be delivered as fast as it would be through a desktop / high bandwidth connection and they won’t hesitate to find other options when it’s not. On the surface, performance testing of mobile apps may appear to be no different from web apps. However, applying the same testing techniques used for traditional web-based systems may produce inaccurate results, or fail to expose performance issues. Mobile apps present unique differences and challenges that must be addressed to ensure performance tests accurately represent production user behavior and load profiles. This session will discuss how to address unique mobile app challenges such as device / OS diversity, mobile user behavior and network variability. Attendees will walk away with an understanding of best practices for ensuring the performance of their business critical mobile applications.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,384
On SlideShare
0
From Embeds
0
Number of Embeds
283
Actions
Shares
0
Downloads
48
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Best Practices for Performance Testing Mobile Apps

  1. 1. © Utopia Solutions Best Practices for Performance Testing Mobile Apps Lee Barnes, CTO Utopia Solutions
  2. 2. © Utopia Solutions Apology I don’t believe in “best practices”… I do believe in guidelines and better approaches for a given situation
  3. 3. © Utopia Solutions3 Key Questions How is mobile different? How can I adapt? Why is performance of mobile important?
  4. 4. © Utopia Solutions4 Why Should I Care? M-Commerce 66% OF SHOPPERS ABANDON TRANSACTIONS 5 OUT OF 10 BECAUSE OF POOR PERFORMANCE 1DELAY SECOND 7%DROP IN CONVERSIONS equals a 2013 Mobile Commerce Insights study conducted by Jumio
  5. 5. © Utopia Solutions5 Still Not Convinced? 2012-2013 World Quality Report
  6. 6. © Utopia Solutions6 Evolution of Performance Testing 1990 2000 2010 2-Tier Client Server Web Mobile Challenges State of the Practice Testing Large Loads Complex / Integrated Systems Proprietary Protocols Skilled Resources In-Lab Testing Tool Maturation Open Source Tools Specialized Resources Cloud-based Testing
  7. 7. © Utopia Solutions Mobile Challenges & Adaptations
  8. 8. © Utopia Solutions8 Mobile Challenges – User Access Native App Mobile Site Full / Responsive Site Your users have options…
  9. 9. © Utopia Solutions9 Assess the User and Load Profile 1. Determine the user profile • Business processes • Native app, mobile site, full site • Preferred browser • Network conditions • Geographic location 2. Determine the load profile • Volume of users by business process and location • Volume of users by access type • Frequency / throughput Incorporate constant feedback from application logs and other monitoring solutions
  10. 10. © Utopia Solutions10 Mobile Challenges - Network Mobile User Connected User 3G 4G InATunnelG Network bandwidth and quality affect both end user experience and system load
  11. 11. © Utopia Solutions11 Assess Impact of Network Variability 1. Assess baseline performance (single user) Observe UX with real devices behind network virtualization solutions 2. Assess performance under load Put system under virtual user load Observe UX with real devices behind network virtualization solutions NV Virtual Users NV Real Devices
  12. 12. © Utopia Solutions12 Network Virtualization Tools Single User (Device and/or Emulator) • Apple Network Link Conditioner • Android Emulator • Charles Proxy • Shunra NV Multiple Virtual Users • Shunra NV • Load test tools (built-in) • Charles Proxy
  13. 13. © Utopia Solutions13 Mobile Challenges – User Location and Volume A high volume of global users is difficult to reproduce in a lab…
  14. 14. © Utopia Solutions14 Utilize Cloud-based Testing Solutions • Achieve production level loads • Test entire infrastructure (vs. behind the firewall) • Distribute load geographically • Assess user impact on real devices
  15. 15. © Utopia Solutions15 Mobile Challenges – Device Performance Measuring System / Network Metrics is Not Enough… Memory CPU Battery Drain Graphics I / O
  16. 16. © Utopia Solutions16 Measuring Device Performance Critical device metrics: • CPU • Memory • Battery • Storage Mobile development platforms • Android DDMS • iOS Instruments Mobile test automation tools • TouchTest (SOASTA) • Trust (Mobile Labs) • MonkeyTalk (Cloud Monkey) Mobile lab / cloud solutions • Device Connect (Mobile Labs) • Keynote DeviceAnywhere • Perfecto Mobile Capture metrics while exercising app Include appropriate devices / configurations Incorporate automation for consistency Device Performance Measurement Solutions
  17. 17. © Utopia Solutions17 Mobile Challenges – Agile Development Analysis Design Develop & Unit Test Acceptance Test Final Acceptance Test Deploy Source: Agile and Test Estimation Sharon Robson How does performance testing fit?
  18. 18. © Utopia Solutions18 Performance Testing in an Agile Environment Plan Design Build Test Deploy Waterfall Performance Testing Too late!
  19. 19. © Utopia Solutions19 Performance Testing in an Agile Environment Sprint 1 Sprint 2 Sprint 3 … Deploy Agile Performance Testing Same Process… Same Risk!
  20. 20. © Utopia Solutions20 Performance Testing in an Agile Environment Sprint 1 Sprint 2 Sprint 3 … Deploy Agile Performance Testing Performance Testing Performance Testing Performance Testing Waterfall Approach • Rigid • 4-6 week test cycle Agile Approach • Flexible • 2 week sprints + =
  21. 21. © Utopia Solutions21 Early Cycle / Ongoing Performance Analysis Performance Testing in an Agile Environment Ultimate Goal… Late Cycle Performance Testing Shift Left
  22. 22. © Utopia Solutions22 Performance Testing in an Agile Environment How? Map business and end user requirements onto system components Benchmark in development / QA environment Look for trends in measurements vs. absolutes Use service virtualization to stub out 3rd party or yet to be developed services Execute unit performance tests as part of continuous integration Unit / Component Performance Testing
  23. 23. © Utopia Solutions23 Performance Testing in an Agile Environment How? Measure resource / performance profile of native apps Use mobile site analysis services / tools to analyze services and pages User APM solutions to profile code Use device profiling solutions to understand / compare base device performance Application / Code Profiling
  24. 24. © Utopia Solutions24 Waterfall Agile Service Model Perform turnkey performance test Timing End of development cycle Duration Weeks Tools Traditional load test tools Environment Dedicated / production like Role of development Throw system over wall Wait for results Waterfall Agile Service Model Perform turnkey performance test Provide mentorship to agile team Timing End of development cycle Ongoing Duration Weeks Hours / days Tools Traditional load test tools Traditional load test tools APM tools Service virtualization Continuous integration Environment Dedicated / production like Development / desktop Role of development Throw system over wall Wait for results Constant focus on performance Performance Testing in an Agile Environment What about my Performance Testing Center of Excellence?
  25. 25. © Utopia Solutions25 Summary Key takeaways… • Mobile users are not the same as connected users • Performance is more than backend / network • Push performance analysis to the left in the development cycle
  26. 26. © Utopia Solutions26 Questions… …. and Answers! Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: lee.barnes@utopiasolutions.com Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com ?

×