Mobile Testing Success: Real World Strategies and Techniques

333 views

Published on

Today, consumers spend more time on mobile apps than on the web. With this increased demand and paradigm shift toward mobile devices, the role of the software tester is evolving and becoming more complex. Since mobile testing is a relatively new domain, software testers face the challenge of understanding not only what to test but how to test. Clint Sprauve focuses on real world strategies and techniques for mobile app testing including device provisioning, mobile network virtualization, multi-OS platform coverage, and hybrid app testing. Learn how companies across various industries—insurance, finance, and entertainment—are implementing successful mobile testing strategies and techniques to meet this growing challenge. In addition, Clint highlights what is most important when creating a mobile testing strategy for your organization—object recognition options (native, text, and image), mobile app performance, and device security.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
333
On SlideShare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mobile Testing Success: Real World Strategies and Techniques

  1. 1. T10 Concurrent Class 10/3/2013 11:15:00 AM "Mobile Testing Success: Real World Strategies and Techniques" Presented by: Clinton Sprauve HP Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. Clint Sprauve Hewlett-Packard Clint Sprauve is a senior solutions architect for HP Software. Previously, Clint was the director of product strategy for the Borland Solutions Division of Micro Focus, where he created Borland’s mobile strategy for functional test automation. He has more than twenty years of experience in the software development and quality assurance industry. Clint has served as the senior product marketing manager for the Silk Testing Solutions at Borland Software and Segue Software, and served as a senior technical sales engineer for both companies.
  3. 3. 9/19/2013 Mobile Testing Success Real World Strategies & Techniques Clint Sprauve Solutions Architect & Software Innovation Coach © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. What You’ll Learn Today Current State of the Mobile Industry Mobile Opportunities and Challenges Mobile Performance Mobile Success – Case Studies © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 1
  4. 4. 9/19/2013 Android – Device Fragmentation According to Google’s own numbers, only 37.9 percent of Android devices are running Jelly Bean, the latest version of Android. That number is only slightly higher than that of Android 2.3 Gingerbread, which still commands a 34.1 percent share in spite of being released in 2010. source: http://venturebeat.com/2013/07/30/android-fragmention-2013/ 3 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Android – Device Fragmentation “The availability of cheap Android phones (rarely running the most recent version) means that [Android phones] have a much greater global reach than iOS, so app developers have a wider audience to build for…” source: http://venturebeat.com/2013/07/30/android-fragmention-2013/ 4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2
  5. 5. 9/19/2013 Why Fragment the OS? • At it’s core, it’s about choice • Smartphone manufacturers can sell a variety of phones • Different configurations • Different prices • Different consumer market segments source: http://venturebeat.com/2013/07/30/android-fragmention-2013/ 5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Android vs. iOS 6 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3
  6. 6. 9/19/2013 Mobile creates business disruptions Location-aware, frictionlesspayment taxis & private cars Accelerated check deposit processing from image Check in, change and monitor flights 7 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Business challenges The competition is going mobile Accelerated time to market Improving customer interaction Transforming and streamlining processes New sales channels 8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4
  7. 7. 9/19/2013 IT challenges Chaos and change Mobile drives IT to face a new reality New/unknown technology Accelerate delivery Complexity Performance Risk 9 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Mobile success and failure is very visible Very poor app ranking in Google play Competitors have 4 star apps Where is your app? April 2012 10 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5
  8. 8. 9/19/2013 Could better mobile testing have prevented this? A very public launch With very public quality issues led to … public apologies and executive turn over 11 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Why is mobile testing so difficult? 1. User expectations are exceedingly high… it must work right the first time - Incredibly competitive market 2. Real devices are critical - Test results can be dependent on device, OS, and network provider 3. Speed and velocity is essential - Pace of change in mobile platforms and devices is incredibly fast Mobile development projects must be agile in fast and frequent sprints 4. Mobile is not an isolated effort. - 12 Must be integrated in normal deliver processes Full lifecycle approach leads to quality outcomes © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6
  9. 9. 9/19/2013 Key considerations for mobile testing success Five dimensions of mobile testing Reality Real smartphones Automation Device agnostic Real Tablets Real mobile service 13 Flexible and robust Test device management Cloud based service Shared, remote access Secure, centralized ALM integration Enterprise visibility End to end lifecycle Development Functional Short learning curve Performance Monitoring © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Is this your mobile test lab? Real devices, but how will it scale? • • • • • • • 14 Automation Security Location Change mgt Governance Risk exposure Cost © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7
  10. 10. 9/19/2013 Mobile Performance Why mobile performance has become a competitive differentiator Mobile applications are everywhere Impacting: • Front and back-end economics • Loss of revenue • Loss of customer • Available anytime, anywhere • Significant business opportunity • Additional sales channel • Analyst influence 15 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. What happens when customers “struggle” on a mobile device? (TeaLeaf Industries, 2011) You 43% Abandon the transaction, try later on computer 16% Become more likely to prefer a competitor 14% Log a complaint with customer service 12% Go directly to a competitor Your Competitor 16 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
  11. 11. 9/19/2013 Network Virtualization – Reduce Mobile Risk By not considering actual network conditions, it can be a trap! Without real network conditions Real network conditions Capacity > 500+ Users VS Capacity < 200 Users Performance test results in a false sense of security. With real network conditions; Test results accurate and reliable. 17 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. ROI: Value of Network Virtualization HP/Shunra customer survey: •Network Virtualization eliminated >15 performance related production incidents per year • ROI < 90 Days # of Performance Related Production Incidents Per Month Relative Costs to Fix a Problem (very expensive to fix in production) 6 5 5 4 3.7 3 2 1 0 (Steve McConnell’s “Code Complete”, pp. 960 - Microsoft Press) Results Before vs. After Network Virtualization *Source: 2011 Shunra for HP Customer Survey 18 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9
  12. 12. 9/19/2013 Functional Testing Object Recognition Types © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Object Recognition Types Native Recognition - recognize objects by their native id and work with the actual components within the application (i.e., text box, text field, button) Image Recognition - captures what is seen on the screen of the device. It captures the pictures and images and uses complex algorithms to recognize objects while executing the test Optical Character Recognition (OCR) - allows the tester to extract the text from screen (also known as “screen scrapping”) 20 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10
  13. 13. 9/19/2013 Native Object Recognition Advantages Flexible and Robust Easier to maintain 21 Adjust to UI Changes quicker Faster test execution Disadvantages Difficult with customer objects Hard to keep up with OS Updates More coding knowledge required Multi-OS Updates hinder automation activities © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Image Recognition Advantages OS Independent Easier to build 22 Portable Works with what you see on the screen Disadvantages Slow Execution Difficult to maintain Increased test maintenance Reuse is extremely difficult © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 11
  14. 14. 9/19/2013 Optical Character Recognition (OCR) Advantages OS Independent Portable Easier to build Works with what you see on the screen Disadvantages Slow Execution Difficult to maintain Increased test maintenance Reuse is extremely difficult Not always dealing with text 23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Case Study Large US Airline Challenges: Struggled with automation of mobile testing, leveraging existing QA processes, tools & knowhow Engagement • Started with a private cloud & function test automation • Added complementary public cloud - device coverage. • Performance plans Currently: Extremely fast ramp up, Running regression, functional, automation, offshore testing team. 24 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12
  15. 15. 9/19/2013 Case Study Large US Financial Services Firm Challenges: Handset procurement and selection, regression testing around new versions and new devices, utilizing offshore resources for domestic US networks. Engagement • Started with private cloud and functional test based automation • Added complementary public cloud - device coverage. • Extended and doubled the capacity recently • Performance testing plans Currently: Running regression, functional automation, hundreds of scripts, constant execution, dedicated team. 25 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Summary • Take time to understand your company’s mobile strategy • Review your team’s skill set • Performance is as important as design • Seek help from mobile vendors • Mobile Testing is complex, but you can be successful © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13
  16. 16. 9/19/2013 Additional Resources Resource URL Mobile Testing @ HP .com www.hp.com/go/mobiletesting Blog and Community www.hp.com/go/thefutureoftesting HP Mobile Testing 27 HP Unified Functional Testing HP Performance Center HP LoadRunner © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Thank you © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14

×