Mobile App Testing: Moving Outside the Lab


Published on

No matter how thorough the test team or how expansive the test lab, Chris Munroe knows that defects still abound in mobile apps after launch. With more “non-software” companies launching mobile apps every day, testers have increased pressure to ensure apps are secure and function as intended. In retail and media especially, audiences are incredibly diverse and expect apps to work every time, everywhere, and on every device. These expectations make it imperative for companies to take every possible step to make their mobile apps defect free. This is increasingly difficult to do when all your testing occurs within the confines of the lab—and your users live in the wild. Using real-world examples from USA Today, Chris identifies why you need to test your mobile apps both inside and outside the lab—and do so in a way that is secure, effective, and timely.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Why is it so hot?
  • We passed an important inflection point in 2011… more smartphones sold than PCs + laptops worldwide. But while it’s easy to think of mobile as peaking, we’re just entering this era. And by 2015, tablets will also outsell computers.In short, the future (and increasingly, the present) of software applications is mobile.
  • Well, we’re wading into the shallow end first… no big surprise here: mobile’s big.And it’s coming for you, no matter what company or industry you’re in. The only remaining question is, are you ready to thrive or will you be lucky to survive?
  • Let’s look at it in a different, and even more striking way… the G20 (where your firm makes its money), mobile devices outnumber fixed internet devices by 50% in 2010. And in 2015, that becomes nearly 500%.
  • As I mentioned consumerization is a driving force behind this current shift and by looking at the consumer market, we know it is impossible to resist. Whether it is at home or work, it is human nature to want greater social interaction, more control over our surroundings and more actionable information.Speaking of actionable information, Just last week I met with VisaQuate and they have a great business doing just that. They take BI data, which was already transformed and transform it even further into even more actionable information. I love it, at some point in the future, my dashboard at work will be so simple and yet precise that I can predict the next year with a 5 second glance. For now, I only wish I could get better information on one of a million mobile devices in the world.
  • Mobile App Testing: Moving Outside the Lab

    1. 1. The necessity of reaching beyond the lab in mobile app testing StarCanada | 2013 Matt Johnston | CMO, uTest | |
    2. 2. The Challenge Agenda | 2
    3. 3. Agenda • The Mobile App Landscape • The Limits of the Lab • Real-World Examples – USA Today • What you can do • Q&A | 3
    4. 4. The Challenge Mobile App Landscape Changing Rapidly | 4
    5. 5. Mobile Apps Boom Market • ABI: mobile revenues will hit $133 billion in 2014 • B2B mobility no longer just for email – Business apps: CRM, ERP, HR systems – Productivity apps: docs, spreadsheets, presentations – Collaboration apps: email, IM, publishing • B2C mobility growth even steeper – – – – – – – Retail: location-intelligent m-commerce Media: magazines & newspapers going purely digital Travel: mobile bookings, check-ins, maps, deals Education: tablets in every classroom Healthcare: patient records, physician notes Social: no explanation required Gaming: ditto | 5
    6. 6. Mobile’s Just Getting Started | 6
    7. 7. Changing Mobile App Landscape • No company or industry is immune – – – – – – Retail: geo-aware m-commerce Media: going purely digital Travel: bookings, check-ins, maps Education: tablets in classroom Healthcare: patient records, notes Social: no explanation required | 7
    8. 8. The Future Is Here Source: Business Insider, 2012 | 8
    9. 9. To Make Matters Worse • Users less tolerant of spotty quality • Social & app stores give every user a megaphone • Switching costs lower than ever • Opportunity cost of imperfect app quality has spiked | 9
    10. 10. More Than Just Mobile • Fragmentation of users, devices, OSes, browsers, locations E-Readers Cars Household Appliances Phones Consoles & Tablets Gaming Laptops & Desktops Connected TVs | 10
    11. 11. So What Could Possibly Go Wrong? | 11
    12. 12. The Challenge The Limits of the Lab | 12
    13. 13. In-The-Lab In-House Manual Testing Outsourced Manual Testing In-The-Lab Testing In-House Outsourced Test Automation Test Automation | 13
    14. 14. Understanding the Landscape I n The Wi l |
    15. 15. Needing a New Approach The Test Lab Is Here Users Are Here – Testing only inside the lab is no longer sufficient – Mobile apps require more test coverage and a new mindset |
    16. 16. The Challenge Example USA Today | 16
    17. 17. At the Forefront of Innovation • Timeline of USA Today Pioneering – 1982: 1st national newspaper to introduce color – 1996: Early adopter of active web presence – 2008: Among 1st major media brand with native apps - iOS, Android, Windows Phone, Kindle, Nook, Connected TV, gaming – 2012: Redesigned & re-launched entirely new UI/UX across all - Printed paper + website + 14 native apps |
    18. 18. Impact on Testing • Great for USA Today’s readers • Great for USA Today’s sales & marketing • Vast pressure on IT leaders to ensure consistent quality across: – – – – – – Device makers & models Browsers OSes Carriers Languages (localized content) Locations (geo-based intelligence) |
    19. 19. The Goal • Maintain Quality – Launch nothing less than a 4.5 star app – With every new version, across 14 native apps. • Increase Test Coverage – Real-world conditions – Cost-effective – Scalable • Meet Tight Deadlines – Must have apps ready - For each new platform that gains traction - For each new device |
    20. 20. Two Distinct Choices • Traditional Outsourcing – Hidden costs – Little connection to user base – High noise-to-signal ratio • Crowd- and Expert-sourcing – Lives outside the lab, in the wild - Professional testers - Real devices - Real-world conditions – Mirrors USA Today’s user base - Technically - Geographically - Demographically – Profiled community of rated testers – Increasingly popular complement to in-the-lab testing |
    21. 21. In-the-Wild Testing • uTest’s In-the-wild testing solution for USA Today – Provide on-demand complement to in-house QA/QE team – Enable USA Today to tap into 60,000+ testers from 190 countries – Extend USA Today’s testing beyond the lab to reduce escape rate - Across OS, browsers, devices & carriers - Across locations, languages, industries & hobbies – Dedicated uTest project manager - Test team management - Triaging services – Team of dedicated remote testers + large team of on-demand flex testers |
    22. 22. The Approach • Testing Scope – USA Today specified needs via detailed test cases – Worked with uTest to coordinate tester selection, which varied by product and project, and access to uTest platform to consume results – uTest would also conduct exploratory testing, tapping into tester creativity – Testing was done for iPhone, iPad, Android & Kindle Fire |
    23. 23. Mobile Test Objectives • Tester Location – USA Today needed testers from all over the globe, including Greece, Singapore, India, Canada, Spain, Israel, UK, US and other locations • Test Team Size – Some test cycles with broader objectives, such as the new iPad application, involved 60+ testers. Others, where the scope was much tighter, required smaller teams of 10-12 testers • Device Coverage – Where fragmentation is an issue (eg: Android) USA Today worked to ensure testing extended to a pre-set list of the most popular devices and operating systems |
    24. 24. Mobile Test Results • Hit the Ground Running – USA Today needed to launch the new app for the Kindle Fire the same day the device launched - Able to test the app with testers all over the world, almost instantly • Flexibility – Test team size could be tailored to maximize value and minimize costs – USA Today’s team could see test results in real time to view latest bug reports, user reviews and test case results • Quality – Usually a lack of mobile testing talent – Real-world testing met their needs – USA Today able to consistently meet 4.5 star app-store rating goal |
    25. 25. The Challenge A Call To Action For QA Leaders (Three, Actually) | 25
    26. 26. 1. Keep On Keeping On • Keep testing in-the lab – – – – Keep training Keep automating Keep challenging Keep innovating | 26
    27. 27. 2. Look beyond the Lab • Thousands of companies have moved a portion of testing closer to where users live. Why? 1. Apps consumed under endless range of environments - Exponential spike in devices, OSes & browsers - Imperfect connectivity - Infinite number of locations 2. Users have less patience than ever for buggy apps - Switching costs lower - App quality issues carry higher costs 3. In-the-lab testing alone cannot meet the demands of modern apps | 27
    28. 28. And Lead The Expedition • In-house QA leadership is vital – You have the business and product context – You know your risk profile – You know what your firm cares about | 28
    29. 29. 3. Be The Voice For QA • Mobile Challenges will be “met” – with or without QA’s buy-in • Want QA to be heard? Want quality to be built in? – – – – – Don’t wait: No one else is coming to the rescue Get informed: Develop an thoughtful opinion Get involved: Find out when/where decisions are being made Be proactive: Build a plan for quality from the ground up Be protective: You’re a protector of the user… of the brand 29 |
    30. 30. The Challenge Questions? Answers. | 30
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.