EuroSTAR Software Testing Conference 2010 presentation on The Mobile App Quality Challenge by Doron Reuveni. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/

1. The Mobile App Quality Challenge: How To Build Quality Into Your Mobile AppsDoron Reuveni |uTest CEO & Co-Founder |doronr@utest.com

2. uTest| on-demand testing
Agenda
•The Mobile Boom
•A Strategy For Mobile App Quality
•Mobile Testing Challenges
•Crowdsourcing 101
•How Crowdsourced Testing Works
•Sneak Preview of Mobile Testing Tool
1

3. uTest| on-demand testing
MOBILE MARKET BOOM
2

4. uTest| on-demand testing
The Mobile Boom
3
•$6.2 billion to be spent in 2010 at mobile app stores
•Mobile app stores expected to exceed4.5B downloads in 2010
•Expected to surpass $21.6 billion in 2013
•US subscribers:
204K (1985)
277MM (2009)

5. uTest| on-demand testing
Why Is Mobile Booming?
4
•Mobile is always on, always accessible
•Takes the web with you
Connectivity
Publishing
Sharing
•It’s smarter than traditional web
Geo-intelligent
Contextual
•Combines the best of traditional web and phones
•Continues to innovate
Laptops smart phones net books tablets

6. uTest| on-demand testing
Mobile Meets The Enterprise
5
•Not just for kids (or consumers)
•Mobile applications in the enterprise:
Business apps: CRM, ERP and HR systems
Productivity apps: docs, spreadsheets and presentations
Collaboration apps: email, IM and publishing
•Clinical, financial and customer data being accessed
•Growing pressure for QA to ensure mobile apps work properly and are private and secure
Handsets makers & models
Operating systems
Carriers
Locations

7. uTest| on-demand testing
STRATEGY FOR MOBILE APP QUALITY
6

8. uTest| on-demand testing
First Things First
7
•Mobile app quality strategy begins with a decision

9. uTest| on-demand testing
The Big Decision
8
•Mobile Web vs. Native Apps

10. uTest| on-demand testing
Mobile Web Path
9
•Argument For
Single platform (the web)
One app to build, launch and maintain
All that’s needed for some companies
•Argument Against
Less controlled user experience
Slower user experience
No app store distribution
Lack of standards across mobile browsers

11. uTest| on-demand testing
Native Apps Path
10
•Argument For
Rich media functionality
Controlled user experience
Faster experience for users
App store distribution
•Argument Against
Multi-platform
Must build, test and maintain multiple apps
Not necessary for some companies

12. uTest| on-demand testing
Alternative Paths
11
•Wholesale Applications Community (WAC)
Consortium of dozens of mobile leaders
–Carriers: AT&T, China Mobile, Orange, Verizon and Sprint
–OEMs: LG, Samsung and Sony Ericsson
Goal: build app once and have it work on all devices, carriers & OS
Good theory and wonderful on paper… but will it work?
–J2EE all over again?
–Competing interests?
–Trying to be all things to all people?
•Hybrid apps
Native app “shell” or “wrapper”
Shared back-end with existing web app (data, functionality, etc)

13. uTest| on-demand testing
Before You Begin
12
•Regardless of which path you choose, back-end work remains
API / web services
Architecture
Database
Back-end functionality
•The important lesson: app quality begins before coding or testing

14. uTest| on-demand testing
Word To The Wise
13
•Start with your users –what do theywant?
Reach users on their terms (web; mobile web; native apps)
What are your users trying to access?
–Static content vs. rich content
–Commerce vs. functionality
•Key fact
Consumer conversion (from surf to purchase) on iPhoneor Android native apps is 3X -10X times that of mobile web sites
Why?
–User comfort with device
–Usability of the native app
–Page load speed
•Forget about making the mobile web; make the web mobile

15. uTest| on-demand testing
TOP MOBILE TESTING CHALLENGES
14

16. uTest| on-demand testing
1. Coverage
•Web Testing Matrix Is Tough
15OS & Browser
Features

17. uTest| on-demand testing
1. Coverage
•Mobile Testing Matrix Is Insane
16
OS & Browser
FeaturesHandset Makers& Models

18. uTest| on-demand testing
1. Coverage
•Mobile Testing Matrix Is Insane
17
OS & Browser
Features
Handset Makers
& ModelsWirelessCarriers

19. uTest| on-demand testing
1. Coverage
•Mobile Testing Matrix Is Insane
18OS & Browser
Features
Handset Makers
& ModelsWirelessCarriersLocation

20. uTest| on-demand testing
2. Managing Resources & Costs
•Managing peaks & valleys; unexpected twists & turns
•Enough to make QA leaders sick

21. uTest| on-demand testing
3. Time-to-Market
•Never-ending race for each new app and version
•Can exhaust the best test managers and teams

22. uTest| on-demand testing
4. Mobile App Quality
•Becoming impossible for QA leaders to assure quality

23. uTest| on-demand testing
What’s At Stake
22
•Billions of dollars and the futureof apps
•Mobile app competition hotter than ever
Media attention
User adoption
App store approvals
•Pressure to build, test & launch apps never been higher
•Apps device sales

24. uTest| on-demand testing
CROWDSOURCING DEFINED:
23
“Taking a service traditionally performed by a person or team and sourcing it to a large online group of people as an open project.”
-Jeff Howe Wiredjournalist & author of Crowdsourcing

25. uTest| on-demand testing
Crowdsourcing Across Industries
24

26. uTest| on-demand testing
Mob Mentality
BUT crowds often look (and act) like unruly mobs
25

27. uTest| on-demand testing
From Mob to Community
And mobsdon’t work in every category
Example: delivering a skilled service like software testing or development requires an orderly “community” capable of consistently producing the desired results
26

28. uTest| on-demand testing
The challenge is to turn this–
27

29. uTest| on-demand testing
Into this
28

30. uTest| on-demand testing
HOW CROWDSOURCING WORKS IN TESTING
29

31. uTest| on-demand testing
Crowdsourced Testing 101
•Traditionally, companies had two options:
Option 1: Build robust in-house staff & lab
–Cost prohibitive for many companies
Option 2: Outsource or offshore
–Loss of control, communication challenges and quality risks
•A third, alternative trend has emerged to complement:
Option 3: Crowdsourcing
–On-demand extension of in-house testing efforts
–Coverage across location, language, OS, browser, handset or carrier
30

32. uTest| on-demand testing
Where Crowdsourcing Fits
•In-House vs. Offshoring
31In-HousePros:
•More control
•More visibility
•App/biz knowledgeCons:
•Fixed costs
•Scalability
•Coverage
•Higher commitment
OffshoringPros:
•Perceived cost
•Scalability
•Lower commitmentCons:
•Adaptability
•Diversity of testers
•Transparency
•Communication
•Hidden costs
Crowdsourcing
Pros:
•Coverage
•Agility
•Creativity
•Total costCons:
•Coordination
•Learning curve
•Signal-to-noise ratio

33. uTest| on-demand testing
Where Crowdsourcing Fits
•Manual vs. Automated
32ManualPros:
•Agility
•Creativity
•Explores the edgesCons:
•Scalability
•Repetitive tasks
•Human error
•Exhaustive coverage
AutomatedPros:
•Good for mainstream UX
•Scalability
•Cost
•Speed for repetitive tasksCons:
•In-the-lab vs. In-the-wild
•Ignores the unexpected
•Limits imagination
•Hinders adaptabilityCrowdsourcing

34. uTest| on-demand testing
Where Crowdsourcing Fits
•Exploratory vs. Scripted
33ExploratoryPros:
•Agility
•Creative
•Explores the edgesCons:
•Scale
•Repetitive
•Human Error
ScriptedPros:
•Predictability
•Scalability
•ControlCons:
•In-the-lab vs. In-the-wild
•Ignores the unexpected
•Limits testers’ creativity
Crowdsourcing

35. uTest| on-demand testing
Where Crowdsourcing Fits
•On-Device vs. Off-Device
34On-DevicePros:
•Agility
•Creative
•Explores the edgesCons:
•Time-consuming
•Cost-prohibitive
•Coverage
Off-DevicePros:
•Cost
•Speed
•ConvenienceCons:
•In-the-lab vs. In-the-wild
•Limited user expertise
•Detached from real world UX
Crowdsourcing

36. uTest| on-demand testing
How Crowdsourced Testing Works
35CustomersTesting CommunityTesters paidSpecify Testing Needs
-Handset Maker, Model & OS
-Carrier, Location & Language
Customerapproves
Selected testers invited
to projectTesters report in real-time

37. uTest| on-demand testing
Crowdsourced Testing Benefits
36
•Crowdsourced testing benefits:
Higher quality apps:
–Real-world testing;
–Invite favorite testers back
Get to market faster:
–2-3 day test cycles instead of 2-3 weeks
–24x7 testing cycles
Contain costs:
–Costs a fraction of traditional outsourced/offshore testing
–No long-term contracts or fixed costs

38. uTest| on-demand testing
REAL-WORLD MOBILE EXAMPLES
37

39. uTest| on-demand testing
Mobile Firms Who Use The Crowd
38

40. uTest| on-demand testing
Case Study: 1Cast
•Seattle-based mobile company
•iPhone& Android apps for news & videos
Challenge:
–Launch leading apps in a multi-platform world --with a startup team
–Between Wi-Fi, 3G, Edge, and others, 1Cast needed more hands-on testing resources than they could afford to hire
Strategy:
–Using only in-house QA was unrealistic financially
–Unwilling to commit or relinquish control to an offshore testing firm
–Researched creative alternatives and discovered crowdsourced testing
Results:
–Global testing coverage at a fraction of the cost of alternatives
–Specified testing requirements and built a virtual testing team
–Integrated crowdsourcedtesting into product development methodology to support testing of weekly builds
39

41. uTest| on-demand testing
Case Study: Tapulous
•Palo Alto-based mobile company
•Gaming apps for iPhoneand Android
Challenge:
–Ensure functionality for Tap Tap Revenge across OS, carrier, device and location ahead of a major upcoming launch
–Testing specific features, including multi-player online mode, post-game screens, offline usage and 3G connectivity
Strategy:
–In-house team was too familiar with app; needed a fresh set of eyes
–Outsourcing firms were prohibitively expensive and excessive
–Crowdsourcing was an ideal blend of beta users, in-house and outsourcing
Results:
–Testing coverage across all of North America; usability and functionality
–Several high-priority bugs discovered beforehigh-profile launch
–Achieved comprehensive testing coverage in weeks, not months over a blend of devices, locations and users
40

42. uTest| on-demand testing
IN CONCLUSION
41

43. uTest| on-demand testing
Summary
•Market growing exponentially –including B2B & enterprise
•Testing mobile is more complex than web or desktop
•Top 4 mobile testing challenges
Coverage
Peaks & valleys
Time-to-market
App quality
•Crowdsourcing enables coverage across complex testing matrices
Geographic Locations & Languages
Handset Makers & Models
Wireless Carriers
OS & Browsers
42

44. uTest| on-demand testing
SNEAK PREVIEW: REMOTE MOBILE ACCESS
43

45. uTest| on-demand testing
Remote Mobile Access
•A story…
A tester –let’s say Roy from Boston –reports a critical bug on your Blackberry app
Your QA manager and developer review it, reproduce it and ultimately, the bug is fixed
The developer or tester wants to see the bug fix in the wild –ON the actual device that first encountered the bug
Now, you can “borrow” that tester’s device remotely
•Can also be used for
Reproducing bugs on the device
Running automated diagnostics scripts on the device
44

46. uTest| on-demand testing
QUESTIONS?
45
Doron Reuveni | CEO | uTest, Incw: www.utest.com e: doronr@utest.comp: 1.800.445.3914