Crowdtesting combines 'in-the-lab' and 'in-the-wild' approaches to app testing. It augments traditional QA efforts by testing apps across real devices, locations, and use cases that better represent how users experience apps. This helps close the gap between internal definitions of quality and users' actual experiences. Crowdtesting provides functional testing, usability testing, localization testing, security testing, and load testing through professional, global networks of testers. Tools like the 360-degree app quality dashboard help analyze crowdtesting results alongside other user feedback.

1. |
INTRO TO CROWDTESTING
C O M B I N I N G ‘ I N - T H E - L A B ’ & ‘ I N - T H E - W I L D ’ A P P R O A C H E S
Peter Shih | VP of Channel Development | pshih@applause.com

2. |
• Formerly known as uTest, the leader of in-the-wild software testing
– Founded by pair of Israeli entrepreneurs
– Founded in 2007; GA Launch in 2008
• Offices around the globe
– Boston, San Francisco, NYC, Seattle, Tel Aviv, Berlin & Warsaw
• World’s largest & most mature crowdtesting company
– Raised 5 rounds of VC funding, totaling $80 million
- Latest round led by Goldman Sachs in 2014
– Larger than all other crowdtesting companies combined
2
A L I T T L E A B O U T T H E C O M P A N Y

3. |
3,000+
CUSTOMERS
3
B Y T H E N U M B E R S
70,000+
B U I L D S T E S T E D
175,000+
P R O F E S S I O N A L
T E S T E R S
500 Million+
A P P S T O R E
R E V I E W S A N A LY Z E D
OVER 1 MILLION
I N - T H E - W I L D I S S U E S R E P O RT E D

4. | 4
C O M P A N I E S T H A T U S E C R O W D T E S T I N G
Retail Travel SocialMedia

5. | 5
C O M P A N I E S T H A T U S E C R O W D T E S T I N G
Health/FitnessFinancialSoftware Telco

6. |
The Challenge
PROBLEM WITH
THE STATUS QUO
•••

7. |
Test Labs Are Here Users Are Here
Modern apps & sites require real-world testing that mirrors the many
different combinations of devices, use cases and locations of users
7
T H E P R O B L E M W I T H T H E S T A T U S Q U O

8. |
• ‘Inside-out’ approach is still valuable, but it’s merely the starting point
• 360
ₒ
app quality requires an ‘outside-in’ view, which comes in many forms
– Functional defect reports
– Crash reports
– Security vulnerabilities
– Load and performance issues
– Localization issues
– Usability feedback
– User sentiment analytics
• And across all phases of the SDLC
– Research
– Pre-launch
– Post-launch
8
D E T A I L S O F ‘ O U T S I D E - I N ’ A P P R O A C H

9. |
360
ₒ
APP QUALITY
QUALITY THAT MATCHES YOUR USERS’ EXPECTATIONS
9
A P P L A U S E E N A B L E S …

10. |
DESIGN &
DEVELOP
1
TEST &
RELEASE
2
LISTEN &
LEARN
3
ANALYZE
& PLAN
4
360º
APPQUALITY
5
EG: IBM
EG: JIRA EG: CRITTERCISM
EG: FLURRY
10
“Should I launch?”
“What should I do next?”
S O L V I N G T W O M A S S I V E C I O P R O B L E M S

11. |
• Modern apps have different recipe for success
– Infinitely more variables make up the test coverage matrix
- Devices, OSes, browsers, carriers, locations, use cases
– Traditional test playbooks & methodologies are merely starting point
• Crowdtesting augments traditional QA efforts
– True for web, but even more so for mobile and smart TVs/cars/watches
– Move beyond the QA lab, closer to where users work, live & play
– Close the loop between user POV and internal definition of quality
• 360°app quality isn’t just about testing or dev ops
– Requires modern tools and analytics that focus on users-centric quality
– Necessity due to outsized user expectations + voice in app stores
3 K E Y T A K E A W A Y S
11

12. |
The Challenge
A 360
ₒ
APPROACH
TO APP QUALITY
•••

13. |
I N - T H E - W I L D T E S T I N G S E R V I C E S
FU N C TION A L
TESTIN G
U SA B ILITY
TESTING
LOC A LIZATION
TESTIN G
LOA D
TESTIN G
SEC U R ITY
TESTIN G
Do your apps function flawlessly
in the hands of your users?
Are your apps as intuitive and
easy-to-use as you think?
Are your apps primed to thrive in
different markets & cultures?
Can your apps handle the peaks
& pressures of the real world?
Are your apps protecting your
users data & privacy?

14. |
• Real-world functional testing solutions
– Ensuring apps work well in the hands of users
• Compliments in-house testing
– Team of testers provide you with scale, coverage & expertise
• Testing on real devices, in real locations
– Across OS versions, mobile browsers, wireless carriers
– Exploratory testing & test case execution
– Test around the clock or after major releases
• Integrates with your bug tracking systems
– JIRA, Bugzilla, Mantis, Pivotal, IBM Rational, Microsoft TFS and more
• Fully-managed service to reduce your overhead
– Integrated with your test services from QualiTest
– Handle all tester communications and bug triaging
– Professional testers from any region, on any device or OS
– Integrated with your internal team, processes and systems
F U N C T I O N A L T E S T I N G A P P R O A C H
14

15. | 15
I N - T H E - W I L D F U N C T I O N A L D E F E C T
Learn About The Tester
Tiered
Approval
Chat Directly
With The Tester
Reproducible
Action Steps
U I N OT R EN D ER IN G PR OPER LY

16. |
• Real-world usability testing solutions
– Ensuring your apps are designed intuitively for intended audience
• Expert-led usability studies
– Remote unmoderated testing performed by industry experts with an
average of 12 years of expert usability experience
• Audience matching
– Survey 25+ respondents that match the demographics of your users
• Video and audio experience captured
– Watch and analyze your respondents use your app for the first time
• Comprehensive report & consulting
– Receive a complete report with findings, benchmarks and
recommendations on how to improve user experience
– Leverage your usability expert for advice on how to minimize
potential confusion & frustration for first-time users
U S A B I L I T Y T E S T I N G A P P R O A C H
16

17. |
• Real-world localization testing solutions
– Validate content and app localizations to ensure correct context
across languages and cultures
• Go beyond content
– In addition to translations, verify formatting, currency conversions, taxes,
shipping options and more
• In-market localization testers
– Understand the true implications of your localization choices by leveraging
cultural natives
• Double verification
– All issues and recommendations are verified by an additional cultural native
• Available string reporting
– Output a string document to facilitate easier updating of your apps
17
L O C A L I Z A T I O N T E S T I N G A P P R O A C H

18. |
• Real-world security testing solutions
– Protect your users privacy and data
• Expert-led and managed security testing
– A white hat security expert will work with you to develop a
comprehensive test plan
• White hat security testers
– Trusted team of security testers will attack your site through
a combination of manual penetration and automated tools,
looking for common vulnerabilities and creative new attacks
• Comprehensive report
– Receive a complete report with findings and
recommendations on how to improve your site security
• Remediation consulting
– Leverage your security expert for advice on how to fix
specific vulnerabilities and mitigate different types of risks
S E C U R I T Y T E S T I N G A P P R O A C H
18

19. |
• Real-world load testing solutions
– Understand how your app performs under real world pressures
• Expert-led and managed load testing
– A Performance Engineer will assess current infrastructure,
develop JMeter scripts, execute and analyze tests
• Scalable and flexible load scenarios
– A range of options to test your apps and sites and APIs
- Large number of concurrent users
- Varying amount of time on the app or site
- With various bandwidth constraints to mirror real-world conditions
• Comprehensive report
– Receive a complete report with findings and recommendations on
how to improve your apps’ performance
19
L O A D T E S T I N G A P P R O A C H

20. | 20
A P P Q U A L I T Y T O O L S
APPLAUSE
SD K
Discover crashes in real time
and distribute builds
A PPLA U SE
A N A LYTIC S
Listen to what your users are
saying in the app stores
3 6 0 ° A PP QU A LITY
D A SH B OA R D
Access all of your app quality
results in a single place
MOB ILE B ETA
MA N A GEMEN T
Run & manage beta programs
at a global scale

21. |
3 6 0 ° A P P Q U A L I T Y D A S H B O A R D

22. |
M O B I L E S D K ( I N C L U D E D I N F U N C T I O N A L )

23. | 23
U S E R S E N T I M E N T A N A L Y T I C S

24. | 24
U S E R S E N T I M E N T A N A L Y T I C S

25. | 25
C O M P E T I T I V E A N A L Y S I S

26. | 26
U S E R R E V I E W S E N T I M E N T

27. |
The Challenge
MATURING THE
CROWDTESTING MODEL
•••

28. |
• Anyone can join, but many do not qualify for paid projects
– ‘Test the Tester’ – initial round of validations
- Hard skill vetting: technical testing skills; device knowledge; OS expertise
- Soft skill betting: following policies and processes, communication skills
– Ongoing validations from every project:
- Tester Ratings, based on the quality of reports and impact of defects found
› Value determines how much tester gets paid for their work
› Quality affects tester’s performance ratings
› Judged by customers, project managers, and team leads
- Training and Mentoring programs to sharpen tester skills
› Crowd  Teams  Community of Experts
• Protecting customer IP
– NDA in our standard Terms of Use for every tester
– Custom NDA supported for each customer
– Proof: 70,000+ test cycles with zero breaches of IP
Q U A L I T Y Q U A L I T Y Q U A L I T Y
28

29. |
• Project Managers are highly aligned with internal Dev & QA teams
– Support your expectations, timelines, and priorities
– Team Leads in various time zones to provide around-the-clock coverage
– Recommend new ideas based on experience with many international companies
• Integration with your systems and test methodologies
– Bug Tracking Systems: JIRA, HP Quality Center, IBM Rational, Microsoft TFS,
Github, Bugzilla, Pivotal Tracker, Asana, Mantis, FogBugz, Rally, and more
– Methodologies such as continuous integration, test-driven development, agile,
waterfall, lean software development, behavior driven development, etc.
• Continuous improvement and optimization
– Assess results from every project and deliver monthly management reports
– Identify favorite testers and invite new members to optimize team performance
F U L L Y - M A N A G E D S E R V I C E
29

30. |
The Challenge
FINAL POINTS
•••

31. |
• App quality was traditionally conveyed by companies to end users
– Testing in the lab was sufficient due to smaller coverage/environments
• App quality is increasingly being overruled by end users
– Through app store ratings & reviews, social media, blogs, and more
– Across multiple dimensions: usability, security, interoperability, etc.
– And most importantly, the cost and publicity of failure is higher than ever
• Applause delivers tools & services that address both views
– ‘Inside-out’ point of view
- Collaborating with internal development & QA on strategy and methodology
– ‘Outside-in’ point of view
- Complementary ‘in-the-wild’ testing on real devices across location & OS
- Reveal important feedback from end users before launch
31
K E Y T A K E A W A Y S

32. |
The Challenge
Q&A
•••