This document provides an overview of a presentation on implementing crowdsourced testing. The presentation was given by Rajini Padmanaban from QA InfoTech. The agenda covers topics such as understanding crowdsourced testing, different forms of crowdsourcing, exercises to demonstrate crowdsourced testing, its relevance and limitations, practices for successful crowdsourced testing efforts, and examples. The document includes slides on defining quality, software development lifecycles, types of crowdsourcing, motivating crowds, practices, and what not to crowdsource.

1. MH
AM Tutorial
4/29/13 8:30AM
Implementing Crowdsourced Testing
Presented by:
Rajini Padmanaban
QA InfoTech
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Rajini Padmanaban
As director of engagement at QA InfoTech, Rajini Padmanaban leads the engagement and relationship
management for some of QA InfoTech's largest and most strategic accounts. Rajini has more than eleven
years of professional experience, primarily in the software quality assurance area. She actively advocates
software quality assurance through evangelistic activities; writes on test trends, technologies, and best
practices; speaks at conferences including STAREAST 2012 and STARWEST 2012; and provides
insights on software testing to analyst firms such as Gartner and IDC. Read Rajini’s official blog and reach
her at rajini.padmanaban@qainfotech.net.

3. Implementing Crowd
Sourced Testing
Rajini Padmanaban
Director of Testing Engagements
Your Software Testing Partner
We help you build better software
Agenda (1 of 2)
Topic
Time (in
minutes)
A Peek into Software Quality
10
Crowd Sourced Testing - Defined
10
Understanding Varied forms for Crowd Sourcing 15
Let’s be the Crowd – Exercise Time
20
Crowd Sourced Testing Relevance in Current
10
Scenario
Limitations of Crowd Sourced Testing
15
Slide 2
Corp04
1

4. Agenda (2 of 2)
Topic
Time (in
minutes)
Practices for a Successful Crowd Sourced Test
Effort
30
Know What Not to Crowd Source
Getting Stake Holder Buy-in
Case Study & Examples
15
15
30
Myths and Facts
Conclusion and Q&A
15
15
Slide 3
Requirements
A device to connect to the internet for one of
our exercises (laptop, smart phones, tablets )
Slide 4
Corp04
2

5. A Peek into Software Quality
Test is responsible for
Quality
Focused on creating a quality deliverable
Ensure we don’t sacrifice quality for the sake of schedule
Empower the rest of the team to partake in improving quality
Slide 5
Defining Quality
•
•
•
•
•
•
•
ISO 9000: Degree to which a set of inherent characteristics fulfill
requirements
Six Sigma: Number of defects per million opportunities
Philip B. Crosby: Conformance to requirements
Joseph M. Juran: Fitness for use by the customer
Gerald M. Weinberg: Value to some person
Robert Pirsig: The result of care
American Society for Quality: A subjective term for which each person
has his or her own definition. In technical usage, quality can have two
meanings:
a. The characteristics of a product or service that bear on its ability to
satisfy stated or implied needs
b. A product or service free of deficiencies
Slide 6
Corp04
3

6. Software Development
Slide 7
Attacking the Inverse Exponential
Testability
Fault Injection
Stress Testing
End-to-End
Exploratory
Code Inspection
Leverage the Beta Test Crowd
Slide 8
Corp04
4

7. Portfolio Selection Theory
Diversification in
investing tells us
that risk lessens as
the number of
investments in the
portfolio increases
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
Slide 9
Audience Interaction Time
What Does Quality Mean to you ?
Slide 10
Corp04
5

8. What is Crowd Sourced Testing?
Dailycrowdsource.com
Crowd sourcing is the process of getting work, usually online, from a crowd of people. The word is a
combination of the words 'crowd' and 'outsourcing'. The idea is to take work and outsource it to a crowd
of workers
CrowdSourcing.org (similar definition on Wikipedia)
Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that
exploits the benefits, effectiveness, and efficiency of crowd sourcing and the cloud platform towards
software quality assurance and control. With this new form of software testing, the product is put to test
under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all,
bug-free
I prefer the
word engaging
rather than
delegating
Crowd Source Testing.com
Crowd sourcing your software testing consists of delegating onto a number of internet users the task of
testing your web or software project while in development to ensure that it contains no defects, referred
to as bugs
Slide 11
What is Crowd Sourced Testing?
.so what really is Crowd Sourced Testing?
• Is it all about a pool of testers leveraged and paid
per valid bug? No, not just that
• Think:
Slide 12
Corp04
• Sourcing relevant people from within your company
• Across disciplines and levels
• Sourcing end users from various disciplines – e.g.
teachers, students, nurses, bankers
• Partnerships with universities, organizations leveraging
domain knowledge
• In essence, think of the community at large to test your
product
6

9. What is Crowd Sourced Testing?
Technologies
Company Scale
Domains
Internal
Crowd
Sourcing
Services
Companies
Product
Companies
External
Crowd
Sourcing
Slide 13
Understand Factors that Motivate the Crowd
Portfolio Selection Theory
10%
Crowd Sourced Testing:
Portfolio Selection Theory
applied in QA
10%
10%
10%
10%
10%
10%
10%
10%
10%
Slide 14
Corp04
7

10. A Practical Solution to Leverage
Slide 15
Set # 1
• Crowd Sourcing:
• Is not restricted to any single company, technology,
domain
• Has a much larger yet simpler meaning than what it is
often portrayed to be
Slide 16
Corp04
8

11. Types of Crowd Sourcing - Explained
Crowd Creation
Crowd Voting
Crowd Wisdom
Crowd Funding
Slide 17
Types of Crowd Sourcing - Explained
• Crowd Creation
• Invite crowd to create subject content
• Common usage areas: software
development, translations, photos repository
• Content usage by crowd or by organizations
• Typical crowd motivators – Money, fun,
community involvement, brand loyalty
• Well known examples – Linux, iStockPhoto,
99 designs
Slide 18
Corp04
9

12. Types of Crowd Sourcing - Explained
Slide 19
Types of Crowd Sourcing - Explained
• Crowd Voting
• Leverage crowd’s judgement to organize,
filter, stack rank content
Most popular of crowd
• Common usage areas: retail, media, simple
sourced versions
(1:10:89 rule)
yet powerful decisions across domains
1% create
• Results used by an organization
10% vote and rate
89% consume
• Typical crowd motivators – Fun, community
involvement, brand loyalty
• Well known examples – American Idol,
Threadless.com
-Jeff Howe, Author of CrowdSourcing
Slide 20
Corp04
10

13. Types of Crowd Sourcing - Explained
Slide 21
Types of Crowd Sourcing - Explained
• Crowd Wisdom
• Harnesses crowd’s knowledge to solve
problems, predict future outcomes
• Common usage areas: reality shows, quality
assurance, exchanges markets
• Results used by an organization, crowd
• Typical crowd motivators – Money, fun,
product transparency, brand loyalty
• Well known examples – Who wants to be a
Millionaire
Slide 22
Corp04
11

14. Types of Crowd Sourcing - Explained
Did You Know?
The crowd’s answer in
“Who Wants to be a
Millionaire” was right 91%
of the time compared to
“Ask an Expert”, which was
right 65% of the time
Source: The Wisdom of
Crowds, James Surowiecki
Slide 23
Types of Crowd Sourcing - Explained
• Crowd Funding
• Leverages crowd to finance individuals or
groups that might otherwise be denied
credit or opportunity
• Common usage areas: ideas in developing
nations, educational domain
• Results have far reaching impact
• Typical crowd motivators – Money, social
causes / community involvement
• Several interesting examples at:
http://www.alumnifutures.com/2012/07/crowdsourced.html
Slide 24
Corp04
12

15. Set # 2
• Crowd Sourcing Options are:
• plentiful
• available for a diverse set of community needs – both
commercial and not-for-profit
• to be customized based on results needed and the
crowd motivating factors
Slide 25
Exercise Time – Let’s Crowd Source
Walt Disney – he won 26
oscars including 4 in a
same year
Who has won the most oscars?
1.Walt Disney
2.Elizabeth Taylor
3.Meryl Streep
4.Jack Nicholson
Slide 26
Corp04
13

16. Exercise Time – Let’s Crowd Source
Mexico derived from an Aztec
legend that a city would be built
where they spot an eagle eating a
serpent, which is now Mexico City
On which national flag is there an eagle and
a snake?
1.China
2.Mexico
3.Greece
4.Spain
Slide 27
Exercise Time – Let’s Crowd Source
Idaho, followed by Washington
and Oregon (in 2011)
Source:
http://www.potatopro.com/Newsletters/20111110.htm
Which American state produces most
potatoes?
1. Oregon
2. California
3.Idaho
4.Washington
Slide 28
Corp04
14

17. Set # 3
Did You Know?
Collective wisdom
of the crowd often
surpasses that of
an expert
The crowd’s answer in
“Who Wants to be a
Millionaire” was right 91%
of the time compared to
“Ask an Expert”, which was
right 65% of the time
Source: The Wisdom of
Crowds, James Surowiecki
Exercise Time – Let’s Crowd Source
• Please take about 10 minutes to test
www.amazon.com from usability and
accessibility standpoints
• Need tools - screen readers, magnifiers?
• http://en.wikipedia.org/wiki/List_of_screen_readers
• http://mediaaccess.org.au/digital-technology/assistivetech/screen-magnifiers/ - e.g. Magnifier on Windows, Zoom
for Mac
Slide 30
Corp04
15

18. Exercise Time – Let’s Crowd Source
• Discuss test results / feedback –
• Goal of our study: demonstrating richness
and diversity of crowd’s feedback as end
users
• Feedback from our visually challenged crowd
Amazon
Accessibility Issues
Slide 31
Product Quality Scenario – as it stands today
•
•
•
•
Shrinking release cycles
Close scrutiny on overall spend
Collective ownership of quality
Need to:
• Focus on product domain knowledge; not just
disciplinary knowledge
• Understand competing products
• Creatively emulate end user scenarios
• Mimic user environments via lab and simulations
• Consider global product distribution effects
Slide 32
Corp04
16

19. A Practical Solution to Leverage
•
•
•
•
•
•
Distribute quality effort
Enhance productivity through global solutions
Pool in end users into quality implementation
Use live environments to test
Analyze potential partnerships for SME
Flexible, selective and cost effective testing
.........Bring in the Crowd
Slide 33
When does Crowd Sourced Testing
Succeed?
• Diversity of knowledge, background,
experience
• Independence in testing process
• Wide spread domain background for product
requiring SMEs
• End user scenarios and environments difficult
to simulate in-house
• Work aligns with factors that motivate crowd
Slide 34
Corp04
17

20. Inherent Challenges
• Random test efforts do not fit into quality strategy
• Choosing, sustaining a crowd sourced team
• Ongoing motivation, floating crowd
Any Challenges from
• Keeping everyone in sync on product dynamics
your experience that
• Communication challenges
you would like to share?
• Management overhead including logistics
• Stakeholder buy in
• Securing product IP before release
Slide 35
Perceived Limitation
Quality of product adversely impacted by an
amateur crowd
- Solidify your implementation plans
- Use instrumentation wherever possible
- Refer case studies – understand successes
and failures
Slide 36
Corp04
18

21. Set # 4
• Current product scenario demands creative solutions to
test within existing constraints
• Crowd Sourcing is not a no-brainer solution to all problems
• Not even a stand-alone solution in most cases
• Understand its strengths and challenges in customizing
it to your needs
• Understand “What, When and How to Crowd Source in
Testing”
Slide 37
Practices for Successful Crowd Testing
Customize your
practices mindful of
your constraints
Slide 38
Corp04
19

22. “What, When and How” of Crowd Testing
• When to Crowd Test:
• Product works reasonably well E2E
• Ready to incorporate crowd’s feedback
• No time or resources for formal testing .try to
avoid this situation
• Source content files are ready
• Ongoing feedback from a chosen SME team
at specific stages
Slide 39
Examples
Started off with informal testing due to
lack of time
Slide 40
Corp04
Ongoing MVP programs at Microsoft
Product ready to be tested E2E and
feedback incorporated
20

23. “What, When and How” of Crowd Testing
• What to Crowd Test:
• User facing features
• Areas where external team feedback is
important – e.g. design, feature set, performance
• Specialized areas of test:
• localization (context based verification)
• performance
• compatibility, devices testing
• Content testing – valuable SME knowledge
• Align crowd’s focus areas into test strategy
• Know what not to crowd source
Slide 41
Examples
Exam Grading
Content Testing using SME /
Localization
Slide 42
Corp04
21

24. “What, When and How” of Crowd Testing
• How to Crowd Test:
• Pick the right areas, team and time
• Minimize duplication, overhead of sifting through
known issues
• Clear internal ownership:
• Communication, technical query resolution
• Prompt follow up and responses
• Team up-to-date on product changes
• Use of collaborative tools
• Think about interactions amongst testers
Slide 43
“What, When and How” of Crowd Testing
• How to Crowd Test:
• Leverage cloud, VPC for ease, secured access
• Identify crowd motivators
• Use management theories – Maslow’s
hierarchy of needs?
• Identify tasks that align with crowd motivators
• Work on stakeholders buy-in
Self Actualization – Pursue Inner Talent,
Creativity, Fulfillment
Self Esteem – Achievement, Mastery,
Recognition, Respect
Belonging – Friends, Family, Spouse, Lover
Safety – Security, Stability, Freedom from Fear
Psychological – Food, Water, Shelter, Warmth
Slide 44
Corp04
22

25. Set # 5
• Understand when a crowd test effort succeeds:
• Diversity of knowledge, background, experience
• Independence Examples, Bestprocess
Any in the testing
• Wide spread domain background for product
Practices from your
requiring SMEs
experience that you
• End user scenarios and environments difficult to
would like to share?
simulate in-house
• Work aligns with factors that motivate the crowd
Slide 45
Know What Not to Crowd Test
• Features with moving pieces, demanding close
collaboration
• Sensitive IP
• Environment specific complex testing
• Tasks requiring immediate and regular turnaround
- BVTs
• Core testing activities
•
•
•
•
Test automation, TDD scripts
Regression testing
First round of performance, security, integration testing
Mundane testing tasks that don’t need diversity
Slide 46
Corp04
23

26. Some Examples
• Simple performance tests such as PLTs – use tools
• Understand pros and cons specific to your scenario –
study by University of Texas, Austin* on Crowd Sourcing for Usability Testing vs. Lab Usability Testing on a college website
Lab Usability Test
Crowd Sourced Usability Test
Participants
5
55 (14 spammers)
Participant Demographics
Students
Crowdworkers
Age
24 to 33
19 to 51
Education level
Bachelor’s degree and Master’s degree
All levels
Experience with similar
websites
Speed
Yes: 100%
Yes: 77%
No: 23%
Less than 4 hours total.
Participant
Costs
None
Approximately 30 min. per session.
$2.92 for pilot test
$23.41 for final test
(Avg: $0.48/tester)
* - http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf
Slide 47
Some Examples
Major Problems Identified
Lab Usability Test
Crowd Sourcing Usability Test
Font size too small
Out-of-date information
Menu overlap
Irrelevant picture
Invisible tools
Information not cross-linked
Lack of sort function
Navigation unclear
Search box difficult to locate
Slide 48
Corp04
24

27. Some Examples
Advantages
Disadvantages
More participants
Lower Quality Feedback
High Speed
Less Interaction
Low Cost
Spammers
Various Backgrounds
Less Focused User Groups
In Conclusion:
1. In this scenario, usability testing would be better off done by students of the college website;
choosing target users is very important
2. If crowd sourced testing is relevant based on your user profile, design the effort with care; using
same tests and questions as that of lab testers for crowd testers may not yield great results (see
section 4.1.2 on Test Redesign)
Slide 49
Stake holder Buy In
• Stakeholder resistance to Crowd Sourcing
largely inline with model’s challenges:
• Product IP, privacy issues
• Internal team motivation
• Quality of test effort and product
• Additional overhead in effort management
• Randomized test effort – tactical in nature
Slide 50
Corp04
25

28. Stake holder Buy In
1. Show steady progress
2. Stick to pre-defined
communication protocols
3. Communicate the good and the
bad
4. Be on top of new stakeholders
or new concerns
Identify stakeholders;
prioritize the team;
engage early
1. Supplemental test technique
2. Acknowledge problems
3. Explain the technique as
applicable to your product; don’t
assume their know-how
Walk through the test
strategy including crowd
sourcing plans;
understand their
concerns
Ongoing
Communication
1. Practical demonstration of the
model; adds to your confidence too!
2. Additional overhead at start but
pays off in longer run
3. Run a pilot like a regular CS test
program but of smaller scale
Undertake Pilot if
needed
1. Educated decision
2. Explain implementation plans –
what & what not, when, how,
3. Define checks and balances for
internal team and CS team roles
4. Map solutions to address each
concern identified earlier
Explain Solutions
Slide 51
Set # 6
• Acknowledging what not to crowd source will:
• Fasten stakeholder approval process
• Help not impact internal team motivation
Slide 52
Corp04
26

29. Successful Work Patterns
Core
Unique
Future
Work Skills Work Skills Work Skills
In-Job
Behaviors
Organizational
Citizenship
Behaviors
Slide 53
Successful Work Patterns
Core
Work
Skills
Unique
Work
Skills
Future
Work
Skills
In-Job
Behaviors
Organizational
Citizenship
Behaviors
Slide 54
Corp04
27

30. Localization Testing
• Hard, Large-Scale
Problem
• Windows 7 ships in 100
languages
• Thousands of strings
and screens per release
• Traditional model of
localization testers
expensive and difficult to
find
Slide 55
Let’s Play
Slide 56
Corp04
link
28

31. The Language Quality Game
Slide 57
Results
Total Screens Reviewed: Over 500,000
Total Number of
Over 4,500
Reviewers:
Screens per Reviewer: Average 119
Significant Quality Improvements for Windows 7
Positive Impact on Ship Schedule
Team Morale and Subsidiary Engagement
Internal sourcing alleviates security and access issues
Slide 58
Corp04
29

32. Amplify Skill with Volume
Individual dialects,
nuances, hard to
detect with a single
vendor – crowd does
a better job
Slide 59
Reduce Cost with Discovery,
Instrumentation
No need to install
Telemetry to direct
effort
Slide 60
Corp04
30

33. Reduce Risk with Diversity
Portfolio theory in
Quality Assurance
Slide 61
Trust and Transparency Increase
Effectiveness
Inclusion = Trust, Trust =
Enthusiasm
Subsidiary
engagement with
Language Quality
Game
Momentum for Win 7
Slide 62
Corp04
31

34. Players earn points
for disaster relief
agencies
Microsoft donates $
based on leader
board
Individual players
can sponsor tasks
or scenarios
Slide 63
Results
Players Over 1,000
Feedback increase: > 16x
Feedback received: 10,000
Significant Quality Improvements for Communicator “14”
Positive Impact on Ship Schedule
Team Morale and Dog-food User Engagement
Source : Ross Smith, Director of Test, Microsoft Corporation
Slide 64
Corp04
32

35. Other Examples
• Working with Blind Relief Association to bring in
real users into accessibility testing
• Working with universities for content grading
• Mobile application testing across devices within
the company
Slide 65
Other Examples
Listen to Ross Smith, Director of Test at Microsoft,
on his thoughts on Crowd Sourced Testing
Slide 66
Corp04
33

36. Crowd Sourced Testing – Myths and Facts
Really?
1. Crowd Sourced Testing - supplemental technique, not stand
alone
2. Does not work in all situations; areas of niche to be reserved for
internal testing
3. Internal team to build a sense of empowerment that the crowd
adds to product quality
#1
Crowd Sourced Testing impacts core testing team adversely; threatens their positioning
in the product team
Slide 67
Crowd Sourced Testing – Myths and Facts
Really?
1. Think of Open Source Software….Linux - popular example of
open software collaboration
2. Crowd Sourcing easier for testing than development as crowd
often represents user base
3. Security, IP easier to manage in testing – no access to source
code
4. Practices discussed extendible to development as well
5. Top Coder.com another good example
#2
Crowd Sourcing is only for software testing. Development is a very technical and
specialized area to leverage the crowd for
Slide 68
Corp04
34

37. Crowd Sourced Testing – Myths and Facts
Partly True…
1. Management overhead slightly more...but this is inevitable in
current day global development models
2. Crowd is a smart and self – sufficient group; YOU DON’T WANT
TO MICRO-MANAGE
3. Best practices of what, when, how to crowd source will make
the effort streamlined and not chaotic
#3
Management overhead is significantly higher in Crowd Sourced Testing. Given the short
project deadlines, we do not have time or resources to manage a crowd sourced test
team
Slide 69
Crowd Sourced Testing – Myths and Facts
Partly True…
1. Scale potential of resources is huge
2. However, getting right resources at right time is challenging
3. Maintain a pool / common database and engage with the crowd
on ongoing basis even in lean periods
#4
Crowd Testers can be ramped up or down at very short notice, giving great head count
flexibility
Slide 70
Corp04
35

38. In Conclusion
Let’s:
• Pictorially walkthrough
Crowd Sourcing
• Revisit Take-Aways
• Look at Call to Action
Slide 71
Crowd Sourcing – Pictorial Walkthrough
Slide 72
Corp04
36

39. Crowd Sourcing – Pictorial Walkthrough
Slide 73
Crowd Sourcing – Pictorial Walkthrough
Slide 74
Corp04
37

40. Take-Aways revisited
• CS used across companies, domains,
technologies
• Understand varied manifestations, crowd
motivators, to customize your implementation
• Educated decision of what, when, how to
crowd source
• Acknowledge model’s challenges; be
transparent in seeking stakeholder approval
Slide 75
Call to Action
• Evaluate programs in your
group; gradually build on them
• Start small
• Try a Pilot / Proof of Concept
• Register to be a crowd
sourced tester – internally,
externally http://www.qainfotech.com/CS_Reg_Step1.php
Slide 76
Corp04
38

41. About QA InfoTech
• An independent software quality assurance and testing company,
founded in 2003, currently employing 600 people
• Five testing “Centers of Excellence” across the USA and India
• World-class testing labs
• Experience working with clients across various domains
• Bagged the “Top 100 places to work for in India*” award, three years
in a row
• Focus on the right balance of people, processes, technology
•
CMMi III, ISO 9001:2008, 20000-1:2005 certified
* Study conducted by Great Places to
Work Institute, India
Slide 77
QA InfoTech facilities in India
Q&A – Let’s find answers together!
Slide 78
Corp04
39

42. References
•
http://outsideinmarketing.wordpress.com/2010/05/08/four-types-of-crowdsourcing/
•
http://explore2win.blogspot.in/2012/10/what-is-crowdsourcing.html
•
http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf
Slide 79
Thank You
For more information, please:
• Contact us at info@qainfotech.com
• rajini.padmanaban@qainfotech.net
• Visit us at www.qainfotech.com
• Read our blog at www.qainfotech.com/blog
• Follow us on Twitter at www.twitter.com/qainfotech
USA
Office
International
Headquarters
Farmington Hills
Michigan, U.S.A.
Phone: +1-248-719-3409
Noida
Uttar Pradesh, India
Phone: +91-120-4292222
(Three additional testing facilities in India)
Slide 80
Corp04
40