Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Implementing Crowdsourced Testing


Published on

In today’s market, global outreach, quick time to release, and a feature rich design are the major factors that determine a product’s success. Organizations are constantly on the lookout for innovative testing techniques to match these driving forces. Crowdsourced testing is a paradigm increasing in popularity because it addresses these factors through its scale, flexibility, cost effectiveness, and fast turnaround. Join Rajini Padmanaban as she describes what it takes to implement a crowdsourced testing effort including its definition, models, relevance to today’s development world, and challenges and mitigation strategies. Rajini shares the facts and myths about crowdsourced testing. She spans a range of theory and practice including case studies of real-life experiences and exercises to illustrate her message, and explains what it takes to maximize the benefits of a crowdsourced test implementation.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Implementing Crowdsourced Testing

  1. 1. MH AM Tutorial 4/29/13 8:30AM Implementing Crowdsourced Testing Presented by: Rajini Padmanaban QA InfoTech Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Rajini Padmanaban As director of engagement at QA InfoTech, Rajini Padmanaban leads the engagement and relationship management for some of QA InfoTech's largest and most strategic accounts. Rajini has more than eleven years of professional experience, primarily in the software quality assurance area. She actively advocates software quality assurance through evangelistic activities; writes on test trends, technologies, and best practices; speaks at conferences including STAREAST 2012 and STARWEST 2012; and provides insights on software testing to analyst firms such as Gartner and IDC. Read Rajini’s official blog and reach her at
  3. 3. Implementing Crowd Sourced Testing Rajini Padmanaban Director of Testing Engagements Your Software Testing Partner We help you build better software Agenda (1 of 2) Topic Time (in minutes) A Peek into Software Quality 10 Crowd Sourced Testing - Defined 10 Understanding Varied forms for Crowd Sourcing 15 Let’s be the Crowd – Exercise Time 20 Crowd Sourced Testing Relevance in Current 10 Scenario Limitations of Crowd Sourced Testing 15 Slide 2 Corp04 1
  4. 4. Agenda (2 of 2) Topic Time (in minutes) Practices for a Successful Crowd Sourced Test Effort 30 Know What Not to Crowd Source Getting Stake Holder Buy-in Case Study & Examples 15 15 30 Myths and Facts Conclusion and Q&A 15 15 Slide 3 Requirements A device to connect to the internet for one of our exercises (laptop, smart phones, tablets ) Slide 4 Corp04 2
  5. 5. A Peek into Software Quality Test is responsible for Quality Focused on creating a quality deliverable Ensure we don’t sacrifice quality for the sake of schedule Empower the rest of the team to partake in improving quality Slide 5 Defining Quality • • • • • • • ISO 9000: Degree to which a set of inherent characteristics fulfill requirements Six Sigma: Number of defects per million opportunities Philip B. Crosby: Conformance to requirements Joseph M. Juran: Fitness for use by the customer Gerald M. Weinberg: Value to some person Robert Pirsig: The result of care American Society for Quality: A subjective term for which each person has his or her own definition. In technical usage, quality can have two meanings: a. The characteristics of a product or service that bear on its ability to satisfy stated or implied needs b. A product or service free of deficiencies Slide 6 Corp04 3
  6. 6. Software Development Slide 7 Attacking the Inverse Exponential Testability Fault Injection Stress Testing End-to-End Exploratory Code Inspection Leverage the Beta Test Crowd Slide 8 Corp04 4
  7. 7. Portfolio Selection Theory Diversification in investing tells us that risk lessens as the number of investments in the portfolio increases 10% 10% 10% 10% 10% 10% 10% 10% 10% 10% Slide 9 Audience Interaction Time What Does Quality Mean to you ? Slide 10 Corp04 5
  8. 8. What is Crowd Sourced Testing? Crowd sourcing is the process of getting work, usually online, from a crowd of people. The word is a combination of the words 'crowd' and 'outsourcing'. The idea is to take work and outsource it to a crowd of workers (similar definition on Wikipedia) Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that exploits the benefits, effectiveness, and efficiency of crowd sourcing and the cloud platform towards software quality assurance and control. With this new form of software testing, the product is put to test under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all, bug-free I prefer the word engaging rather than delegating Crowd Source Crowd sourcing your software testing consists of delegating onto a number of internet users the task of testing your web or software project while in development to ensure that it contains no defects, referred to as bugs Slide 11 What is Crowd Sourced Testing? .so what really is Crowd Sourced Testing? • Is it all about a pool of testers leveraged and paid per valid bug? No, not just that • Think: Slide 12 Corp04 • Sourcing relevant people from within your company • Across disciplines and levels • Sourcing end users from various disciplines – e.g. teachers, students, nurses, bankers • Partnerships with universities, organizations leveraging domain knowledge • In essence, think of the community at large to test your product 6
  9. 9. What is Crowd Sourced Testing? Technologies Company Scale Domains Internal Crowd Sourcing Services Companies Product Companies External Crowd Sourcing Slide 13 Understand Factors that Motivate the Crowd Portfolio Selection Theory 10% Crowd Sourced Testing: Portfolio Selection Theory applied in QA 10% 10% 10% 10% 10% 10% 10% 10% 10% Slide 14 Corp04 7
  10. 10. A Practical Solution to Leverage Slide 15 Set # 1 • Crowd Sourcing: • Is not restricted to any single company, technology, domain • Has a much larger yet simpler meaning than what it is often portrayed to be Slide 16 Corp04 8
  11. 11. Types of Crowd Sourcing - Explained Crowd Creation Crowd Voting Crowd Wisdom Crowd Funding Slide 17 Types of Crowd Sourcing - Explained • Crowd Creation • Invite crowd to create subject content • Common usage areas: software development, translations, photos repository • Content usage by crowd or by organizations • Typical crowd motivators – Money, fun, community involvement, brand loyalty • Well known examples – Linux, iStockPhoto, 99 designs Slide 18 Corp04 9
  12. 12. Types of Crowd Sourcing - Explained Slide 19 Types of Crowd Sourcing - Explained • Crowd Voting • Leverage crowd’s judgement to organize, filter, stack rank content Most popular of crowd • Common usage areas: retail, media, simple sourced versions (1:10:89 rule) yet powerful decisions across domains 1% create • Results used by an organization 10% vote and rate 89% consume • Typical crowd motivators – Fun, community involvement, brand loyalty • Well known examples – American Idol, -Jeff Howe, Author of CrowdSourcing Slide 20 Corp04 10
  13. 13. Types of Crowd Sourcing - Explained Slide 21 Types of Crowd Sourcing - Explained • Crowd Wisdom • Harnesses crowd’s knowledge to solve problems, predict future outcomes • Common usage areas: reality shows, quality assurance, exchanges markets • Results used by an organization, crowd • Typical crowd motivators – Money, fun, product transparency, brand loyalty • Well known examples – Who wants to be a Millionaire Slide 22 Corp04 11
  14. 14. Types of Crowd Sourcing - Explained Did You Know? The crowd’s answer in “Who Wants to be a Millionaire” was right 91% of the time compared to “Ask an Expert”, which was right 65% of the time Source: The Wisdom of Crowds, James Surowiecki Slide 23 Types of Crowd Sourcing - Explained • Crowd Funding • Leverages crowd to finance individuals or groups that might otherwise be denied credit or opportunity • Common usage areas: ideas in developing nations, educational domain • Results have far reaching impact • Typical crowd motivators – Money, social causes / community involvement • Several interesting examples at: Slide 24 Corp04 12
  15. 15. Set # 2 • Crowd Sourcing Options are: • plentiful • available for a diverse set of community needs – both commercial and not-for-profit • to be customized based on results needed and the crowd motivating factors Slide 25 Exercise Time – Let’s Crowd Source Walt Disney – he won 26 oscars including 4 in a same year Who has won the most oscars? 1.Walt Disney 2.Elizabeth Taylor 3.Meryl Streep 4.Jack Nicholson Slide 26 Corp04 13
  16. 16. Exercise Time – Let’s Crowd Source Mexico derived from an Aztec legend that a city would be built where they spot an eagle eating a serpent, which is now Mexico City On which national flag is there an eagle and a snake? 1.China 2.Mexico 3.Greece 4.Spain Slide 27 Exercise Time – Let’s Crowd Source Idaho, followed by Washington and Oregon (in 2011) Source: Which American state produces most potatoes? 1. Oregon 2. California 3.Idaho 4.Washington Slide 28 Corp04 14
  17. 17. Set # 3 Did You Know? Collective wisdom of the crowd often surpasses that of an expert The crowd’s answer in “Who Wants to be a Millionaire” was right 91% of the time compared to “Ask an Expert”, which was right 65% of the time Source: The Wisdom of Crowds, James Surowiecki Exercise Time – Let’s Crowd Source • Please take about 10 minutes to test from usability and accessibility standpoints • Need tools - screen readers, magnifiers? • • - e.g. Magnifier on Windows, Zoom for Mac Slide 30 Corp04 15
  18. 18. Exercise Time – Let’s Crowd Source • Discuss test results / feedback – • Goal of our study: demonstrating richness and diversity of crowd’s feedback as end users • Feedback from our visually challenged crowd Amazon Accessibility Issues Slide 31 Product Quality Scenario – as it stands today • • • • Shrinking release cycles Close scrutiny on overall spend Collective ownership of quality Need to: • Focus on product domain knowledge; not just disciplinary knowledge • Understand competing products • Creatively emulate end user scenarios • Mimic user environments via lab and simulations • Consider global product distribution effects Slide 32 Corp04 16
  19. 19. A Practical Solution to Leverage • • • • • • Distribute quality effort Enhance productivity through global solutions Pool in end users into quality implementation Use live environments to test Analyze potential partnerships for SME Flexible, selective and cost effective testing .........Bring in the Crowd Slide 33 When does Crowd Sourced Testing Succeed? • Diversity of knowledge, background, experience • Independence in testing process • Wide spread domain background for product requiring SMEs • End user scenarios and environments difficult to simulate in-house • Work aligns with factors that motivate crowd Slide 34 Corp04 17
  20. 20. Inherent Challenges • Random test efforts do not fit into quality strategy • Choosing, sustaining a crowd sourced team • Ongoing motivation, floating crowd Any Challenges from • Keeping everyone in sync on product dynamics your experience that • Communication challenges you would like to share? • Management overhead including logistics • Stakeholder buy in • Securing product IP before release Slide 35 Perceived Limitation Quality of product adversely impacted by an amateur crowd - Solidify your implementation plans - Use instrumentation wherever possible - Refer case studies – understand successes and failures Slide 36 Corp04 18
  21. 21. Set # 4 • Current product scenario demands creative solutions to test within existing constraints • Crowd Sourcing is not a no-brainer solution to all problems • Not even a stand-alone solution in most cases • Understand its strengths and challenges in customizing it to your needs • Understand “What, When and How to Crowd Source in Testing” Slide 37 Practices for Successful Crowd Testing Customize your practices mindful of your constraints Slide 38 Corp04 19
  22. 22. “What, When and How” of Crowd Testing • When to Crowd Test: • Product works reasonably well E2E • Ready to incorporate crowd’s feedback • No time or resources for formal testing .try to avoid this situation • Source content files are ready • Ongoing feedback from a chosen SME team at specific stages Slide 39 Examples Started off with informal testing due to lack of time Slide 40 Corp04 Ongoing MVP programs at Microsoft Product ready to be tested E2E and feedback incorporated 20
  23. 23. “What, When and How” of Crowd Testing • What to Crowd Test: • User facing features • Areas where external team feedback is important – e.g. design, feature set, performance • Specialized areas of test: • localization (context based verification) • performance • compatibility, devices testing • Content testing – valuable SME knowledge • Align crowd’s focus areas into test strategy • Know what not to crowd source Slide 41 Examples Exam Grading Content Testing using SME / Localization Slide 42 Corp04 21
  24. 24. “What, When and How” of Crowd Testing • How to Crowd Test: • Pick the right areas, team and time • Minimize duplication, overhead of sifting through known issues • Clear internal ownership: • Communication, technical query resolution • Prompt follow up and responses • Team up-to-date on product changes • Use of collaborative tools • Think about interactions amongst testers Slide 43 “What, When and How” of Crowd Testing • How to Crowd Test: • Leverage cloud, VPC for ease, secured access • Identify crowd motivators • Use management theories – Maslow’s hierarchy of needs? • Identify tasks that align with crowd motivators • Work on stakeholders buy-in Self Actualization – Pursue Inner Talent, Creativity, Fulfillment Self Esteem – Achievement, Mastery, Recognition, Respect Belonging – Friends, Family, Spouse, Lover Safety – Security, Stability, Freedom from Fear Psychological – Food, Water, Shelter, Warmth Slide 44 Corp04 22
  25. 25. Set # 5 • Understand when a crowd test effort succeeds: • Diversity of knowledge, background, experience • Independence Examples, Bestprocess Any in the testing • Wide spread domain background for product Practices from your requiring SMEs experience that you • End user scenarios and environments difficult to would like to share? simulate in-house • Work aligns with factors that motivate the crowd Slide 45 Know What Not to Crowd Test • Features with moving pieces, demanding close collaboration • Sensitive IP • Environment specific complex testing • Tasks requiring immediate and regular turnaround - BVTs • Core testing activities • • • • Test automation, TDD scripts Regression testing First round of performance, security, integration testing Mundane testing tasks that don’t need diversity Slide 46 Corp04 23
  26. 26. Some Examples • Simple performance tests such as PLTs – use tools • Understand pros and cons specific to your scenario – study by University of Texas, Austin* on Crowd Sourcing for Usability Testing vs. Lab Usability Testing on a college website Lab Usability Test Crowd Sourced Usability Test Participants 5 55 (14 spammers) Participant Demographics Students Crowdworkers Age 24 to 33 19 to 51 Education level Bachelor’s degree and Master’s degree All levels Experience with similar websites Speed Yes: 100% Yes: 77% No: 23% Less than 4 hours total. Participant Costs None Approximately 30 min. per session. $2.92 for pilot test $23.41 for final test (Avg: $0.48/tester) * - Slide 47 Some Examples Major Problems Identified Lab Usability Test Crowd Sourcing Usability Test Font size too small Out-of-date information Menu overlap Irrelevant picture Invisible tools Information not cross-linked Lack of sort function Navigation unclear Search box difficult to locate Slide 48 Corp04 24
  27. 27. Some Examples Advantages Disadvantages More participants Lower Quality Feedback High Speed Less Interaction Low Cost Spammers Various Backgrounds Less Focused User Groups In Conclusion: 1. In this scenario, usability testing would be better off done by students of the college website; choosing target users is very important 2. If crowd sourced testing is relevant based on your user profile, design the effort with care; using same tests and questions as that of lab testers for crowd testers may not yield great results (see section 4.1.2 on Test Redesign) Slide 49 Stake holder Buy In • Stakeholder resistance to Crowd Sourcing largely inline with model’s challenges: • Product IP, privacy issues • Internal team motivation • Quality of test effort and product • Additional overhead in effort management • Randomized test effort – tactical in nature Slide 50 Corp04 25
  28. 28. Stake holder Buy In 1. Show steady progress 2. Stick to pre-defined communication protocols 3. Communicate the good and the bad 4. Be on top of new stakeholders or new concerns Identify stakeholders; prioritize the team; engage early 1. Supplemental test technique 2. Acknowledge problems 3. Explain the technique as applicable to your product; don’t assume their know-how Walk through the test strategy including crowd sourcing plans; understand their concerns Ongoing Communication 1. Practical demonstration of the model; adds to your confidence too! 2. Additional overhead at start but pays off in longer run 3. Run a pilot like a regular CS test program but of smaller scale Undertake Pilot if needed 1. Educated decision 2. Explain implementation plans – what & what not, when, how, 3. Define checks and balances for internal team and CS team roles 4. Map solutions to address each concern identified earlier Explain Solutions Slide 51 Set # 6 • Acknowledging what not to crowd source will: • Fasten stakeholder approval process • Help not impact internal team motivation Slide 52 Corp04 26
  29. 29. Successful Work Patterns Core Unique Future Work Skills Work Skills Work Skills In-Job Behaviors Organizational Citizenship Behaviors Slide 53 Successful Work Patterns Core Work Skills Unique Work Skills Future Work Skills In-Job Behaviors Organizational Citizenship Behaviors Slide 54 Corp04 27
  30. 30. Localization Testing • Hard, Large-Scale Problem • Windows 7 ships in 100 languages • Thousands of strings and screens per release • Traditional model of localization testers expensive and difficult to find Slide 55 Let’s Play Slide 56 Corp04 link 28
  31. 31. The Language Quality Game Slide 57 Results Total Screens Reviewed: Over 500,000 Total Number of Over 4,500 Reviewers: Screens per Reviewer: Average 119 Significant Quality Improvements for Windows 7 Positive Impact on Ship Schedule Team Morale and Subsidiary Engagement Internal sourcing alleviates security and access issues Slide 58 Corp04 29
  32. 32. Amplify Skill with Volume Individual dialects, nuances, hard to detect with a single vendor – crowd does a better job Slide 59 Reduce Cost with Discovery, Instrumentation No need to install Telemetry to direct effort Slide 60 Corp04 30
  33. 33. Reduce Risk with Diversity Portfolio theory in Quality Assurance Slide 61 Trust and Transparency Increase Effectiveness Inclusion = Trust, Trust = Enthusiasm Subsidiary engagement with Language Quality Game Momentum for Win 7 Slide 62 Corp04 31
  34. 34. Players earn points for disaster relief agencies Microsoft donates $ based on leader board Individual players can sponsor tasks or scenarios Slide 63 Results Players Over 1,000 Feedback increase: > 16x Feedback received: 10,000 Significant Quality Improvements for Communicator “14” Positive Impact on Ship Schedule Team Morale and Dog-food User Engagement Source : Ross Smith, Director of Test, Microsoft Corporation Slide 64 Corp04 32
  35. 35. Other Examples • Working with Blind Relief Association to bring in real users into accessibility testing • Working with universities for content grading • Mobile application testing across devices within the company Slide 65 Other Examples Listen to Ross Smith, Director of Test at Microsoft, on his thoughts on Crowd Sourced Testing Slide 66 Corp04 33
  36. 36. Crowd Sourced Testing – Myths and Facts Really? 1. Crowd Sourced Testing - supplemental technique, not stand alone 2. Does not work in all situations; areas of niche to be reserved for internal testing 3. Internal team to build a sense of empowerment that the crowd adds to product quality #1 Crowd Sourced Testing impacts core testing team adversely; threatens their positioning in the product team Slide 67 Crowd Sourced Testing – Myths and Facts Really? 1. Think of Open Source Software….Linux - popular example of open software collaboration 2. Crowd Sourcing easier for testing than development as crowd often represents user base 3. Security, IP easier to manage in testing – no access to source code 4. Practices discussed extendible to development as well 5. Top another good example #2 Crowd Sourcing is only for software testing. Development is a very technical and specialized area to leverage the crowd for Slide 68 Corp04 34
  37. 37. Crowd Sourced Testing – Myths and Facts Partly True… 1. Management overhead slightly more...but this is inevitable in current day global development models 2. Crowd is a smart and self – sufficient group; YOU DON’T WANT TO MICRO-MANAGE 3. Best practices of what, when, how to crowd source will make the effort streamlined and not chaotic #3 Management overhead is significantly higher in Crowd Sourced Testing. Given the short project deadlines, we do not have time or resources to manage a crowd sourced test team Slide 69 Crowd Sourced Testing – Myths and Facts Partly True… 1. Scale potential of resources is huge 2. However, getting right resources at right time is challenging 3. Maintain a pool / common database and engage with the crowd on ongoing basis even in lean periods #4 Crowd Testers can be ramped up or down at very short notice, giving great head count flexibility Slide 70 Corp04 35
  38. 38. In Conclusion Let’s: • Pictorially walkthrough Crowd Sourcing • Revisit Take-Aways • Look at Call to Action Slide 71 Crowd Sourcing – Pictorial Walkthrough Slide 72 Corp04 36
  39. 39. Crowd Sourcing – Pictorial Walkthrough Slide 73 Crowd Sourcing – Pictorial Walkthrough Slide 74 Corp04 37
  40. 40. Take-Aways revisited • CS used across companies, domains, technologies • Understand varied manifestations, crowd motivators, to customize your implementation • Educated decision of what, when, how to crowd source • Acknowledge model’s challenges; be transparent in seeking stakeholder approval Slide 75 Call to Action • Evaluate programs in your group; gradually build on them • Start small • Try a Pilot / Proof of Concept • Register to be a crowd sourced tester – internally, externally Slide 76 Corp04 38
  41. 41. About QA InfoTech • An independent software quality assurance and testing company, founded in 2003, currently employing 600 people • Five testing “Centers of Excellence” across the USA and India • World-class testing labs • Experience working with clients across various domains • Bagged the “Top 100 places to work for in India*” award, three years in a row • Focus on the right balance of people, processes, technology • CMMi III, ISO 9001:2008, 20000-1:2005 certified * Study conducted by Great Places to Work Institute, India Slide 77 QA InfoTech facilities in India Q&A – Let’s find answers together! Slide 78 Corp04 39
  42. 42. References • • • Slide 79 Thank You For more information, please: • Contact us at • • Visit us at • Read our blog at • Follow us on Twitter at USA Office International Headquarters Farmington Hills Michigan, U.S.A. Phone: +1-248-719-3409 Noida Uttar Pradesh, India Phone: +91-120-4292222 (Three additional testing facilities in India) Slide 80 Corp04 40