Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Applause crowd testing meetup march 2015


Published on

CrowdTesting @ Applause

Published in: Software
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here
  • Winning the Lottery is Based on This [7 Time Winner Tells All] ★★★
    Are you sure you want to  Yes  No
    Your message goes here
    Are you sure you want to  Yes  No
    Your message goes here
  • Does Penis Size REALLY Matter? The truth comes out... ➤➤
    Are you sure you want to  Yes  No
    Your message goes here
  • Earn $500 for taking a 1 hour paid survey! read more... ★★★
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Applause crowd testing meetup march 2015

  1. 1. | INTRO TO CROWDTESTING C O M B I N I N G ‘ I N - T H E - L A B ’ & ‘ I N - T H E - W I L D ’ A P P R O A C H E S Peter Shih | VP of Channel Development |
  2. 2. | • Formerly known as uTest, the leader of in-the-wild software testing – Founded by pair of Israeli entrepreneurs – Founded in 2007; GA Launch in 2008 • Offices around the globe – Boston, San Francisco, NYC, Seattle, Tel Aviv, Berlin & Warsaw • World’s largest & most mature crowdtesting company – Raised 5 rounds of VC funding, totaling $80 million - Latest round led by Goldman Sachs in 2014 – Larger than all other crowdtesting companies combined 2 A L I T T L E A B O U T T H E C O M P A N Y
  3. 3. | 3,000+ CUSTOMERS 3 B Y T H E N U M B E R S 70,000+ B U I L D S T E S T E D 175,000+ P R O F E S S I O N A L T E S T E R S 500 Million+ A P P S T O R E R E V I E W S A N A LY Z E D OVER 1 MILLION I N - T H E - W I L D I S S U E S R E P O RT E D
  4. 4. | 4 C O M P A N I E S T H A T U S E C R O W D T E S T I N G Retail Travel SocialMedia
  5. 5. | 5 C O M P A N I E S T H A T U S E C R O W D T E S T I N G Health/FitnessFinancialSoftware Telco
  6. 6. | The Challenge PROBLEM WITH THE STATUS QUO •••
  7. 7. | Test Labs Are Here Users Are Here Modern apps & sites require real-world testing that mirrors the many different combinations of devices, use cases and locations of users 7 T H E P R O B L E M W I T H T H E S T A T U S Q U O
  8. 8. | • ‘Inside-out’ approach is still valuable, but it’s merely the starting point • 360 ₒ app quality requires an ‘outside-in’ view, which comes in many forms – Functional defect reports – Crash reports – Security vulnerabilities – Load and performance issues – Localization issues – Usability feedback – User sentiment analytics • And across all phases of the SDLC – Research – Pre-launch – Post-launch 8 D E T A I L S O F ‘ O U T S I D E - I N ’ A P P R O A C H
  10. 10. | DESIGN & DEVELOP 1 TEST & RELEASE 2 LISTEN & LEARN 3 ANALYZE & PLAN 4 360º APPQUALITY 5 EG: IBM EG: JIRA EG: CRITTERCISM EG: FLURRY 10 “Should I launch?” “What should I do next?” S O L V I N G T W O M A S S I V E C I O P R O B L E M S
  11. 11. | • Modern apps have different recipe for success – Infinitely more variables make up the test coverage matrix - Devices, OSes, browsers, carriers, locations, use cases – Traditional test playbooks & methodologies are merely starting point • Crowdtesting augments traditional QA efforts – True for web, but even more so for mobile and smart TVs/cars/watches – Move beyond the QA lab, closer to where users work, live & play – Close the loop between user POV and internal definition of quality • 360°app quality isn’t just about testing or dev ops – Requires modern tools and analytics that focus on users-centric quality – Necessity due to outsized user expectations + voice in app stores 3 K E Y T A K E A W A Y S 11
  12. 12. | The Challenge A 360 ₒ APPROACH TO APP QUALITY •••
  13. 13. | I N - T H E - W I L D T E S T I N G S E R V I C E S FU N C TION A L TESTIN G U SA B ILITY TESTING LOC A LIZATION TESTIN G LOA D TESTIN G SEC U R ITY TESTIN G Do your apps function flawlessly in the hands of your users? Are your apps as intuitive and easy-to-use as you think? Are your apps primed to thrive in different markets & cultures? Can your apps handle the peaks & pressures of the real world? Are your apps protecting your users data & privacy?
  14. 14. | • Real-world functional testing solutions – Ensuring apps work well in the hands of users • Compliments in-house testing – Team of testers provide you with scale, coverage & expertise • Testing on real devices, in real locations – Across OS versions, mobile browsers, wireless carriers – Exploratory testing & test case execution – Test around the clock or after major releases • Integrates with your bug tracking systems – JIRA, Bugzilla, Mantis, Pivotal, IBM Rational, Microsoft TFS and more • Fully-managed service to reduce your overhead – Integrated with your test services from QualiTest – Handle all tester communications and bug triaging – Professional testers from any region, on any device or OS – Integrated with your internal team, processes and systems F U N C T I O N A L T E S T I N G A P P R O A C H 14
  15. 15. | 15 I N - T H E - W I L D F U N C T I O N A L D E F E C T Learn About The Tester Tiered Approval Chat Directly With The Tester Reproducible Action Steps U I N OT R EN D ER IN G PR OPER LY
  16. 16. | • Real-world usability testing solutions – Ensuring your apps are designed intuitively for intended audience • Expert-led usability studies – Remote unmoderated testing performed by industry experts with an average of 12 years of expert usability experience • Audience matching – Survey 25+ respondents that match the demographics of your users • Video and audio experience captured – Watch and analyze your respondents use your app for the first time • Comprehensive report & consulting – Receive a complete report with findings, benchmarks and recommendations on how to improve user experience – Leverage your usability expert for advice on how to minimize potential confusion & frustration for first-time users U S A B I L I T Y T E S T I N G A P P R O A C H 16
  17. 17. | • Real-world localization testing solutions – Validate content and app localizations to ensure correct context across languages and cultures • Go beyond content – In addition to translations, verify formatting, currency conversions, taxes, shipping options and more • In-market localization testers – Understand the true implications of your localization choices by leveraging cultural natives • Double verification – All issues and recommendations are verified by an additional cultural native • Available string reporting – Output a string document to facilitate easier updating of your apps 17 L O C A L I Z A T I O N T E S T I N G A P P R O A C H
  18. 18. | • Real-world security testing solutions – Protect your users privacy and data • Expert-led and managed security testing – A white hat security expert will work with you to develop a comprehensive test plan • White hat security testers – Trusted team of security testers will attack your site through a combination of manual penetration and automated tools, looking for common vulnerabilities and creative new attacks • Comprehensive report – Receive a complete report with findings and recommendations on how to improve your site security • Remediation consulting – Leverage your security expert for advice on how to fix specific vulnerabilities and mitigate different types of risks S E C U R I T Y T E S T I N G A P P R O A C H 18
  19. 19. | • Real-world load testing solutions – Understand how your app performs under real world pressures • Expert-led and managed load testing – A Performance Engineer will assess current infrastructure, develop JMeter scripts, execute and analyze tests • Scalable and flexible load scenarios – A range of options to test your apps and sites and APIs - Large number of concurrent users - Varying amount of time on the app or site - With various bandwidth constraints to mirror real-world conditions • Comprehensive report – Receive a complete report with findings and recommendations on how to improve your apps’ performance 19 L O A D T E S T I N G A P P R O A C H
  20. 20. | 20 A P P Q U A L I T Y T O O L S APPLAUSE SD K Discover crashes in real time and distribute builds A PPLA U SE A N A LYTIC S Listen to what your users are saying in the app stores 3 6 0 ° A PP QU A LITY D A SH B OA R D Access all of your app quality results in a single place MOB ILE B ETA MA N A GEMEN T Run & manage beta programs at a global scale
  21. 21. | 3 6 0 ° A P P Q U A L I T Y D A S H B O A R D
  22. 22. | M O B I L E S D K ( I N C L U D E D I N F U N C T I O N A L )
  23. 23. | 23 U S E R S E N T I M E N T A N A L Y T I C S
  24. 24. | 24 U S E R S E N T I M E N T A N A L Y T I C S
  25. 25. | 25 C O M P E T I T I V E A N A L Y S I S
  26. 26. | 26 U S E R R E V I E W S E N T I M E N T
  27. 27. | The Challenge MATURING THE CROWDTESTING MODEL •••
  28. 28. | • Anyone can join, but many do not qualify for paid projects – ‘Test the Tester’ – initial round of validations - Hard skill vetting: technical testing skills; device knowledge; OS expertise - Soft skill betting: following policies and processes, communication skills – Ongoing validations from every project: - Tester Ratings, based on the quality of reports and impact of defects found › Value determines how much tester gets paid for their work › Quality affects tester’s performance ratings › Judged by customers, project managers, and team leads - Training and Mentoring programs to sharpen tester skills › Crowd  Teams  Community of Experts • Protecting customer IP – NDA in our standard Terms of Use for every tester – Custom NDA supported for each customer – Proof: 70,000+ test cycles with zero breaches of IP Q U A L I T Y Q U A L I T Y Q U A L I T Y 28
  29. 29. | • Project Managers are highly aligned with internal Dev & QA teams – Support your expectations, timelines, and priorities – Team Leads in various time zones to provide around-the-clock coverage – Recommend new ideas based on experience with many international companies • Integration with your systems and test methodologies – Bug Tracking Systems: JIRA, HP Quality Center, IBM Rational, Microsoft TFS, Github, Bugzilla, Pivotal Tracker, Asana, Mantis, FogBugz, Rally, and more – Methodologies such as continuous integration, test-driven development, agile, waterfall, lean software development, behavior driven development, etc. • Continuous improvement and optimization – Assess results from every project and deliver monthly management reports – Identify favorite testers and invite new members to optimize team performance F U L L Y - M A N A G E D S E R V I C E 29
  30. 30. | The Challenge FINAL POINTS •••
  31. 31. | • App quality was traditionally conveyed by companies to end users – Testing in the lab was sufficient due to smaller coverage/environments • App quality is increasingly being overruled by end users – Through app store ratings & reviews, social media, blogs, and more – Across multiple dimensions: usability, security, interoperability, etc. – And most importantly, the cost and publicity of failure is higher than ever • Applause delivers tools & services that address both views – ‘Inside-out’ point of view - Collaborating with internal development & QA on strategy and methodology – ‘Outside-in’ point of view - Complementary ‘in-the-wild’ testing on real devices across location & OS - Reveal important feedback from end users before launch 31 K E Y T A K E A W A Y S
  32. 32. | The Challenge Q&A •••