Your SlideShare is downloading. ×
0
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

uTest Overview - crowd-sourced, "in the wild" testing for mobile and web apps

880

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
880
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. uTest IntroductionIn-the-wild testing for web & mobile appsPresented by Richard Macksoud | Major Account Manager | richardm@utest.com | Oct 2012 |
  • 2. The Challenge The Case For In-The-Wild Testing | 2
  • 3. Testing Where Your Users Are…. Your Test Lab Is Here Your Users Are HereTesting inside the lab is vital. But why are many applications tested only inside the lab(whether manual or automated)? We need a new approach: In-The-Wild Testing |
  • 4. What Is In-The-Wild Testing • Increasingly, companies have moved a portion of their testing closer to where users work, live and play. Why? 1. Apps are used under a wide range of environments - Ever-growing list of devices - Vast range of OS, browsers and third-party apps - Imperfect connectivity (Wi-Fi and cellular) - Infinite number of locations 2. Users have less patience than ever for buggy apps - Switching costs are lower - Social means customer issues are no longer private 3. In-the-lab testing alone cannot meet the testing demands of modern apps | 4
  • 5. Achieving Testing Coverage In-The-Lab Testing In-The-Wild Testing • In-House: • Beta Testing – Pros: In-house team makes communication easier; – Pros: No direct costs; lives outside the test lab environment; professional testers who are tightly integrated into SDLC range of mobile devices, carriers, OS, locations – Cons: Capacity constraints; testing may not match user – Cons: Users are not professional testers; provide subjective demographics; maintaining adequate collection of mobile feedback that is not diagnostic, which raises the cost for the devices is prohibitively expensive company; also puts unfinished product in front of customers • Outsourced Testing • Crowdsourced Testing – Pros: Professional testers; access to range of devices – Pros: Lives outside the test lab across real devices, carriers, – Cons: Hidden costs; not easily scalable; challenges in OS and locations; professional testers who mirror end users; communication and time zone; still lab-based testing access to range of mobile devices & connections – Cons: There is a direct cost; requires careful partner vetting; requires communication efforts between testers & developers | 5
  • 6. The uTest Advantage• Yeah, we’re a software testing company… – But there are 5,000+ testing companies on the planet – Plus, most companies have in-house test teams• So what makes us so special – so unique? – Why do Google, FB, AT&T, HBO & others come back each month?• The difference: – Our model enables us to scale infinitely and immediately - From 3 testers, to 30, to 300, and back to 0 whenever they want – Our model enables us to lower the total cost of testing - EG: SMBs, unlimited exploratory testing for ½ the cost of one QA engr – Our model enables us to test under real-world conditions - Live testers, real devices, imperfect conditions… in-the-wild | 6
  • 7. The Challenge About uTest | 7
  • 8. Who We Are uTest is the leading provider of in-the-wild software testing | 2
  • 9. Since Our 2008 Launch 60,000+ testers from 190 countries 20,000+ Functional, load,test cycles security, L10n & usability Web, mobile & desktop apps1,000+ recurring customers Enterprises, startups & NPOs | 3
  • 10. Among 1000+ CustomersSoftware Retail Media Social Telco | 10
  • 11. The Challenge Types of Testing Services | 11
  • 12. Testing Services • Functional testing • Usability testing • Load testing • Security testing • Localization testing • Across web, desktop & mobile apps | 12
  • 13. Testing Expertise • On-demand teams in-the-wild • Project manager • Test team lead • Dedicated test engineers – On-site – Off-site | 13
  • 14. Consulting Services • Mobile build automation • Release management • Testing methodology • Test process improvement & implementation | 14
  • 15. uTest SaaS Platform • Define Testing Objectives • Choose Coverage • Select Testers • Activate and Lock Tests • Submit Bugs, Test Cases, Feedback • Review Results “Realtime” • Interact with Test Team • Approve and Export Results | 15
  • 16. uTest Apphance • Distribute new builds over the air – Upload a new build of your app for any OS platform – Instantly send to real test devices (in-house or uTest) • Real-time crash report – Crashes in pre- & post-production automatically captured – Crash details captured in real-time – Central dashboard for monitoring trends and taking action • Report bugs and feedback directly from your app – Testers/users shake device to report bugs/feedback – Device data, session information & annotated screenshots attached | 16
  • 17. How It Works Analyze testing Develop test Build core Execute cycles Review and objectives strategy testing team on demand refine • • General test reqs • Identify • Post mortem• Target market • Test team make up • Qualify • Modify strategy• Risk tolerance • Standards • Invite • Deploy across projects• Quality concerns • Timelines • Train Work w/uTest to specify testing needs Selected testers - OS, Browser, Tester Coverage invited to project & trained, if required - Timing, Frequency, Specific Instructions Testers report Customer fixes issues as necessary in real-time uTest provides completed report to Bugs triaged and prioritized for customer customer | 17
  • 18. Functional Testing • Real-world functional testing – Highly complimentary to in-house efforts – Real-world testing, across: - OS & browser - ISP, firewall & anti-virus – Exploratory or test case execution – Manual or automated – Can also test integration between your product and third- party apps (EG: Facebook, Twitter, YouTube, Slideshare) | 18
  • 19. Load Testing • Real-world load testing – Complimentary to existing efforts – Three flavors of load & performance testing - Live load - Synthetic load - Hybrid load – Can utilize automated load tools (SOASTA, Load Runner, BrowserMob, et al) – Performance benchmarking also available | 19
  • 20. Security Testing • Real-world security testing – Augments in-house efforts - Team or solo security probes - Dynamic manual and automated testing – Helps achieve PII- and PCI-compliance – Utilize industry-leading tools – Protection against common vulnerabilities checklist - OWASP 10 - SANS 25 – Focus is on black box security testing - Probing front end UI for security- or privacy-related vulnerabilities | 20
  • 21. Usability Testing • Real-world usability testing – Survey-based usability and UAT – Virtual focus groups by: - Age , gender & location - Education level & employment status - Industries & hobbies – UX experts available for: - Objective usability audit - Survey writing - Focus group management - Results validation and data analysis | 21
  • 22. Localization Testing • Real-world localization testing – Reported issues validated by 2nd party – Real-world localization validation, across: - Locations & languages (native speakers) - Age & education level – Validating all facets of localization: - Language translation - Currency conversion - VATs and other taxes - Shipping options | 22
  • 23. Rating & Reputation System• Each testers is rated by customers• Performance-based ratings based upon a dozen factors: – Participation Level: - Lifetime & recent participation: # active test cycles, # reported issues – Quality of Participation - Approval percentage for bugs, test cases and usability surveys - Accuracy of bug type & severity classifications - Ability to write test cases & create automated test scripts | 23
  • 24. The Challenge Wrap-Up & Take-Aways | 24
  • 25. Key Takeaways • Apps universe has forever changed testing – Exponentially more diverse user environments - Devices - OS & software configs - Locations - User demographics – Users less tolerant – Quality complaints no longer private • QA orgs must play catch-up – By continuing to improve inside the lab – By moving beyond it, where users work, live and play – App quality winners are combining: in-the-lab testing + in-the-wild testing | 25
  • 26. The Challenge Questions? Answers. | 26

×