Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crowd Testing Framework : Mobile Application Testing


Published on

Trends, Challenges & Solutions for Mobile Application Testing : by Siddharth Shinde, Sr QA Specialist - Ishi Systems

Published in: Mobile
  • Test your mobile web and native mobile applications on multiple mobile devices that increasingly command adoption & usage for enterprise and everyday computing.
    Are you sure you want to  Yes  No
    Your message goes here

Crowd Testing Framework : Mobile Application Testing

  1. 1. Crowd testing framework: Trends, Challenges and Solutions for Mobile application Testing 0 Siddharth Shinde Sr.QA Specialist
  2. 2. Agenda  Software testing evolution  Mobile Testing Challenge  Why mobile testing is difficult?  Crowd Testing as a solution  Principles & Features of crowd testing  Success factor  Summary
  3. 3. Evolving IT landscape presents more challenges than ever! More customer driven IT - Testing Goals - Speed - Cost -Quality More cloud based Shorter production cycle More HW and SW choices More open source More users, more places
  4. 4. What it does? For achieving high Quality, Speed and Costs have gets compromised.
  5. 5. Software Testing Evolution SOFTWARE TESTING No dedicated Testing team Dedicated testing team (in house) Outsourced testing SOFTWARE DEVELOPMENT (In house) resources Resources in the cloud
  6. 6. Mobile Testing : Challenge  Only 31% of firms surveyed currently test mobile applications. Why?  Many report they do not have appropriate tools or expertise and limited access to the devices to test effectively mobile applications
  7. 7. Mobile Trends  Transactions in both B2C and B2B  Products and services  Account management  Internal processes (B2E)  Business intelligence  Process management  Social collaboration
  8. 8. Business as usual in mobile testing Functionality Performance/Load Security System
  9. 9. ... Mobile Testing additional hot spots User experience Functionality Screen Size Platform Performance/Load Security System
  10. 10. Device breakup creates a need to perform device level testing & UAT for mobile apps
  11. 11. Why Mobile Testing is difficult? (1/2) Touching the device Touch, swipe, pinch, rotate, move Diversity in platforms, OSs & devices Android is the most fragmented, with 5 major versions corresponding to 9 API sets. In typical markets for feature phones, there can be more than 100 different device types in the installed base Automation challenges Sophisticated user experiences involve touch, gestures, GPS location, audio, sensors (accelerometers) and physical actions (touching the handset to NFC readers). Such interactions can't be fully scripted or simulated, and may involve manual testing on real devices Network performance Cellular networks are nondeterministic, and exhibit wide variations in bandwidth and latency. Users in different locations accessing different operators can experience vastly different application performance Application complexity and sophistication Mobile devices and applications are becoming more sophisticated, using techniques such as context, 3D graphics and gaming. Greater sophistication implies more complex testing New OS versions often break applications Developers have no control over when new OS versions will appear, and when or whether users will upgrade. Thus, it's common for new OS releases to break existing native applications
  12. 12. Why Mobile Testing is difficult? (2/2) New Technology risk Mobile is at the leading edge of new technologies, such as HTML5, that are not yet well-understood in terms of testing Performance variations There is a performance difference of greater than one order of magnitude across devices in the smartphone installed base. An app or HTML5 website that runs well on a top-end device may not be acceptable on a low-performance handset Operator intervention Operators may modify mobile Web content to optimize network performance so that desk-based testing may not show what a real world handset user experiences Contextual issues Mobile applications and websites are used in a wide range of contexts, which raises many new testing challenges. Applications dealing with critical or regulated data may demand much more rigorous testing Peripherals Smartphones are acquiring a growing range of add-on devices, such as Bluetooth peripherals, that are generally not accessible to testing tools Testing user opinions Many applications will be distributed via public app stores, where user reviews and attitudes play a large part in determining whether an app is downloaded. Understanding user reactions to an app and its store collateral before it's published may be valuable
  13. 13. Even <<for free>> app Must be <<right first time>> Must get into the market<<first>>
  14. 14. Outsourcing QA Needs Mobile and Web CROWD TESTING
  15. 15. CROWD – Dictionary meaning  “The mass or multitude of ordinary people”
  16. 16.  Crowdsourcing is the process of getting work or funding, usually online, from a crowd of people. Outsourcing Crowd Sourcing Crowd Testing Crowd
  17. 17. What is crowd testing?  “If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it.” (Paul Herzlich)  “ Crowtesting puts software to test under real-world conditions by real users, allowing companies to gather real insight and feedback and to identify defects quickly and cheaply” [CSTE2013]
  18. 18. Principles Diversity/coverage/realistic Independent and test as a user Payment system Immediate results
  19. 19. Crowd Testing Features
  20. 20. Players and Process
  21. 21.  Crowdtesting ensures that a software produces expected output across all desired platforms & user groups
  22. 22. Who are the crowd testers?  The testing crowd can be a pool of:  Inter-company testers  Intra/inter-project testers in a company  Strategic partners  Domain knowledge experts  Technology experts  Tool experts  Network of end users  Social network users Certified Specialist Recreational
  23. 23. Crowd Testers expertise and infrastructure = Crowd Testing types Crowd Testing Exploratory crowd testing Enterprise crowd testing
  24. 24. Exploratory Crowd Testing  Effective to cover all the many configurations  Good for non critical systems  Simulating real load and testing - Outcome not predictable - Discovering basic failures  To help making it more effective:  Show them the big picture of what they are doing  Formal process  A clear report
  25. 25. Enterprise Crowd Testing Large enterprises utilizing their in-house excess capacity for on-demand • Secured private cloud infrastructure • Testing Managers Organized • Crowd testers screening • Project management, tracking, effort planning Professionalized Standardized • Standardized process and reporting Enterprise Cloud Infrastructure crowd sourced testing
  26. 26. Enterprise («Managed») Crowd Testing  «Enterprise crowd testing, although still in its nascent and emerging stage, promises to be the next wave in the social, mobility and localization era.» [28+_TestExp14] Test Scheduling Crowd Testing Platform Testers Profile Knowledge center Comm. And collaboration Reporting Test management
  27. 27. Principles of crowd-testing Preparation Initiation Execution Evaluation Completion Identify customers needs Fix price Define report template Prepare crowd testers Prepare test environment Initiate testing Supervise and monitor Validate results Prepare client deliverables Testers Payment Present results to client Crowd testers ratings
  28. 28. Crowd Testing … Glossary Crowd Crowd Sourcing Managed Crowd Testing Outsourcing Crowd Testing Beta Testing Exploratory Crowd Testing Enterprise Cloud Testing
  29. 29. Opportunities for SE research in crowd testing: COMMUNITIES  Crowdsourcing in SE  Global Software Engineering  Social Software Engineering  Empirical studies  Software Testing
  30. 30. Effectiveness of different testing methods Crowd Testing Cloud Testing Lab Testing «When» shall we test with the crowd, when not to? Which software domain Which type of tests
  31. 31. Complementing different testing methods Crowd Testing Cloud Testing Lab Testing How crowd and Lab testing can be used together? Need: clients’ want to get the best from the combination of both Efficiency
  32. 32. Complementing different testing methods..cont. Systematic Mobile Testing Crowd Testing Solution to ? Dimensions to account for • Multi OS (Apple, Android, BBMS, MS…) • Multi Technology (Native, Hybrid, Browser…) • Multi manufacturer (Samsung, Micromax, Acer…) • Network Quality (Intermittent, disconnected, bandwidth..) • Device resource (Memory , CPU, Battery..) • Device capability (GPS, Camera…) • System of record (Don’t forget the backend) • Does the user like it? (UX)
  33. 33. Complementing different testing methods..cont. Test Selection Systematic Structural Testing Crowd Testing Test Execution Solution to ? Coverage issues
  34. 34. Complementing different testing methods..cont. Model Based Testing Mobile Testing Model Based Mobile Crowd Testing
  35. 35. Social Software Engineering  How to keep Crowd Testers motivated  How to keep them involved  Appropriate reward mechanisms
  36. 36. Types of Testing that can be Crowdsourced  ALL  Functional  Load/Performance  Usability  Security  Localization  Testing Modes  Exploratory  Test Case Based  Automated
  37. 37. Forms of crowd Fully Private Crowd • Utilizing internal infrastructure • Leverages organizational bench. Public Crowd • Utilizing external freelance capacity • Community and partnering Hybrid Crowd • Optimizing internal and freelance capacity
  38. 38. Application in Industry….Summary
  39. 39. Challenges in Crowd Testing Lack of standards Coverage Measurement No control Critical feature coverage Lack of ownership Reward mechanism Continuous improvement Communication and management
  40. 40. Concerns & Mitigations …..  Concerns : Security and Confidentiality  Mitigation:  Background check  NDA
  41. 41. Benefits Diversity
  42. 42. Few questions you should ask from your provider  Can I specify the criteria for selecting testers?  How do I know which functionality they covered?  Will I know which configurations were covered?  Do the testers know what they need to know?  Who is accountable for quality and schedule?  Delivery Assurance?
  43. 43. Success Factor Managed and controlled delivery Process Governance Structurally developed community Efficient technology backbone
  44. 44. Organization can opt… Build their own crowd testing solution Outsourcing to crowd testing Organizations/vendors
  45. 45. Summary The reason behind the success of this framework is that this method is essentially user-centric. This means it the user who determines the adoption and success quotient of particular software.
  46. 46. In Conclusion Crowd testing is growing exponentially Crowd testing may bring big benefits to QA Still, there are number of challenges to be managed
  47. 47. Contact If you are interested to this topic, please Stop by after the presentation Contact me at
  48. 48. References / Acknowledgements  [TestExp14] «Testing Experience, 25/2014» issue on Crowd testing  [RV_TestExp14] Rama Murari & Venkat Ramesh Atigadda, “Panorama of Crowd Testing”, in Testing Experience, 25/2014  [Stein_TestExp14] Markus Steinhauser “Crowd Testing – An Addition to Traditional Software Testing”, in Testing Experience, 25/2014   [34+_TestExp14] Dieter Speidel, Mayank Mittal & Mithun Sridharan “Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 34+  [28+_TestExp14] Saisubramanian Sivasailem & Dilipkarthik Jayamohan, “The Next Wave in the Social, Mobile and Localization Era  Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 28+   Henry Muccini, DISIM, University of L’Aquila 