Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Build your own MAD-LAB - for Mobile Test Automation

2,561 views

Published on

Slides from my talk on MAD-LAB delivered in vodQA - The Saga Continues! at Vuclip / ThoughtWorks

See blog for more details: https://essenceoftesting.blogspot.com/2017/07/why-i-needed-to-build-my-own-mad-lab.html

Description:
Building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!

Setting the stage - I am coordinating all Testing activities for VIU - an OTT (over-the-top entertainment) product available on Android, iOS and WAP platforms. This product delivers high quality, popular video content in various different languages for consumers in various different regions. One of the main items in my charter is to implement functional test automation for consumer / user functionalities, and to provide quick feedback to the team and stakeholders on the “true” state of the product on all supported platforms for VIU.
In this talk, using the above set context, I will be sharing the following:
The automation strategy
Chosen tech-stack
How (and why) no cloud-based solution worked for me
Implementation details - MAD-LAB - which arose from the learnings of the failed experiments - which resulted in setting up my own real-device in-house lab.
The core implementation (code) of MAD-LAB (already open-sourced)

Published in: Software
  • Be the first to comment

Build your own MAD-LAB - for Mobile Test Automation

  1. 1. MAD-LAB Mobile Automation Devices LAB Anand Bagmar Software Quality Evangelist
  2. 2. About Me @BagmarAnand Blog – essenceoftesting about.me/anand.bagmar
  3. 3. What do you expect from this session?
  4. 4. Context
  5. 5. Ecosystem Platforms • Android (> v4.4.2) • iPhones (> v5) • Desktop / WAP Regions • Indonesia • Malaysia • India • Middle-East • ...
  6. 6. Dynamic Content
  7. 7. Search
  8. 8. Download
  9. 9. Offers
  10. 10. ADs
  11. 11. What this means ... Reality! • Unique installs (Android) - > 1.75M • Unique # of devices (Android) - > 13K
  12. 12. What this means ... Product! • One product, One experience • Unique / Regional / Local content • User detection & offers based on origin • Region • Carrier network • ...
  13. 13. What this means ... Testing! • Video Quality, Dynamic Content, Download Expiry • Simulating Carriers / Networks • Consumer focus! • Experience • Device / OS combinations
  14. 14. Business Requirements • Provide visibility into - • What is Tested • Overall Quality of Product
  15. 15. APIs Payment partners CMS .... .... Architecture
  16. 16. APIs Payment partners CMS .... .... Where are these?
  17. 17. How would you approach Automation for such a product?
  18. 18. What Metrics make sense for determining Quality?
  19. 19. Automation Strategy
  20. 20. Automation Criteria • E2E tests should focus on Business Rules & User Scenarios • Reports should reflect the above • Understood by all Business & Technical stakeholders • Generated automatically • Should run frequently • Every new change in the product-under-test • On a variety of device / OS combinations
  21. 21. What to Automate? Validate what was working, continues to work! • Identify user journeys / scenarios • Prioritize it!
  22. 22. Run Automation against... Usage patterns from Analytics / PlayStore / App Store • Identify devices that matter • Identify devices that are “automation-friendly”
  23. 23. Automation Tech Stack • Cucumber-JVM à For specifying Business Rules • Appium à To allow integration with Android / iOS • Gradle / Groovy à As build tool scripting language • Jenkins à Continuous Integration Server • TTA - Test Trend Analyzer à Trend & Failure Analysis
  24. 24. MVP / POC
  25. 25. Scale & Repeat
  26. 26. Experiment, Learn & Evolve
  27. 27. Experiments • Run tests on Emulators • Use Cloud-based services (Local Vs Remote) • SauceLabs • TestObject • Amazon Device Farm • pCloudy • Devices
  28. 28. Some Experiments didn’t work • Emulators • Video Player • Location • Cloud-based services • Location • Tech Stack • Devices
  29. 29. MAD-LAB
  30. 30. MAD-LAB – Mobile Automation Devices LAB
  31. 31. Innovations in MAD-LAB • Device management (selection, cleanup, app install and uninstall) • Parallel test execution (at Cucumber scenario level) - maximising device utilisation) • Appium server management • Adb utilities (periodic ADB server disconnects) • Custom reporting using cucumber-reports • Video recording of each scenario and embedding in the custom reports • Analytics Automation
  32. 32. Conclusion
  33. 33. Learnings (so far) Blue-print is important! • Stepping-stone approach Experiment! • Know your objective(s) • Fail-fast! Make the Product Testable!!
  34. 34. References • Sample code • https://github.com/anandbagmar/cucumber-jvm-appium-infra • Blog posts • https://essenceoftesting.blogspot.com/search/label/madlab • TTA (Test Trend Analyzer) • https://essenceoftesting.blogspot.com/search/label/tta
  35. 35. Thank you! @BagmarAnand Blog – essenceoftesting about.me/anand.bagmar

×