This document summarizes the trials and tribulations of managing a large-scale mobile device test automation lab at Yahoo. It discusses why mobile testing on real devices is important, Yahoo's CI/CD pipeline for mobile apps, and details of Yahoo's mobile device testing lab including supported frameworks and over 300 iOS and Android devices. It also outlines some of the challenges of running such a lab at scale including hardware issues, understanding user needs, operational processes, and keeping teams engaged over time.

1. Mobile Device Test Automation
The trials and tribulations of managing a mobile device test automation lab at scale
Heemeng Foo, Senior Manager, Engineering Services (Mobile Excellence), email: heemeng@yahoo-inc.com
1

2. Mobile Testing @ Yahoo
2

3. Why test on real mobile devices?
 Simulators/Emulators cover only functional aspects
and idealized hardware
 What’s missing are h/w related issues eg. screen
size, sensors, SoC, GPU, and how they work
together
 “Bloatware” introduced by OEMs
 Devices with low end hardware eg. iPod Touch, Moto
E reveal issues you don’t see on flagship devices
3

4. CI / CD @ Yahoo
 Cloud based CI/CD pipeline for building
iOS/Android/Java/Node/Python apps
 Available to any developer, product manager, designer, or team
 Anyone is able to create a build runway with a “simple” config file
 Runs 24/7/365
 Mobile build runway: Git -> Build -> Test -> Dogfood
 http://spectrum.ieee.org/view-from-the-
valley/computing/software/yahoos-engineers-move-to-coding-
without-a-net (bit.ly/yahoo-build)
4

5. Mobile Device Testing @ Yahoo
 We have ~ 60 iOS/Android apps
 Automation: Mobile Device Lab hooked up to CI/CD pipeline
 Manual: Mobile Device Library open 24/7/365, badge access
 Device Lab in operation since 2013
● 300+ iOS/Android devices, > 1.5mil test runs to date
● ~ 50 MacMini, ~ 40 Linux (bare iron / OpenStack VM)
● Test Frameworks supported:
● Robotium
● Espresso
● UI Automator (Android)
● UI Automation (iOS),
● XCUITest
● Appium5

6. Why support so many frameworks?
 Different teams have different mix of devs/test engineers
 Dev heavy
● frameworks integrated with dev env eg. Espresso, UI Automation
● frameworks in dev language eg. Espresso (Java), XCUITest (objC, Swift)
● Core principle: any test automation should be runnable locally
 Test engineering covering API/Web/iOS/Android – Appium
6

7. Some Eye Candy
7

8. Some Eye Candy
8

9. Disclaimer
 We didn’t build it all - Initial platform was licensed
 I have a great team and support
 As a company, Yahoo has invested heavily in CI/CD and developer
productivity
 Great developer community in the company
9

10. “Trials and Tribulations”
Challenges and Lessons running a Mobile
Device Test Automation Lab
10

11. Challenges / Lessons
 It’s a production system
 Hardware is not software
 Understanding the user
 Operationalize, Operationalize, Operationalize
 Keeping team engaged
1
1

12. It’s a Production System
 Monitoring, bake-in testing, deployment windows all apply
 Canary in the coal mine
 Operations comms - users should not be surprised
 Regular internal end-to-end tests
 Health checks and redundancy
 Splunk (or ELK) is very useful
 Lots of planning needed for major system updates to maintain
uptime
1
2

13. Hardware is not software
 Batteries explode
 iOS - once upgrade, can’t downgrade
 iPhone pre-orders
 iOS upgrade popups
 Mobile hardware moves fast - follow blogs
 WiFi AP Density
1
3

14. Understanding the user
 Every mobile test engineer, Device Lab DevOps engineer knows
how to build a simple iOS/Android app
 End-to-end canary apps: Git -> Build -> Test
 Get involved in a project (and actually use your system)
 Reference implementations speak louder than documentation ie.
point devs to working code
1
4

15. Operationalize, Operationalize, Operationalize
 Runbooks / SOPs
 Maps
 Tools to test modules in isolation
 Tools to test devices in isolation eg. Remote screencapture
 Automate as much as possible (but just enough)
1
5

16. Keeping team engaged
 Interesting (Cool) projects eg. IoT, Robots
 Marketable skills eg. Splunk, Chef, Jenkins training
1
6

17. Conclusion
17

18. Vendor Options
 SauceLabs – limited to Appium
 MobileLabs
 Experitest
 SOASTA
 Perfecto Mobile
 DevicePass by BeyondSoft
1
8

19. Worth it?
1
9

20. Q&A
20

21. http://bit.ly/heemengSlides2016