This document summarizes eBay's approach to mobile testing and test automation. It discusses the challenges of testing across different mobile devices and localizations. It outlines requirements for mobile test automation, including parallel execution and support for multiple apps, languages, and devices. It then describes eBay's test automation of the mobile web using Selenium, and their custom open source projects for automating tests of Android and iOS native apps at scale. These projects integrate mobile testing tools like Calabash and UI Automation into the Selenium grid for a consistent test automation approach across platforms.

1. Mobile Testing and Mobile
Automation at eBay
Michael Palotas & Dominik Dary
Quality Engineering Europe
eBay International AG
September 2012

2. eBay Inc. Facts
Founded in 1995
Based in San Jose, California
27770 Employees worldwide

3. Mobile Facts
Mobile Facts: Mobile Fun Facts:
100,000,000 downloads Every 30 seconds, a woman’s
handbag is purchased on eBay
of eBay Inc’s apps Mobile.
Every 2 minutes, a tablet is
Mobile transaction volume bought through eBay Mobile.
will double in 2012 to 10
billion USD Every week, more than 8,000
cars and 340,000 car parts are
1,7 million items sold through eBay’s mobile
apps.
are listed with using mobile
7,493 tablets sold weekly from
apps every week April – June;
enough to cover the surface of
3 Olympic-sized pools.

4. Who we are
Michael Palotas
Head of Quality Engineering Europe
E-mail: mpalotas@ebay.com
Dominik Dary
Staff Software Engineer in Test
E-mail: ddary@ebay.com

5. Agenda
1. Challenges in Mobile Testing
2. Manual Mobile Testing
3. Our Requirements for Mobile Test
Automation
4. Test Automation of the Mobile
Web, Android Apps & iOS Apps
5. Conclusion

6. First Challenge: Localization

7. Second Challenge: Device Diversity

8. Devices vs. Emulators

9. Manual Mobile Testing
Feature testing is manual
Concentrate on top 5 devices
Functionality, Layout, User
Experience and Localization
In depth EU specific domain
knowledge

10. The importance of parallel Test Execution
• Multiple Apps
• 12 European Sites
• iOS Devices
• iPhone vs. iPad
• Version
• Android Devices
• Manufactures
• Screen Sizes
• OS Modifications
Picture Source: http://www.flickr.com/photos/ippei-janine/483345027/

11. Technical High Level Overview of the Web Tests

12. Mobile Test Automation Requirements
• Use the Load Balancer Features of the Selenium
Grid2 to execute the End-to-End Tests in parallel
• Manage multiple applications / versions / languages
• The Mobile App should not need to be modified
(e.g. like adding a test server)
• Mobile App Runtime Inspection Support

13. Testing of eBay‘s Mobile Products
Android Mobile Web Android Native App iOS Native App

14. Test Automation of the Mobile Web
We use Selenium and specifically
the AndroidDriver and the
iPhoneDriver
Test Cases are written in Java and
are similar to web tests
Tests are only executed on
Emulators
http://code.google.com/p/selenium/wiki/IPhoneDriver
http://code.google.com/p/selenium/wiki/AndroidDriver

15. Android Mobile Web Test Infrastructure
TestNG
Test
Selenium Grid http
http Jetty Server
Hub
Android
Driver
WebView
Android Server
QE Machine Backend Device

16. Page Tests
Flow Objects
Objects Mobile Web Test Case

17. Test Automation of the Android Apps
We have created the Open Source
project Calabash-Driver
Calabash-Driver integrates
Calabash-Android into the Selenium
Grid
Calabash-Driver offers a convenient
Webdriver like Java API for writing
tests http://calabash-driver.github.com

18. Test Automation of Android Apps
Selenium Grid
TestNG Hub
Test
http
http
Instrumentation
http Backend
Calabash
Calabash Android
Jetty Server
Driver Connecto
r
eBay App
ADB
Connec- USB
tion
QE Machine Device
Calabash Server
Backend

19. Dialog Tests
Flow Objects
Objects Android App Test Case

20. Test Automation of the iOS Apps
We have created the Open Source
project iOS-Driver
iOS-Driver integrates Apple’s official
tool UI Automation into the Selenium
Grid
iOS-Driver offers a convenient
Webdriver like Java API for writing
tests https://github.com/freynaud/ios-driver

21. Test Automation of iOS Apps
Selenium Grid
TestNG Hub
Test
http
http
Instru-
iOS eBay App
Jetty Server
ments
Driver Manager
Apple‘s UI
Automa-
tion
QE Machine iOS Simulator
iOS-Server
Backend

22. Dialog Flow Tests
Objects Objects iOS App Test Case

23. Demo

24. Advantages of Calabash-Driver & iOS-Driver
Solutions are integrated into the Selenium
Grid which enables us to scale
The concept of automated testing the Web,
Win32, mobile Web and native apps is the
same
Fast Rampup for the testers and quick
return on investment
Both Solutions are supporting the
interaction with elements based on the
L10n resource bundle key

25. Fork us at Github
http://calabash-driver.github.com
https://github.com/freynaud/ios-driver

26. Conclusion
No single automation solution for mobile
Web, Android and iOS apps – and that’s ok!
Invest into Scalability by Virtualization
Buy the top devices
Open source is your best friend!
Interested in our solutions? Contribute to our projects!

27. Thank You!
Questions or Comments?