Slides from my talk at AnDevCon Boston conference about building automated tests for Android Apps. Comparing Open Source, Commercial tools; comparing tests as code versus record and playback.

1. © 2015 SOASTA. All rights reserved.
July 31, 2015
1

2. © 2015 SOASTA. All rights reserved.
July 31, 2015
2
o Please connect to the shared Google Drive:
http://soasta.io/AnDevConFiles
o Open the Setup document and do the steps

3. © 2015 SOASTA. All rights reserved.
July 31, 2015
3
o Why Automation?
o Test Automation Products
o Getting Started
o Creating Test Scripts from Code
o Creating Test Scripts by Recording
o Running Automated Test Scripts
• On local devices, in cloud devices
o Setting Up Continuous Integration

4. © 2015 SOASTA. All rights reserved.
July 31, 2015
4
o Improved testing efficiency
o Consistent and repeatable testing process
o Improved regression testing coverage
o More tests can be run in less time
o Run 24x7
o Humans free to perform advanced manual tests
o Easy to reproduce defects

5. © 2015 SOASTA. All rights reserved.
July 31, 2015
5
o Survey Question 1:
• What kind of testing are you doing now?
• Doing some automated testing with manual
• Doing mostly automated testing with manual
• Doing only automated testing
o Survey Question 2:
• How are you automating your tests?
• Developers write tests as code
• QA team records tests from manual scripts
o Survey Question 3:
• Are you satisfied with your test automation process?
• Yes. Why are you here?

6. © 2015 SOASTA. All rights reserved.
July 31, 2015
6
o Applause
o Ranorex
o Appium/Selenium
o Robotium
o Perfecto Mobile
o Device Anywhere
o SOASTA TouchTest
o AWS Remote Device Cloud
o MobileLabs
o mAutomate
o Experitest (formerly SeeTest)
o SmartBear Test Complete
o NTT AppKitBox Remote Test
Kit
o MonkeyTalk

7. © 2015 SOASTA. All rights reserved.
July 31, 2015
7
o Is your QA team good at manual testing or writing test code in a
high-level language (Java, JavaScript, etc.)?
o Do you want one tool for Android and iOS?
o Does your company prefer open source (free) or commercial
products?
o Do you want tools that install locally or are SaaS/cloud-based?
o Is your team located together or multi-site?

8. © 2015 SOASTA. All rights reserved.
July 31, 2015
8
Product
Company
Where
License
Appium
OpenSource
(SauceLabs)
Desktop or Server
(Mac, Win, Linux)
OpenSource
TouchTest
SOASTA
Cloud
Commercial
TestComplete
SmartBear
Windows
Commercial
Ranorex
Ranorex
Windows
Commercial
DeviceAnywhere
Keynote
Windows
Commercial

9. © 2015 SOASTA. All rights reserved.
July 31, 2015
9
Service
Support
AWS Remote Device Cloud
New – Android, FireOS
Hundreds of devices
Appium, Calabash, UIAutomator
SauceLabs
Android simulators, devices (beta)
Appium, manual
NTTR AppKit
Android, iOS
Hundreds of devices
Any automation, manual
Keynote Device Anywhere
Android, iOS
Hundreds of devices
Keynote, Selenium
MobileLabs
Android, iOS
Your own devices

10. © 2015 SOASTA. All rights reserved.
July 31, 2015
10
o Record and Playback
• Easiest approach to start mobile test automation
• Script parameterization
o Code-based

11. © 2015 SOASTA. All rights reserved.
July 31, 2015
24
o How many releases of Android [GooglePlay]
o How many device types – most popular
o Emulators
• One per system; cloud hosted e.g. SauceLabs
• Not the same as running on hardware
o Real devices
• Real experience
• Costly, hard to get

12. © 2015 SOASTA. All rights reserved.
July 31, 2015
25
o

13. © 2015 SOASTA. All rights reserved.
July 31, 2015
26

14. © 2015 SOASTA. All rights reserved.
July 31, 2015
27
o Real User Monitoring – collect user metrics
o mPulse
o Crittercism
o Flurry
o …

15. © 2015 SOASTA. All rights reserved.
July 31, 2015
28
o Desktop: direct connect devices to your desktop computer
o Lab: set up devices in a lab with automation
o Device Cloud
• On-premise device cloud: your own devices set up for access by anyone at
your company. Examples: Mobile Labs, Device Anywhere
• Public cloud: publicly hosted devices accessible on a pay-as-you-go model.
Examples: Device Anywhere, NTT Remote Test Kit, SauceLabs, AWS Device
Farm
• Private cloud: Reserved devices hosted by third-party. Examples: NTT Remote
Test Kit, Device Anywhere

16. © 2015 SOASTA. All rights reserved.
July 31, 2015
29
o Record and Playback
o Add screenshot validation
o Add object validation
o Run on a new device
o Adjust for new devices
o Add data input from seed data
o Validate seed data

17. © 2015 SOASTA. All rights reserved.
July 31, 2015
30
o Download from Google drive
http://soasta.io/AnDevConFiles
o Install onto Android device

18. © 2015 SOASTA. All rights reserved.
July 31, 2015
31
o Log into http://soasta.com/touchtest/sign-up

19. © 2015 SOASTA. All rights reserved.
July 31, 2015
32
o Demo

20. © 2015 SOASTA. All rights reserved.
July 31, 2015
33
o Create a Composition to hold the Clip (test script)
o Can easily duplicate a clip for multiple devices
o Hit Play button
o Watch test proceed, look for failures
• Debugging
o Other capture test data: CPU, Memory, Battery, Bytes I/O

21. © 2015 SOASTA. All rights reserved.
July 31, 2015
34
o Remote device can be anywhere
• On someone’s desk
• In a lab
• In a remote device cloud
o Using NTTR Remote Test Kit
• Get a device
• Install an app
• TouchTest Agent
• App to be tested
• Launch TouchTest Agent
• Use API for automation

22. © 2015 SOASTA. All rights reserved.
July 31, 2015
35
o Scommand: command line interface to TouchTest server
• Start a test job
• Get test results
• More…
o Also available as a Jenkins Plug-in
• Click Manage Jenkins
• Click Manage Plug-Ins
• Click Available
• Look for SOASTA CloudTest
• Install

23. © 2015 SOASTA. All rights reserved.
July 31, 2015
36
o Emma