Main advantages and disadvantages of Aws device farm, Bitbar, Kobiton and Saucelabs. Examples of how to configure driver and desired capabilities. Parallelization.

1. Overview
for device farms
Anna Klueva
annaklueva888@gmail.com

2. Anna Klueva
Position: Automation QA Engineer at Grid Dynamics
Experience: 5 years & 2 years in mobile automation
Email: annaklueva888@gmail.com

3. Agenda
● Why cloud?
● Overview of device farms:
○ Kobiton
○ Saucelabs
○ AWS Device farm
○ Bitbar
● Parallelization
● Summary
3

4. 4
Emulator Real device
vs.

5. 5
Why cloud?

6. 6

7. Privileged and Confidential 7
Kobiton: Pros & Cons
● Easy of use
● Easy of setup
● Reporting: no affect on custom
reports
● Cheap
Pros
● Supports ONLY Appium
● Appium version can't be
changed
Cons

8. 8
Kobiton: How it works?
1) Upload app to Kobiton
2) Configure Driver and desired
capabilities in your project
3) Run test on local machine
Example of configuration:
String APPIUM_URL =
"https://username:apiKey@api.kobiton.com";
driver = new AndroidDriver<MobileElement>(
new URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());
private static DesiredCapabilities getCapabilities() {
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("deviceName", “Galaxy Note3”);
capabilities.setCapability("platformVersion", “6”);
capabilities.setCapability("platformName", "Android");
capabilities.setCapability("app", pathToRemoteApp);
return desiredCapabilities;
}

9. 9

10. 10
Upload App to repo
Get info about devices
Get device status
Get logs info
Kobiton: Api description

11. 11
Kobiton Api
Kobiton: How CI/CD process can be organized?

12. 12
1) Health-check for your application
2) Supports different build tools: Maven, Gradle
Kobiton: Specific feature

13. 13

14. 14
Saucelabs: Pros & Cons
● Support different frameworks
● Easy of use
● Easy of setup
● Reporting: no affect on custom
reports
Pros
● Not cheap
Cons

15. 15
1) Upload app via Api
2) Configure driver and
DesiredCapabilities
3) Run tests
Saucelabs: How it works?
String APPIUM_URL = “https://eu1.appium.testobject.com”;
driver = new AndroidDriver<MobileElement>(new
URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());
private static DesiredCapabilities getCapabilities() {
DesiredCapabilities desiredCapabilities = new
DesiredCapabilities();
desiredCapabilities.setCapability("platformName","Android");
desiredCapabilities.setCapability("testobject_api_key",
SAUCELABS_API_KEY);
desiredCapabilities.setCapability("testobject_app_id",
SAUCELABS_UPLOADED_APP_ID);
return desiredCapabilities;
}

16. 16

17. 17
Saucelabs (TestObject) provides:
Saucelabs: Api
Storage API (upload app)
Test Reports API
Logs API
Devices API
Appium Suites API

18. 18
Two ways:
1) Sauce plugins for Jenkins, TeamCity, Bamboo
2) API
Saucelabs: How to integrate with CI/CD flow?

19. 19
Automated Quality Report
Tests included in the Quality Report are:
● Install and Launch tests
● Stress tests
Saucelabs: Specific feature

20. 20
AWS Device farm

21. 21
AWS Device farm: Pros & Cons
● Supports different frameworks
● Ability to test specific cases
● Rich test analytics
Pros
● No local run
● No custom reports, just aws
reports
Cons

22. 22
1) Your project with tests:
1. Configure Driver and DesiredCapabilities
2. Archive project
2) AWS: Upload application to AWS
Device farm
3) AWS: Upload zipped project to aws
5) AWS: Select device and run type
6) AWS: Run test on server
AWS Device farm: How it works?
Example:
String APPIUM_URL = “http://127.0.0.1:4723”;
driver = new AndroidDriver<MobileElement>(new
URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());
/*** We do not need to and SHOULD NOT set our own
DesiredCapabilities* Device Farm creates custom settings at the
server level. Setting your own DesiredCapabilities will result in
unexpected results and failures.**/
private static DesiredCapabilities getCapabilities() {
return new DesiredCapabilities();
}

23. 23

24. 24
Upload app
Upload package with tests
Get info about devices
Get info about user
Run tests on AWS Device cloud
AWS device farm: API description

25. 25
Two ways:
1) Jenkins plugin
“aws-device-farm”
2) API
AWS Device farm: How to integrate with CI/CD flow?

26. 26
1) Build tools: Maven, Gradle
2) AWS Device farm client application
AWS Device farm: Specific feature

27. 27

28. 28
Bitbar: Pros & Cons
● Supports different frameworks
● Easy of use
● Rich test analytics
● Integration with third party
systems
Pros
● Not easy of setup
● Not cheap
Cons

29. 29
Client-side: run tests locally on remote real devices
Server-side: run tests on Bitbar server on remote real device
Bitbar: How it works?

30. 30
1) Upload app to Bitbar server
2) Configure Driver and
DesiredCapabilities
3) Bitbar: create project
4) Run tests
Bitbar: client-side run
Example of configuration:
String APPIUM_URL = “https://appium.bitbar.com”;
driver = new AndroidDriver<MobileElement>(new URL(APPIUM_URL+"/wd/hub"),
Configuration.getCapabilities());
private static DesiredCapabilities getCapabilities() {
DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
desiredCapabilities.setCapability("platformName","Android");
desiredCapabilities.setCapability("deviceName", "Android Device");
desiredCapabilities.setCapability("testdroid_apiKey", BITBAR_API_KEY);
desiredCapabilities.setCapability("testdroid_project", BITBAR_PROJECT);
desiredCapabilities.setCapability("testdroid_testrun", "Demo1");
desiredCapabilities.setCapability("testdroid_app",UPLOADED_APP_PATH);
desiredCapabilities.setCapability("testdroid_device","LG Google Nexus 5");
desiredCapabilities.setCapability("testdroid_target", "Android");
return desiredCapabilities;
}

31. 31
1) Upload app to server
2) Configure Driver and Desired
Capabilities
3) Archive and upload your tests to
Bitbar server
4) Select devices
5) Run tests on server
Bitbar: server-side run
Example:
String APPIUM_URL = “http://127.0.0.1:4723”;
driver = new AndroidDriver<MobileElement>(new
URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());
/** app - don't change path to the .apk and .apk name. Bitbar stores you
uploaded app application.apk*/
private static DesiredCapabilities getCapabilities() {
DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
desiredCapabilities.setCapability("platformName","Android");
desiredCapabilities.setCapability("deviceName", "Android Device");
desiredCapabilities.setCapability("app",
System.getProperty("user.dir") + "/application.apk");
return desiredCapabilities;
}

32. 32

33. 33
Bitbar API provides:
Bitbar: API description
Upload app
Get info about devices
Run tests and get results

34. 34
Two ways:
1) Jenkins plugin: “Bitbar Run-in-Cloud”
2) Bitbar API
Bitbar: How to integrate with CI/CD flow?

35. 35
● AI Testbot and AppCrawler
● Integration with JIRA, Slack, Hipchat
Bitbar: Specific feature

36. 36
Parallelization in Saucelabs:
Parallelization
Parallelization in Bitbar and AWS:
Set of tests Test cases
Result(Runs on real devices)
Test Test sessions/scripts
Results
(Runs on real devices)

37. 37
Let’s summarize
Kobiton Saucelabs AWS Device farm Bitbar
Easy of setup
Easy of use
Reporting tools
Documentation
Pricing 0. 10$ per minute 0. 18$ per minute 0. 17$ per minute 0. 19$ per minute
Parallel

38. 38
Supported frameworks
Kobiton Saucelabs AWS Device farm Bitbar
Appium
Calabash
Espresso
Robotium
XCUITest
KIF

39. 39
1) If you use third party apps (Google Account,
Apple ID), the personal data can not be deleted
2) Appium version is updated randomly
3) Documentation is not up-to-date
Tricky cases

40. Q & A

41. www.griddynamics.com
Thank you!