Build a Large Scale In-House Test Lab for Mobile Apps

Building a Large-Scale
In-House Test Lab for
Mobile Apps
4 June 2014
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com

© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
2
Agenda
• Intro & Benefits: In-House Mobile App Testing
• Challenges w/Apps, Devices, Testing & Market
• Infrastructure – both Hardware & Software
• Tips & Tricks for Building In-House Lab
• Testdroid Update
• Q&A
Agenda

3
Agenda
• Q&A
Agenda

4
Manual vs. Automation
Smaller coverage, More money
burnt & time wasted, Error-
prone
Large coverage,
quickly
completed, Less
money & time
wasted, Exact
results.

5
Prerequisites
• Testing environment as authentic as possible
– Real devices <-> emulators/simulators
• Test coverage needs to be adequate
– Software is not identical in all Android/iOS devices
– Hardware is even more fragmented landscape
• Surrounding testing conditions (e.g. network)
• Try not to build in repetitive, manual processes

6
In-House Mobile App Testing
• Continuous Testing with real devices
• Cloud/Out-of-Premises vs. In-House
• Improve app ratings & your brand reputation
• Operational costs vs. Lost revenue?
• Consequences of not-being-tested-enough?

7
Reasons to Build In-House Labs
• Pre-Market devices or other hardware
• Corporate security policies
• Availability and usage level
• Special setups

8
Core Benefits of CI and Testing
• Version control and build revision
• Build automation, instant execution & testing
• All types of testing & integrations enabled
• Frequent commits, Code consolidation -> Fast builds
• CI/Testing is the best fit for mobile agile process

9
Agenda
• Q&A
Agenda

10
Why Apps Fail?

11
EXAMPLE: Android devices
Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/

12
EXAMPLE: Android OEMs

13
EXAMPLE: Android OS

14
OS Versions
ICS JB KK
17.7M test runs, 288 unique
Android devices, 50+
configurations

15
Display Resolution
17.7M test runs, 288 unique
Android devices, 50+
configurations

16
Memory

17
Coverage? IS THAT ALL YOU NEED?
Picture by Double Encore - http://www.doubleencore.com/2014/05/android-user-data-2014/

18
Data, results & setup
information were gather
between 1 January – 31
March 2014 (Q1’2014)
- 17.7M test/device runs
- 288 unique devices
- Thousands of userswww.testdroid.com
Research

19
Global Coverage

20
Robust/Unstable – Q1’14
The Most Robust Devices
Samsung Infuse 4G SGH-I997
Lenovo A60
Dell Streak 7 M02M001
HTC ChaCha A810e
Coolpad 7295
Samsung Galaxy Europa GT-I5500
Samsung Acclaim SCH-R880
Motorola Motosmart XT389
Sharp Aquos Phone Zeta SH-02E
Samsung Galaxy mini GT-S5570
The Most Unstable Devices
Samsung Galaxy S IV Active SGH-I537
Sony Xperia J ST26i
Samsung Galaxy S IV Altius SCH-R970
Samsung Galaxy S IV Altius SGH-M919
Samsung Galaxy Note 8.0 GT-N5110
Samsung Galaxy S IV Altius SGH-I337
Samsung Galaxy Mega GT-I9205
HTC One M7 4.3
Asus Transformer Pad TF300TL
Sony Xperia Acro S LT26w
99.805 %
99.797%
99.794%
99.785%
99.783%
99.780%
99.778%
99.775%
99.771%
99.770%
29.722%
36.376%
39.007%
41.455%
43.284%
44.737%
45.536%
45.794%
47.689%
50.576 %

21
Agenda
• Q&A
Agenda

22
Hardware – Top 3 Things
• Device Control Servers
– Hard disks
– RAM
– Energy Efficiency
• USB Hubs
• WiFi Infrastructure

23
Software – Top 3 Things
• Testdroid
Enterprise
• Continuous
Integration – e.g.
Jenkins CI
• Other agile tools

24
How Testdroid Helps?
• Speed up dev &
testing cycles
significantly
• Agile development
& integration with
mobile practices
• Continuous Testing

25
Simple Setup
Device clusters
Device Control
Server(s)
Cluster #1 Cluster #2 Cluster #3
USB Hubs,
Cables
Cluster #4 Cluster #5

26
Developers
Source
code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Commit
Users / Consumers

27
Agenda
• Q&A
Agenda

28
1

29
2

30
3

31
4

32
5

33
Agenda
• Q&A
Agenda

34
Testdroid Update

35
Testdroid Update

36
Stay tuned!
A Step-by-step guide
how to build an in-
house test lab for
mobile apps.
Will be available at
www.testdroid.com
Coming Soon!

37
Deployment Options

38
Only $999!
For Trial Purpose!
We’re giving a 30-day trial license with Mac Mini.
In worst case, you got a Mac Mini with $999 – and it the best scenario you’ll have a
product that will save you a lot of money, time and results as a better application!

39
Agenda
• Q&A
Agenda

40

Build a Large Scale In-House Test Lab for Mobile Apps

More Related Content

What's hot

Similar to Build a Large Scale In-House Test Lab for Mobile Apps

Recently uploaded

Build a Large Scale In-House Test Lab for Mobile Apps