Building a Large-Scale
In-House Test Lab for
Mobile Apps
4 June 2014
Ville-Veikko Helppi
Technical Product Manager
ville-v...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
2
Agenda
• Intro & Benefits: In-House Mobile App Testin...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
3
Agenda
• Intro & Benefits: In-House Mobile App Testin...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
4
Manual vs. Automation
Smaller coverage, More money
bu...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
5
Prerequisites
• Testing environment as authentic as p...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
6
In-House Mobile App Testing
• Continuous Testing with...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
7
Reasons to Build In-House Labs
• Pre-Market devices o...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
8
Core Benefits of CI and Testing
• Version control and...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
9
Agenda
• Intro & Benefits: In-House Mobile App Testin...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
10
Why Apps Fail?
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
11
EXAMPLE: Android devices
Picture by OpenSignal 2013 ...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
12
EXAMPLE: Android OEMs
Picture by OpenSignal 2013 - h...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
13
EXAMPLE: Android OS
Picture by OpenSignal 2013 - htt...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
14
OS Versions
ICS JB KK
17.7M test runs, 288 unique
An...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
15
Display Resolution
17.7M test runs, 288 unique
Andro...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
16
Memory
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
17
Coverage? IS THAT ALL YOU NEED?
Picture by Double En...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
18
Data, results & setup
information were gather
betwee...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
19
Global Coverage
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
20
Robust/Unstable – Q1’14
The Most Robust Devices
Sams...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
21
Agenda
• Intro & Benefits: In-House Mobile App Testi...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
22
Hardware – Top 3 Things
• Device Control Servers
– H...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
23
Software – Top 3 Things
• Testdroid
Enterprise
• Con...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
24
How Testdroid Helps?
• Speed up dev &
testing cycles...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
25
Simple Setup
Device clusters
Device Control
Server(s...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
26
Developers
Source
code
Repository
Build
failed
Succe...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
27
Agenda
• Intro & Benefits: In-House Mobile App Testi...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
28
1
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
29
2
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
30
3
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
31
4
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
32
5
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
33
Agenda
• Intro & Benefits: In-House Mobile App Testi...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
34
Testdroid Update
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
35
Testdroid Update
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
36
Stay tuned!
A Step-by-step guide
how to build an in-...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
37
Deployment Options
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
38
Only $999!
For Trial Purpose!
We’re giving a 30-day ...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
39
Agenda
• Intro & Benefits: In-House Mobile App Testi...
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
40
Upcoming SlideShare
Loading in...5
×

Testdroid: Build a Large Scale In-House Test Lab for Mobile Apps

3,439
-1

Published on

Watch a live presentation at http://bit.ly/1qXApb0

Mobile app development relies on agile methodology pretty well. This means companies building their apps and game for Android and iOS are heavily using agile methods in their daily doings, but also in technology that enables them to do daily builds and enable instant testing as part of the development. This webinar focus on in-house test labs and explains how this sort of environment can be built.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,439
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
24
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Testdroid: Build a Large Scale In-House Test Lab for Mobile Apps

  1. 1. Building a Large-Scale In-House Test Lab for Mobile Apps 4 June 2014 Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com
  2. 2. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 2 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  3. 3. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 3 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  4. 4. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 4 Manual vs. Automation Smaller coverage, More money burnt & time wasted, Error- prone Large coverage, quickly completed, Less money & time wasted, Exact results.
  5. 5. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 5 Prerequisites • Testing environment as authentic as possible – Real devices <-> emulators/simulators • Test coverage needs to be adequate – Software is not identical in all Android/iOS devices – Hardware is even more fragmented landscape • Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes
  6. 6. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 6 In-House Mobile App Testing • Continuous Testing with real devices • Cloud/Out-of-Premises vs. In-House • Improve app ratings & your brand reputation • Operational costs vs. Lost revenue? • Consequences of not-being-tested-enough?
  7. 7. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 7 Reasons to Build In-House Labs • Pre-Market devices or other hardware • Corporate security policies • Availability and usage level • Special setups
  8. 8. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 8 Core Benefits of CI and Testing • Version control and build revision • Build automation, instant execution & testing • All types of testing & integrations enabled • Frequent commits, Code consolidation -> Fast builds • CI/Testing is the best fit for mobile agile process
  9. 9. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 9 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  10. 10. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 10 Why Apps Fail?
  11. 11. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 11 EXAMPLE: Android devices Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
  12. 12. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 12 EXAMPLE: Android OEMs Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
  13. 13. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 13 EXAMPLE: Android OS Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
  14. 14. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 14 OS Versions ICS JB KK 17.7M test runs, 288 unique Android devices, 50+ configurations
  15. 15. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 15 Display Resolution 17.7M test runs, 288 unique Android devices, 50+ configurations
  16. 16. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 16 Memory
  17. 17. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 17 Coverage? IS THAT ALL YOU NEED? Picture by Double Encore - http://www.doubleencore.com/2014/05/android-user-data-2014/
  18. 18. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 18 Data, results & setup information were gather between 1 January – 31 March 2014 (Q1’2014) - 17.7M test/device runs - 288 unique devices - Thousands of userswww.testdroid.com Research
  19. 19. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 19 Global Coverage
  20. 20. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 20 Robust/Unstable – Q1’14 The Most Robust Devices Samsung Infuse 4G SGH-I997 Lenovo A60 Dell Streak 7 M02M001 HTC ChaCha A810e Coolpad 7295 Samsung Galaxy Europa GT-I5500 Samsung Acclaim SCH-R880 Motorola Motosmart XT389 Sharp Aquos Phone Zeta SH-02E Samsung Galaxy mini GT-S5570 The Most Unstable Devices Samsung Galaxy S IV Active SGH-I537 Sony Xperia J ST26i Samsung Galaxy S IV Altius SCH-R970 Samsung Galaxy S IV Altius SGH-M919 Samsung Galaxy Note 8.0 GT-N5110 Samsung Galaxy S IV Altius SGH-I337 Samsung Galaxy Mega GT-I9205 HTC One M7 4.3 Asus Transformer Pad TF300TL Sony Xperia Acro S LT26w 99.805 % 99.797% 99.794% 99.785% 99.783% 99.780% 99.778% 99.775% 99.771% 99.770% 29.722% 36.376% 39.007% 41.455% 43.284% 44.737% 45.536% 45.794% 47.689% 50.576 %
  21. 21. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 21 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  22. 22. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 22 Hardware – Top 3 Things • Device Control Servers – Hard disks – RAM – Energy Efficiency • USB Hubs • WiFi Infrastructure
  23. 23. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 23 Software – Top 3 Things • Testdroid Enterprise • Continuous Integration – e.g. Jenkins CI • Other agile tools
  24. 24. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 24 How Testdroid Helps? • Speed up dev & testing cycles significantly • Agile development & integration with mobile practices • Continuous Testing
  25. 25. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 25 Simple Setup Device clusters Device Control Server(s) Cluster #1 Cluster #2 Cluster #3 USB Hubs, Cables Cluster #4 Cluster #5
  26. 26. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 26 Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Commit Users / Consumers
  27. 27. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 27 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  28. 28. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 28 1
  29. 29. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 29 2
  30. 30. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 30 3
  31. 31. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 31 4
  32. 32. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 32 5
  33. 33. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 33 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  34. 34. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 34 Testdroid Update
  35. 35. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 35 Testdroid Update
  36. 36. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 36 Stay tuned! A Step-by-step guide how to build an in- house test lab for mobile apps. Will be available at www.testdroid.com Coming Soon!
  37. 37. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 37 Deployment Options
  38. 38. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 38 Only $999! For Trial Purpose! We’re giving a 30-day trial license with Mac Mini. In worst case, you got a Mac Mini with $999 – and it the best scenario you’ll have a product that will save you a lot of money, time and results as a better application!
  39. 39. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 39 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  40. 40. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 40
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×