Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Build a Large Scale In-House Test Lab for Mobile Apps

3,801 views

Published on

Watch a live presentation at http://offer.bitbar.com/build-a-large-scale-in-house-test-lab-for-mobile-apps

Mobile app development relies on agile methodology pretty well. This means companies building their apps and game for Android and iOS are heavily using agile methods in their daily doings, but also in technology that enables them to do daily builds and enable instant testing as part of the development. This webinar focus on in-house test labs and explains how this sort of environment can be built.

Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/

Published in: Technology
  • Be the first to comment

Build a Large Scale In-House Test Lab for Mobile Apps

  1. 1. Building a Large-Scale In-House Test Lab for Mobile Apps 4 June 2014 Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com
  2. 2. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 2 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  3. 3. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 3 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  4. 4. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 4 Manual vs. Automation Smaller coverage, More money burnt & time wasted, Error- prone Large coverage, quickly completed, Less money & time wasted, Exact results.
  5. 5. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 5 Prerequisites • Testing environment as authentic as possible – Real devices <-> emulators/simulators • Test coverage needs to be adequate – Software is not identical in all Android/iOS devices – Hardware is even more fragmented landscape • Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes
  6. 6. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 6 In-House Mobile App Testing • Continuous Testing with real devices • Cloud/Out-of-Premises vs. In-House • Improve app ratings & your brand reputation • Operational costs vs. Lost revenue? • Consequences of not-being-tested-enough?
  7. 7. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 7 Reasons to Build In-House Labs • Pre-Market devices or other hardware • Corporate security policies • Availability and usage level • Special setups
  8. 8. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 8 Core Benefits of CI and Testing • Version control and build revision • Build automation, instant execution & testing • All types of testing & integrations enabled • Frequent commits, Code consolidation -> Fast builds • CI/Testing is the best fit for mobile agile process
  9. 9. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 9 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  10. 10. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 10 Why Apps Fail?
  11. 11. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 11 EXAMPLE: Android devices Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
  12. 12. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 12 EXAMPLE: Android OEMs Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
  13. 13. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 13 EXAMPLE: Android OS Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
  14. 14. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 14 OS Versions ICS JB KK 17.7M test runs, 288 unique Android devices, 50+ configurations
  15. 15. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 15 Display Resolution 17.7M test runs, 288 unique Android devices, 50+ configurations
  16. 16. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 16 Memory
  17. 17. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 17 Coverage? IS THAT ALL YOU NEED? Picture by Double Encore - http://www.doubleencore.com/2014/05/android-user-data-2014/
  18. 18. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 18 Data, results & setup information were gather between 1 January – 31 March 2014 (Q1’2014) - 17.7M test/device runs - 288 unique devices - Thousands of userswww.testdroid.com Research
  19. 19. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 19 Global Coverage
  20. 20. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 20 Robust/Unstable – Q1’14 The Most Robust Devices Samsung Infuse 4G SGH-I997 Lenovo A60 Dell Streak 7 M02M001 HTC ChaCha A810e Coolpad 7295 Samsung Galaxy Europa GT-I5500 Samsung Acclaim SCH-R880 Motorola Motosmart XT389 Sharp Aquos Phone Zeta SH-02E Samsung Galaxy mini GT-S5570 The Most Unstable Devices Samsung Galaxy S IV Active SGH-I537 Sony Xperia J ST26i Samsung Galaxy S IV Altius SCH-R970 Samsung Galaxy S IV Altius SGH-M919 Samsung Galaxy Note 8.0 GT-N5110 Samsung Galaxy S IV Altius SGH-I337 Samsung Galaxy Mega GT-I9205 HTC One M7 4.3 Asus Transformer Pad TF300TL Sony Xperia Acro S LT26w 99.805 % 99.797% 99.794% 99.785% 99.783% 99.780% 99.778% 99.775% 99.771% 99.770% 29.722% 36.376% 39.007% 41.455% 43.284% 44.737% 45.536% 45.794% 47.689% 50.576 %
  21. 21. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 21 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  22. 22. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 22 Hardware – Top 3 Things • Device Control Servers – Hard disks – RAM – Energy Efficiency • USB Hubs • WiFi Infrastructure
  23. 23. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 23 Software – Top 3 Things • Testdroid Enterprise • Continuous Integration – e.g. Jenkins CI • Other agile tools
  24. 24. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 24 How Testdroid Helps? • Speed up dev & testing cycles significantly • Agile development & integration with mobile practices • Continuous Testing
  25. 25. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 25 Simple Setup Device clusters Device Control Server(s) Cluster #1 Cluster #2 Cluster #3 USB Hubs, Cables Cluster #4 Cluster #5
  26. 26. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 26 Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Commit Users / Consumers
  27. 27. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 27 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  28. 28. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 28 1
  29. 29. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 29 2
  30. 30. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 30 3
  31. 31. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 31 4
  32. 32. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 32 5
  33. 33. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 33 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  34. 34. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 34 Testdroid Update
  35. 35. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 35 Testdroid Update
  36. 36. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 36 Stay tuned! A Step-by-step guide how to build an in- house test lab for mobile apps. Will be available at www.testdroid.com Coming Soon!
  37. 37. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 37 Deployment Options
  38. 38. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 38 Only $999! For Trial Purpose! We’re giving a 30-day trial license with Mac Mini. In worst case, you got a Mac Mini with $999 – and it the best scenario you’ll have a product that will save you a lot of money, time and results as a better application!
  39. 39. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 39 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
  40. 40. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 40

×