Maximize the Benefits from
Your Test Automation
Investment

14 November 2013
Ville-Veikko Helppi
Technical Product Manager...
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
Challenges in Today’s App Dev
Fact: Most typical application fails when tested
across different device variants used by co...
Why Do They Fail?

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

6
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
How to Test: Manual vs. Automation
• Costs & Assets
– Time, People, Infrastructure, Tools, Training

• App Development & T...
Manual

Automation
Large
coverage, quick
ly
completed, Less
money & time
wasted, Exact
results.

Smaller coverage, More
mo...
Improving Time-to-Market

Repository

Successful
build

Commit

Users / Consumers

Build
failed

Test Cases
Application
QA...
Tips to Improve the App Quality
• Automate generic things as much as you can
• During the development your app changes –
t...
Test Automation Will Deliver You…
• App Compatibility & Global coverage
Test requirements / Planning

Smoke testing

Regre...
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
Example: Cost of Manual Testing
• Test setup for each device – 5 minutes
• 100 test cases, each take about 15 seconds to
e...
Example: Cost of Test Automation
• Creation of test cases/scripts – 100 test cases running 15
minutes per each = 1500 minu...
Benefits with Testdroid Cloud
• Instant availability of hardware and no need to
buy any device by yourself
• Testing gets ...
Example: On-Premise Setup
• Creation of test cases/scripts – 100 test cases running 15
minutes per each = 1500 minutes (25...
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
Mobile Apps for Different Verticals
Games
Utilities & Tools
Video Streaming & Multimedia

Banking & Payment
Retail & Trave...
Mobile Apps for Different Verticals
Games
Utilities & Tools
Video Streaming & Multimedia

Banking & Payment
Retail & Trave...
Mobile Apps for Different Verticals
• Usability!
• Functional and
Utilities & Tools
Behavioral testing
Video Streaming & M...
Mobile Apps for Different Verticals
• Performance!
• Connectivity, robustness
Utilities & Tools
and durability
Video Strea...
Mobile Apps for Different Verticals
Games
Utilities & Tools
Video Streaming & Multimedia

Banking & Payment
Retail & Trave...
Mobile Apps for Different Verticals
Games
Utilities & Tools
Video Streaming & Multimedia

Banking & Payment
Retail & Trave...
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
Tip #1: Test early, Test often

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

27
Tip #2: Plan What to Automate
Time

MANUAL

Infrastructure
Tools

AUTOMATED

People

Training

© Copyrights by Bitbar Tech...
Tip #3: Use only Real Devices

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

29
Tip #4: Use Atomic Test Units

Test iteration #1

Test iteration #2

© Copyrights by Bitbar Technologies Ltd. 2013
All rig...
Tip #5: Create Hermetic Tests

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

31
Tip #6: Use All Possible Devices

Testdroid Cloud’s 250+ Android devices
= 93-95% global Android volumes!
© Copyrights by ...
Tip #7: Separate Apps and Tests

Your App

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

33
Tip #8: Output Everything to Logs

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

34
Tip #9: End-to-End Testing

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

35
Tip #10: Integrate Dev & Testing
Repository

Successful
build

Commit

Users / Consumers

Build
failed

Test Cases
Applica...
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
Testdroid Products
Complete Solution for Mobile Apps/Games Testing

© Copyrights by Bitbar Technologies Ltd. 2013
All righ...
Testdroid
– Because it is important to know what app testing costs!

© Copyrights by Bitbar Technologies Ltd. 2013
All rig...
Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improv...
© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

41
Upcoming SlideShare
Loading in …5
×

Testdroid: Maximize the benefits from your test automation investment

1,922
-1

Published on

Watch a live presentation at http://bit.ly/1nQMp9o

Test automation can actually turn your development to super agile, enable you to start making revenue sooner, improve the overall quality of your applications and process. This new, superior way of development and testing can make you very cost-effective, increase significantly your developer productivity and speed up time-to-market.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,922
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
21
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Testdroid: Maximize the benefits from your test automation investment

  1. 1. Maximize the Benefits from Your Test Automation Investment 14 November 2013 Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com 1
  2. 2. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 3
  3. 3. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 4
  4. 4. Challenges in Today’s App Dev Fact: Most typical application fails when tested across different device variants used by consumers • Failures: Both SW and HW cause failures • Emulators do not offer realistic platform for dev • Already installation of app fails – not tested at all • Application crashes, leaks memory, consumes too much resources, behaves not as meant to behave • UI scaling/display/resolution – common problem • Application performance is bad, draining battery © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 5
  5. 5. Why Do They Fail? © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 6
  6. 6. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 7
  7. 7. How to Test: Manual vs. Automation • Costs & Assets – Time, People, Infrastructure, Tools, Training • App Development & Testing Cycle – Together – Testing after Daily Builds (Continuous Integration) • Tools & Technology – Frameworks – Myriad of frameworks available to help you • Test Coverage & Reusability – Global coverage is CRITICAL to SUCCESS • Outcome – Learning, Improving, Faster © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 8
  8. 8. Manual Automation Large coverage, quick ly completed, Less money & time wasted, Exact results. Smaller coverage, More money burnt & time wasted, Error-prone © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 9
  9. 9. Improving Time-to-Market Repository Successful build Commit Users / Consumers Build failed Test Cases Application QA & Testers Error reporting Source code Developers © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 10
  10. 10. Tips to Improve the App Quality • Automate generic things as much as you can • During the development your app changes – the same must apply for testing! • Carefully select (testing) technology & partner • Use all 24 hours per day with test automation • Cloud-based platform is the only solution to get you quickly covered in the global markets © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 11
  11. 11. Test Automation Will Deliver You… • App Compatibility & Global coverage Test requirements / Planning Smoke testing Regression testing Regression testing Functional & Interoperability testing Regression testing Functional & Interoperability testing Functional & Interoperability testing Performance & Stress testing Regression testing Compatibility testing Performance & Stress testing Test Automation during the Mobile App Development Process • Reusability • Test efficiency & effectiveness • Faster time-to-market © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 12
  12. 12. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 13
  13. 13. Example: Cost of Manual Testing • Test setup for each device – 5 minutes • 100 test cases, each take about 15 seconds to execute – 25 minutes • 4 different application variants: tablet, phone and 2 localized versions (English & German) • Avg. 2 bugs (of 100 test cases) are found on each test run – 20 minutes to report bugs • 15 devices will be used for testing TIME FOR TESTING: 3000 min (50 hours) COST FOR TESTING: $60/h x 50 = $3,000 (x4) = $12K © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 14
  14. 14. Example: Cost of Test Automation • Creation of test cases/scripts – 100 test cases running 15 minutes per each = 1500 minutes (25h) COST FOR TEST CREATION: 25 hours x $60/h = $1,500 • Tools for test automation with the following setup: – Unlimited devices (250+ Android devices) – Unlimited execution times TESTDROID CLOUD: $1,500/month TOTAL COSTS FOR TEST AUTOMATION: $3,000/month Direct savings compared to manual: $9,000 ROI ratio: (Gains-Investm.) / Investm.: $9K-$3K/$3K = 2.00 © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 15
  15. 15. Benefits with Testdroid Cloud • Instant availability of hardware and no need to buy any device by yourself • Testing gets done much faster – typically the same outcome can be achieved in 10% of time • Test cases are reusable for your next project • Immediate feedback – all reports as logs, screenshots, performance stats etc. • QA can focus on quality – not pressing keys! © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 16
  16. 16. Example: On-Premise Setup • Creation of test cases/scripts – 100 test cases running 15 minutes per each = 1500 minutes (25h) COST FOR TEST CREATION: 25 hours x $60/h = $1,500 TESTDROID ENTERPRISE: $2,000/month (15 devices) TOTAL COSTS FOR TEST AUTOMATION: $3,500/month Direct savings compared to manual: $8,500 ROI ratio: (Gains-Investm.) / Investm.: $8.5K-$3.5K/$3.5K = 1.43 © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 17
  17. 17. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 18
  18. 18. Mobile Apps for Different Verticals Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel • Testdroid has helped thousands of app developers in these subverticals! • What are the critical elements in each these verticals? Mobile © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 19
  19. 19. Mobile Apps for Different Verticals Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Mobile • User Experience! • Resource consumption & validation (CPU, Mem…) • Fully utilizing hardware – e.g. Touch screen • Mobile games require access to graphics APIs (e.g. OpenGL ES) © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 20
  20. 20. Mobile Apps for Different Verticals • Usability! • Functional and Utilities & Tools Behavioral testing Video Streaming & Multimedia • Metrics analysis for all captured data (e.g. Banking & Payment logs, screenshots, perf Retail & Travel stats) Mobile • Connection with Social Media + other apps © Copyrights by Bitbar Technologies Ltd. 2013 Games All rights reserved. 21
  21. 21. Mobile Apps for Different Verticals • Performance! • Connectivity, robustness Utilities & Tools and durability Video Streaming & Multimedia • Screen orientation (portrait vs. landscape) Banking & Payment • Graphics Retail & Travel quality, streaming Mobile capabilities • Technologies Ltd. 2013 © Copyrights by Bitbar User profiles Games All rights reserved. 22
  22. 22. Mobile Apps for Different Verticals Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Mobile • Security! • Secure connectivity with back-end systems • Top Quality – extremely brand sensitive vertical • Compliances and verification between real devices and infrastructure © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 23
  23. 23. Mobile Apps for Different Verticals Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Mobile • Data! • Connectivity and data connection with back-end systems • Configurability of the app • Bad quality can hurt the brand and leave you without customers © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 24
  24. 24. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 25
  25. 25. Tip #1: Test early, Test often © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 27
  26. 26. Tip #2: Plan What to Automate Time MANUAL Infrastructure Tools AUTOMATED People Training © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 28
  27. 27. Tip #3: Use only Real Devices © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 29
  28. 28. Tip #4: Use Atomic Test Units Test iteration #1 Test iteration #2 © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 30
  29. 29. Tip #5: Create Hermetic Tests © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 31
  30. 30. Tip #6: Use All Possible Devices Testdroid Cloud’s 250+ Android devices = 93-95% global Android volumes! © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 32
  31. 31. Tip #7: Separate Apps and Tests Your App © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 33
  32. 32. Tip #8: Output Everything to Logs © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 34
  33. 33. Tip #9: End-to-End Testing © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 35
  34. 34. Tip #10: Integrate Dev & Testing Repository Successful build Commit Users / Consumers Build failed Test Cases Application QA & Testers Error reporting Source code Developers © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 36
  35. 35. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 37
  36. 36. Testdroid Products Complete Solution for Mobile Apps/Games Testing © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 38
  37. 37. Testdroid – Because it is important to know what app testing costs! © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 39
  38. 38. Agenda • Common Challenges in Today’s App Lifecycle • How Test Automation Can Shorten Your App’s Time-to-Market and Improve the Quality? • Return on Investment with Test Automation • Different App Verticals – What is Important? • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 40
  39. 39. © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 41

×