Mobile Test Automation

4,457 views

Published on

With more and more applications competing for users attention it is getting increasingly important to ensure that your app works on all Android devices as it should. If you launch your app without proper testing you risk wasting your marketing and development efforts by getting bad ratings and having your users disappointed just because your app does not work on their device.

In his presentation Jouko Kaasila from Bitbar goes through the most common approaches to tackle the testing challenge from application developer's point of view and explain why testing for device compatibility is a must for any serious Android application developer. He will also give examples of most common urban myths about testing and go through some best practices to minimize the cost and effort related to setting up your test infrastructure.

Jouko Kaasila is a co-founder of Bitbar, the company behind Testdroid Android testing tools. He has more than 14 years of experience in mobile software from early mobile services through Smartphone operating systems to Android platform. He is a serial entrepreneur with strong passion to build and help others to build businesses around mobile technologies and he is always keen to exchange views and experiences in this field.

1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
4,457
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
213
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09
  • 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09 12/09/09
  • Mobile Test Automation

    1. 1. PRESENTATION OVERVIEW − Why to test at all? − Richness of Android devices – a real opportunity − Android success story in numbers − Urban myths about mobile testing − Options for professional grade testing − Typical application implementations pitfalls − Some automated testing frameworks for Android1 info@bitbar.com
    2. 2. WHY TO TEST AT ALL? Beta test approach  Systematic testingPre launch activities: Light manual testing for Testing criteria: Some level of basiccatching most obvious bugs. Usually done on acceptance criteria needs to be created: Whatone or max 2-3 devices. devices are supported, what use cases are important, what is the expected performanceBeta phase: App is distributed to 10-100 betausers who will report bugs and any other issues Setting up test automation: Test cases are converted to test scripts, that are executedLaunch: Based on the feedback from beta every time when the app changestesting a new version is created and launched tothe app stores. Testing throughout the development cycle: Testing starts when first features are implemented and continues even when the app is released2
    3. 3. WHY TO TEST AT ALL?  Some quotes: Summary: Developers need to avoid treating end-users as beta testers and conduct pre- launch app testing to avoid critical reviews due to poor usability once released to market “If there is a major bug that is not caught before the launch, then a lot of expensive development and marketing work is quickly wasted” "The first wave of users gives the app bad ratings, which then undermine [the number of] downloads and the app would [eventually] sink into oblivion."3 Source: http://www.zdnet.com/mobile-app-testing-imperative-but-not-foolproof- 7000000787/
    4. 4. WHY TO TEST AT ALL?  2012 Cap Gemini Quality Report (Mobile Testing) “Mobile design and development fits nicely with iterative development. Looping around the cycle of requirements, design, build and test in smaller chucks means that testing need not simply be a cost “tacked-on” to the end of a project. There are some types of such as penetration and security testing that should only be done towards the end of the project, but for functionality, performance and ease-of-use testing we recommend that this be done iteratively during the mobile project.”4 Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012- mobile-testing/
    5. 5. WHY TO TEST AT ALL?  2012 Cap Gemini Quality Report (Mobile Testing) “Notable from the survey is that 18% of responding companies say that they do not have enough time to test mobile apps, and 65% do not have the right tools. Also 52% cite lack of devices as a reason not to do mobile testing”5 Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012- mobile-testing/
    6. 6. THE RICHNESS OF ANDROID ECOSYSTEM Fragmentation is innovation Samsung Galaxy S2 < 10% market share Graphical representation of fragmentation in the Android ecosystem. Each rectangle represents the market share of one6 Android device. Samsung Galaxy S2, the most successful Android to date, has less than 10% market share. Source: Open Signal Maps, May 2012
    7. 7. ANDROID SUCCESS STORY IN NUMBERS − Over 500m Android Q2/2012 Q2/2012 Q2/2011 Q2/2011 devices shipped since OS shipments market shipments market YoY change share share Android 1.0 − About 220m iOS devices Android 104.8 68.1% 50.8 46.9% 106.5% have been shipped since 2007 iOS 26 16.9% 20.4 18.8% 27.5% Blackberry 7.4 4.8% 12.5 11.5% -40.9% Symbian 6.8 4.4% 18.3 16.9% -62.9% Windows 5.4 3.5% 2.5 2.3% 115.3%7 info@bitbar.com Source: IDC, 8.8.2012
    8. 8. URBAN MYTHS ABOUT ANDROID TESTINGMyth #1: Testing on emulators is enoughThis app works perfectly onAndroid 2.3 and 4.0 emulatorsbut on some of the mostpopular devices looks likethis:8 info@bitbar.com
    9. 9. URBAN MYTHS ABOUT ANDROID TESTINGMyth #2: Testing on 5-8 most common devices isenoughYes, if reaching every everyfourth Android user is ok foryour CEO:9 info@bitbar.com
    10. 10. URBAN MYTHS ABOUT ANDROID TESTINGMyth #3: Exploratory testing just before launch isenough  Exploratory testing is not very good for catching any regressions  Exploratory testing is not very repeatable  Exploratory testing (any manual testing) does not scale very well  However, exploratory testing has its own merits – it is not enough just by itself10 info@bitbar.com
    11. 11. URBAN MYTHS ABOUT ANDROID TESTINGMyth #4: Users will understand if there are someissues  Application not working and having bugs is #1 reason for uninstalling the app  Users don’t update their apps very frequently  Quality issues are #1 reason for bad reviews in Google Play (about 80%), Application content related issues are only (20%)11 info@bitbar.com
    12. 12. TYPICAL APPLICATION IMPLEMENTATIONPITFALLS Mix different technologies and frameworks Extensive use of web views − Webkits differ from one device to another − All sorts of timeout due to most content coming from server − Issues in http headers across devices − CSS&Javascript cause their own problems Use cross platform tools − Especially the ones that generate the code on server side are very difficult to debug − Wrapping HTML5 to an app is quite ok − The time you save on development, you loose on debugging12 info@bitbar.com
    13. 13. HOW TO SET UP A PROFESSIONAL GRADETESTING Set up Continuous Integration and version control Automate most common use cases − Address the basics first − Establish a repeatable baseline − Keep adding new tests as your functionality grows − Do not try to automate everything – there is still time and place for manual checks as well − Maintain your test library as your code evolves (store both in version control) Define your target devices: − Depending your ambition level anything from 20 to 50 will do. − Test automatically and often with these devices − To get a reference point run occasionally tests with all possible devices (especially before any release)13 info@bitbar.com
    14. 14. TESTDROID PRODUCTS Testdroid Cloud Testdroid Cloud− Device cloud for developers to run their tests on 100+ real Android devices, in parallel− Provides detailed test results and analytics Testdroid Private Cloud− Private on-premise device cloud for Enterprises to run their tests on Android devices in their own development lab− Provides detailed test results and analytics Testdroid Recorder− Developer tool to simplify and accelerate the creation of test scripts14 info@bitbar.com
    15. 15. INTRODUCING TESTDROID APP CRAWLER Intelligent systematic app See video: http://youtu.be/eYvhyjf7HoY exerciser− Device cloud for developers to run their tests on 100+ real Android devices, in parallel− Provides detailed test results and analytics Visual easy to read results− Visual browser for inspecting the crawlers path through the app− Screenshots, logs and performance info Android testing will never be the same!15 info@bitbar.com
    16. 16. TEST AUTOMATION FRAMEWORKS FORANDROID - OVERVIEWThis is an overview of OPEN SOURCE Android testing frameworks – there are tens ofclosed proprietary testing tools/frameworks that are not covered hereSelected frameworks:- Monkey exerciser- Monkey Runner- Robotium- Roboelectric- Calabash- Monkey Talk16 info@bitbar.com
    17. 17. AUTOMATION FRAMEWORKS FOR ANDROID -OVERVIEWTwo main approaches: Android Instrumentation Framework Instrumentation wrapped to the app itself17 info@bitbar.com
    18. 18. AUTOMATION FRAMEWORKS FOR ANDROID –MONKEY EXERCISER What: - A UI/Application exerciser monkey for creating stream of random user events such as clicks, touches, gestures and some system level events Usage: $ adb shell monkey -p your.package.name -v 500 Pros: - Useful way to stress test your application to find non-obvious usage patterns and corner cases Cons: - No way to create any other inputs than clicks which make for instance logging in to an app impossible18 info@bitbar.com
    19. 19. AUTOMATION FRAMEWORKS FOR ANDROID –MONKEY RUNNER What: - An API toolkit for writing programs that control an Android device or emulator from outside of Android code Usage: device.press(KEYCODE_MENU,DOWN_AND_UP) Pros: - Only way to create tests that span across application boundaries Cons: - UI elements can only be clicked with x,y clicks that do not scale => you have to create one script for each resolution. There is no validation mechanism for any events created by clicks19 info@bitbar.com
    20. 20. AUTOMATION FRAMEWORKS FOR ANDROID –ROBOTIUM What: - Its like Selenium for Android. A JUnit based helper library/framework Usage: solo.clickOnText("Configuration"); Pros: - Powerful UI, functional and system testing capabilities (because its based on JUnit) Cons: - Cannot cross the application boundaries in test scripts (because it is using Android Instrumentation Framework)20 info@bitbar.com
    21. 21. AUTOMATION FRAMEWORKS FOR ANDROID –CALABASH What: - Cucumber based test automation framework for Android and iOS Usage: And I select ”Options" from the menu Pros: - Creating scripts does not require any programming skills, Same scripts work across platformsCons: - Wraps the application under test with an http server so you will be testing a different binary that the one you are going to ship.21 info@bitbar.com
    22. 22. AUTOMATION FRAMEWORKS FOR ANDROID –MONKEY TALK What: - Cross platform testing tool/framework for Android and iOS Usage: Input address Verify “123 Main St.” %timeout=500 Pros: - Same script works for your Android and iOS app (if the apps are identical)Cons: - Uses proprietary scripting language. Requires you to wrap your app with a testing library so you will be testing a different binary that the one you are going to ship.22 info@bitbar.com
    23. 23. AUTOMATION FRAMEWORKS FOR ANDROID –ROBOELECTRIC What: - Running unit tests on virtual machine – instead of an emulator or real devices Usage: activity.findViewById(R.id.press_me_button); Pros: - Ultra fast! Can be run in every code commit Cons: - It is just for unit testing – it doesnt give you any info on how your app runs on target devices23 info@bitbar.com
    24. 24. FURTHER INFORMATION Additional information on web: − Company and Products: http://www.bitbar.com – Testdroid Cloud: http://cloud.testdroid.com – Testdroid App Crawler: http://youtu.be/eYvhyjf7HoY − Demo video (3min): http://youtu.be/zj11lzUmScg?hd=124 info@bitbar.com

    ×