Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Testdroid: Different android test automation frameworks - what works you the best?

15,085
views

Published on

Interested in a presentation video? Sign up at testdroid.com/webinars-archive …

Interested in a presentation video? Sign up at testdroid.com/webinars-archive

Implementing the test automation as part of your daily activities can provide you an enormous value: coverage to detect bugs and errors, early and later during the development, reducing the costs of failure, save time through its repeatability and earlier verification. Today, there are bunch of different options available for testing frameworks on Android – what would work the best for you?

Published in: Technology

0 Comments
27 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
15,085
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
251
Comments
0
Likes
27
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Different Android Test Automation Frameworks – What works you the best? 21 November 2013 Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com 1
  • 2. Agenda • Different Ways to Automate Your App Testing • The Benefits and Characteristics of the Top 5 Android Test Automation Frameworks • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 3
  • 3. Agenda • Different Ways to Automate Your App Testing • The Benefits and Characteristics of the Top 5 Android Test Automation Frameworks • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 4
  • 4. Different Approaches To Testing © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 5
  • 5. Different Ways of Doing Test Automation Hand written test scripts Record and Playback Automatic test exercisers Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty Fastest & extremely automated, excellent for smoke testing/quick testing, availability Benefits: Accurate, specific to your testing needs, plenty of options with frameworks, tools Tradeoffs: Takes a lot of time, ties resources Compelling Recorder+Playback to write test cases/scripts, error- tools available for only few test prone (humans) automation frameworks Not accurate as real test cases © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 6
  • 6. Option #1: Cloud-Based Testing Users / Consumers Successful build Build failed Commit Repository Test Cases Application QA & Testers Error reporting Source code Developers © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 7
  • 7. Option #2: Continuous Integration Users / Consumers Successful build Build failed Commit Repository Test Cases Application QA & Testers Error reporting Source code Developers © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 8
  • 8. Manual Automation Large coverage, quick ly completed, Less money & time wasted, Exact results. Smaller coverage, More money burnt & time wasted, Error-prone © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 9
  • 9. How to Improve the App Quality • Automate generic things as much as you can • During the development your app changes – the same must apply for testing! • Carefully select (testing) technology & partner • Use all 24 hours per day with test automation • Cloud-based platform is the only solution to get you quickly covered in the global markets © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 10
  • 10. Agenda • Different Ways to Automate Your App Testing • The Benefits and Characteristics of the Top 5 Android Test Automation Frameworks • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 11
  • 11. Android Instrumentation Framework • InstrumentationTestRunner – the primary plumbing for running tests on Android • Android Instrumentation Framework is built on top of JUnit – a standard test framework on for any Java development • Mock objects – methods for creating mock system objects such as content, service and intent. © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 12
  • 12. Family Tree of Test Frameworks Calabash Appium Robotium Espresso Android Instrumentation Framework UI Automator JUnit © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 13
  • 13. Robotium: Basics • Open source library extending JUnit with plenty of useful methods for Android UI testing • Powerful and robust automatic black-box test cases for Android apps • Supports native, hybrid and mobile web testing “It’s like Selenium but for Android” • Semi-active open source community with monthly releases © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 14
  • 14. Robotium: Code Example // Public void for the operation public void testRecorded() throws Exception { // Wait for the text 'Hello!' to be shown for newbie if (solo.waitForText("Hello!")) { // R class ID identifier for 'Sign in' - and click it solo.clickOnView(solo.findViewById("com.twitter.android.R.id.sign_in")); // R class ID identifier for entering username solo.enterText((EditText) solo.findViewById("com.twitter.android.R.id.login_username"),"username"); // R class ID identifier for entering password solo.enterText((EditText) solo.findViewById("com.twitter.android.R.id.login_password"),"password"); // R class ID identifier for clicking log in solo.clickOnView(solo.findViewById("com.twitter.android.R.id.login_login")); // Wait until log in is done solo.waitForActivity("HomeTabActivity"); } // Activate the text field to compose a tweet solo.clickOnView(solo.findViewById("com.twitter.android.R.id.menu_compose_tweet")); // Type the tweet solo.enterText((EditText) solo.findViewById("com.twitter.android.R.id.edit"), "Testdroid"); // Tweeting! solo.clickOnView(solo.findViewById("com.twitter.android.R.id.composer_post")); } © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 15
  • 15. Robotium: ExtSolo - Extension Library • Includes useful methods that have not been merged to Robotium, for instance: – – – – – – Automatic scaling of x,y clicks for any resolution Multi-path drags Automatic screenshots on test failure Mock locations Change device language Control WiFi connection • You can download the library at http://testdroid.com/products/apis-plugins-and-libraries © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 16
  • 16. Calabash: Basics • Behavior driven test framework for native Android, native iOS and mobile web • Tests are described in Cucumber and then converted to Robotium or Frank in run time • Supports currently about 80 different natural language commands (controllers). New controllers can be implemented in Ruby or Java • If your app uses custom UI Widgets you have to create controllers for those in order to test your app • Calabash tools include a command line inspector for finding right UI element names/ids © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 17
  • 17. Calabash: Architecture • Calabash installs an HTTP server as an instrumentation package that listens commands from Calabash server • Tests are executed on server side • Each test scenario is described in Cucumber • Ruby Client library converts Cucumber commands to either Robotium or Frank method calls • Webview support is implemented with Javascript injection to the Webview Test server Android device Step definitions, Ruby client library Calabash HTTP server © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. Your app Features 18
  • 18. Calabash: Code Example Feature: Login feature Scenario: As a valid user I can log into my app I wait for text "Hello" Then I press view with id "Sign in" Then I enter text "username" into "login_username" Then I enter text "password" into "login_password" Then I wait for activity "HomeTabActivity" Then I press view with id "menu_compose_tweet" Then I enter text "Testdroid" into field with id "edit" Then I press view with id "composer_post" © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 19
  • 19. uiautomator: Basics • Google’s test framework for testing native Android apps across device • Works only on Android API level >=16 • Runs JUnit test cases with special privileges (test cases can span across different processes) • No support for web view (only handle to web view elements as canvas object) © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 20
  • 20. uiautomator: Code Example // Public void for the operation public void testSignInAndTweet() throws Exception { // Starting application: getUiDevice().wakeUp(); // Press Home button to ensure we're on homescreen getUiDevice().pressHome(); // Select 'Apps' and click button new UiObject(new UiSelector().description("Apps")).click(); // Select 'Twitter' and click new UiObject(new UiSelector().text("Twitter")).click(); // Locate and select 'Sign in' UiSelector signIn = new UiSelector().text("Sign In"); // If button is available, click UiObject signInButton = new UiObject(signIn); if (signInButton.exists()) { signInButton.click(); // Set the username new UiObject(new UiSelector().className("android.widget.EditText").instance(0)).setText("username"); new UiObject(new UiSelector().className("android.widget.EditText").instance(1)).setText("password"); new UiObject(new UiSelector().className("android.widget.Button"). text("Sign In").instance(0)).click(); // Wait Sign in progress window getUiDevice().waitForWindowUpdate(null, 2000); // Wait for main window getUiDevice().waitForWindowUpdate(null, 30000); } new UiObject(new UiSelector().description("New tweet")).click(); // Typing text for a tweet new UiObject(new UiSelector().className("android.widget.LinearLayout").instance(8)). setText("Awesome #Testdroid!"); // Tweeting! new UiObject(new UiSelector().text("Tweet")).click(); } © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 21
  • 21. Appium: Basics • Uses Selenium Webdriver (W3C standard) as a scripting framework • Supports native Android, native iOS and mobile web: – Android via uiautomator (API level >=16) and Selendroid (API level <16) – iOS via UI Automation – Mobile web as Selenium driver for Android and iOS • You can write your Appium scripts on almost any programming language (Haskell/Go/Clojure/Java/Ruby) © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 22
  • 22. Appium: How it works? • Appium is an HTTP server that creates and handles WebDriver sessions • It starts an Appium server on the device that is listening commands from the main Appium process • It receives json requests from client libraries over HTTP • On Android Appium executes these commands as either uiautomator or Selendroid commands depending on the API level of the device • Tests are driven from a Selenium script on Appium server Test server Android device UI Automator controller or Selendroid driver Bootstrap.jar Your app Selenium script © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 23
  • 23. Appium: Test Lifecycle 1. Appium installs Bootstrap.jar as an instrumentation package which contains an uiautomator server and a tcp server 2. The commands from a Selenium script are executed on the main Appium server that relays the commands over TCP/IP to the tcp server running on the device 3. The UI Automation server converts the Selenium commands to uiautomator commands on the fly 4. If the device has API level <=17 Appium installs a Selendroid server that converts the Selenium commands to Android Instrumentation (JUnit) commands 5. All test verification and result processing is happening on server side in the main Appium Server © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 24
  • 24. Appium: Code Example # wait for hello sleep(3) textFields = driver.find_elements_by_tag_name('textField') assertEqual(textFields[0].get_attribute("value"), "Hello") # click sign-in button driver.find_elements_by_name('Sign in')[0].click() # find the text fields again, and enter username and password textFields = driver.find_elements_by_tag_name('textField') textFields[0].send_keys("twitter_username") textFields[1].send_keys("passw0rd") # click the Login button (the first button in the view) driver.find_elements_by_tag_name('button')[0].click() # sleep sleep(3) # click the first button with name "Compose" driver.find_elements_by_name('Compose')[0].click() # type in the tweet message driver.find_elements_by_tag_name('textField')[0].send_keys(”#Testdroid is awesome!") # press the Send button driver.find_elements_by_name('Send')[0].click() # exit driver.quit() © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 25
  • 25. Espresso: Basics • • • • • The latest Android test automation framework from Google A custom Instrumentation Testrunner with special privileges Works on API levels 8 (Froyo), 10 (Gingerbread), 15-19 (IJK) Thin layer on top of Android Instrumentation Framework Uses the Hamcrest matcher library https://github.com/hamcrest © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 26
  • 26. Espresso: Basics 2 • Easy API for extending the framework: – You can write new matchers: onView(myCustomMatcher<View>) – You can write new actions: perform(myCustomAction) – You can write new checks: check(myCustomAssertion) • Reliable: Synchronizes with the UI thread • It’s fast because there is no need for any sleeps (tests run on same millisecond when the app becomes idle) • No support for web views • https://code.google.com/p/android-test-kit/ © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 27
  • 27. Espresso: Code Example public void testEspresso() { // Check if view with the text 'Hello.' is shown onView(withText("Hello.")).check(matches(isDisplayed())); // R class ID identifier for 'Sign in' - and click it onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/sign_in", null, null))).perform(click()); // R class ID identifier for entering username onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/login_username", null, null))).perform((typeText("username"))); // R class ID identifier for entering password onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/login_password", null, null))).perform((typeText("password"))); // R class ID identifier for clicking log in onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/login_login", null, null))).perform(click()); // Activate the text field to compose a tweet onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/menu_compose_tweet", null, null))).perform(click()); // Type the tweet onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/edit", null, null))).perform((typeText(”#Testdroid"))); // Tweeting! onView(withId(getInstrumentation().getTargetContext().getResources() .getIdentifier("com.twitter.android:id/composer_post", null, null))).perform(click()); } © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 28
  • 28. Summary – Comparison Matrix Robotium uiautomator Espresso Appium Calabash Android Yes Yes Yes Yes Yes iOS No No No Yes Yes Mobile web Yes No Yes Yes (Android) Limited to x.y clicks (Android & iOS) (Android) Java Java Java Almost any Ruby Testdroid Recorder UI Automator viewer Hierarchy Viewer Appium.app CLI Supported API levels All 16 => 8, 10, 15-19 All All Community Contributors Google Google Active Pretty quiet Scripting Language Test creation tools © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 29
  • 29. Agenda • Different Ways to Automate Your App Testing • The Benefits and Characteristics of the Top 5 Android Test Automation Frameworks • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 30
  • 30. Tip #1: Test early, Test often © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 32
  • 31. Tip #2: Plan What to Automate Time MANUAL Infrastructure Tools AUTOMATED People Training © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 33
  • 32. Tip #3: Use only Real Devices © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 34
  • 33. Tip #4: Use Atomic Test Units Test iteration #1 Test iteration #2 © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 35
  • 34. Tip #5: Create Hermetic Tests © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 36
  • 35. Tip #6: Use All Possible Devices Testdroid Cloud’s 250+ Android devices = 93-95% global Android volumes! © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 37
  • 36. Tip #7: Separate Apps and Tests Your App © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 38
  • 37. Tip #8: Output Everything to Logs © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 39
  • 38. Tip #9: End-to-End Testing © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 40
  • 39. Tip #10: Integrate Dev & Testing Repository Successful build Commit Users / Consumers Build failed Test Cases Application QA & Testers Error reporting Source code Developers © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 41
  • 40. Agenda • Different Ways to Automate Your App Testing • The Benefits and Characteristics of the Top 5 Android Test Automation Frameworks • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 42
  • 41. Testdroid Products Complete Solution for Mobile Apps/Games Testing © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 43
  • 42. Testdroid – Because it is important to know what app testing costs! © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 44
  • 43. Testdroid Blog and Webinars – Because it is important to how to automate your testing! © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 45
  • 44. Agenda • Different Ways to Automate Your App Testing • The Benefits and Characteristics of the Top 5 Android Test Automation Frameworks • 10 Tips for Professional Mobile App Testing • Testdroid Update • Q&A © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 46
  • 45. © Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 47