Your SlideShare is downloading. ×
Mobile app Testing Planning Test Cases for Android Apps
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Mobile app Testing Planning Test Cases for Android Apps

7,737
views

Published on

A reliable user interface (UI) is essential for an app’s success in today’s competitive market. There- fore, extensive testing of the UI with some special care and attention to functionality and user …

A reliable user interface (UI) is essential for an app’s success in today’s competitive market. There- fore, extensive testing of the UI with some special care and attention to functionality and user experience is indispensable. Challenges become even more complex when it comes to the Android platform and the number of unique issues it poses (Android poses significant challenges regard- ing UI). The keyword “fragmentation” symoolizes the biggest obstacle in the broad testing of mobile applications and refers to the difficulties caused by Android devices that are being re- leased onto the market in all shapes, sizes, and configuration types. This article will describe how Android emulators can offer a broad testing coverage of a wide range of device types using some tricks and simple practices.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
7,737
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
106
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. The Magazine for Professional Testers
  • 2. By Erik Nijkamp Planning Test Cases for Android Apps A reliable user interface (UI) is essential for an app’s success in today’s competitive market. Therefore, extensive testing of the UI with some special care and attention to functionality and user experience is indispensable. Challenges become even more complex when it comes to the Android platform and the number of unique issues it poses (Android poses significant challenges regarding UI). The keyword “fragmentation” symbolizes the biggest obstacle in the broad testing of mobile applications and refers to the difficulties caused by Android devices that are being released onto the market in all shapes, sizes, and configuration types. This article will describe how Android emulators can offer a broad testing coverage of a wide range of device types using some tricks and simple practices. Introduction – Testing in a fragmented device landscape One of the biggest challenges for the common Android developer in his daily routine is the wide range of end devices and operating system versions. According to a study conducted by OpenSignal, in July 2013 there were more than 11,828 different Android end-user devices available in the market, all of which differed in type, size, screen resolution, and specific configurations. Considering the previous year’s survey, which recorded only 3,997 different devices, this is an increasing challenge and a constantly growing obstacle. 3. CPU: The “application binary interface” (ABI) defines the instruction set of the CPU. The main distinction here is made between ​​ ARM and Intel-based CPUs. 4. Memory: A device has working memory (RAM) and the predefined heap memory of the Dalvik VM (VM heap). It is the first two features, OS and display, where special care needs to be taken, as they are directly noticeable by the end user and should be continuously and rigorously covered by testing. As to the Android versions, in July 2013 there were eight different versions running simultaneously in the market leading to inevitable fragmentations. In July, 34.1 % out of almost 90 % of those devices were running the Gingerbread version (2.3.3–2.3.7), 32.3 % the Jelly Bean (4.1.x) and 23.3 % the Ice Cream Sandwich (4.0.3–4.0.4). Figure 1. Distribution of 11,828 Android device types (OpenSignal Study, July 2013 [1]) From a mobile app development point of view, there are four fundamental characteristics defining an end device: 1. OS: The Android operating system version (1.1 to 4.3) which is technically defined by the “API level” (1 to 18). 2. Display: The screen is predominantly defined by the screen resolution (measured in pixels), screen density (measured in DPI), and/or screen size (measured diagonally in inches). 64 Testing Experience – 24/2013 Figure 2. Distribution of 16 Android Versions (OpenSignal Study, July 2013 [1]) Considering the device’s displays, a study from April 2013 conducted by TechCrunch showed that the vast majority (79.9 %) of active devices are using the “normal” screen size that is anywhere between 3 and 4.5 inches in size. On these devices the screen densities vary between “mdpi” (~160 dpi), “hdpi” (~240 dpi) and “xhdpi” (~320 dpi). The exception to the rule, with 9.5 %, is the category of devices with a low density “ldpi” (~120 dpi) and a small screen size.
  • 3. With the help of the 2013 Handset Detection Study it is easy to find a list of representative devices. A piece of interesting trivia is that 30 % of Android users across India have a device with a very low resolution of 240 × 320 pixels, as seen with the Samsung Galaxy Y S5360 in the list above. In addition, the 480 × 800 resolution pixels is the one most used at this point in time (seen in the Samsung Galaxy S II). The “What” Mobile apps must provide the best user experience as well as be shown correctly (UI testing) on various smartphones and tablets of different sizes and resolutions (keyword “responsive design”). At the same time, apps must be functional and compatible (compatibility testing), with as many device specifications as possible (memory, CPU, sensors, etc.). Figure 3. Distribution of common screen sizes and density (“buckets”) (Google Study, April 2013) [2] If this diversity is ignored during the quality assurance process, it can definitely be expected that bugs will sneak into the app, followed by a storm of bug reports and ending with negative user reviews in the Google Play Store. Thus, the question at hand is: how can you practically tackle this challenge with a reasonable level of testing effort? Defining test cases and an accompanying testing process is an effective weapon in dealing with this challenge. Test cases – “Where”, “What”, “How”, and “When” to test? The “Where” To save some expensive time on your testing effort, we recommend first reducing the previously mentioned list of 32 combinations of Android versions and display screens to 5–10 variations representing the top devices used in the market. When choosing the reference devices, you should ensure there is a wide enough range of versions and screen types being covered. As a reference, you can use OpenSignal’s survey to help select the most widely used devices, or use the Infographic of Handset Detection [3]. To satisfy curious minds, how to map the size and resolution of a screen to density (“ldpi”, “mdpi”, etc.) and resolution buckets (“small”, “normal”, etc.) of the above statistics can be obtained from the Android documentation [5]. Device model Display Spec. (Width × Height, Size) Display Buckets (Size, Density) Lowest Android Version RAM Samsung I9300 Galaxy S III 720 × 1280, 4.8″ normal, xhdpi 4.0.4 1024 MB Samsung I9100 Galaxy S II 480 × 800, 4.3″ normal, hdpi 2.3.4 1024 MB Samsung Galaxy Y S5360 240 × 320, 3.0″ small, ldpi 2.3.5 290 MB LG Nexus 4 E960 768 × 1280, 4.7″ normal, xhdpi 4.2 2048 MB Asus Google Nexus 7 800 × 1280, 7.0″ large, tvdpi 4.1 1024 MB HTC Wildfire S 320 × 480, 3.2″ small, ldpi 2.3 512 MB Figure 5. Six examples of Android end devices with high diversity and distribution (Handset Detection Study, February 2013) [3] Together with the previously derived “direct” fragmentation issues (concerning Android versions and screen characteristics), the “contextual” fragmentation has its own pivotal role. This role involves the variety of different situations or contexts in which the user is using the end device in his or her own environment. As an example, you should consider stress [4] and exploratory testing to ensure flawless performance if there is an unsteady network connection, interruptions of incoming calls, locking of the screen, etc. Hardware Software Environment User • GPS • Camera • Coverage • Attitude sensor • SD Card • Battery • Memory • CPU • Display • API level • Vendor • modifications • Installed • apps • System settings • Accuracy of position • Slow data connection • Reception breaks • Empty battery • Offline capabilities • Rotation • Phone calls • Modify volume • Device buttons • Screen (nn) lock Figure 6. Different aspects of testing Android devices It is necessary to prepare in advance a list of all possible test scenarios that will cover the most common functions of the app. Early detection of bugs and easy modifications in the source code is only possible through continuous testing. The “How” One pragmatic way to take this broad variety into account is the Android emulator – offering an adjustable tool that can virtually imitate Android end-user devices on a standard PC. Briefly, the Android emulator is the ideal tool in the QA process to perform continuous regression testing (UI, unit and integration tests) with various device configurations (compatibility testing). During exploratory testing, the emulator can be configured to a wide range of different scenarios. For example, the emulator can be set in such a way that will simulate changes in the connection speed or quality. Nevertheless, QA on real devices is indispensable. In practice, the virtual devices used as reference can still differ in some small (yet, for certain apps, very essential) aspects, such as no provider-specific adjustments in the Android operating system or no support for headphones and bluetooth. The performance on real hardware plays its own significant role in the evaluation process and should be tested (usability testing) on all possible end devices taking into account such aspects as touch hardware support and the physical form of the device. Testing Experience – 24/2013 65
  • 4. The “When” References Now that we have defined where (reference devices), what (test scenarios) and how (Android emulator and real devices) to test, it is crucial to sketch a process and define when to execute which test scenarios. Therefore we suggest the following two-stage process: [1] http://opensignal.com/reports/fragmentation-2013/ 1. Regression testing with virtual devices. This consists of continuous and automated regression testing on virtual reference devices to identify essential errors early on. The rationale here is to identify bugs quickly and cost-efficiently. 2. Acceptance testing with real devices. This involves intensive (predominantly manual) testing on real devices before releasing the app to the Google Play Store in a “staged rollout” (for example with alpha and beta tester groups in Google Play [5]). In the first stage, test automation greatly helps to implement this strategy in an affordable manner. In this stage, only test cases which can be easily automated (ie can be executed on a daily basis) should be included. In the ongoing development of an app, this automated testing provides a safety net for the developers and testers. The daily test runs ensure that core functionality is working properly, the overall stability and quality of the app is transparently reflected by the test statistics, and identified regressions can be easily correlated with recent changes. Such tests can be easily designed and directly recorded from the tester’s computer using SaaS solutions such as TestObject’s UI mobile app testing in the cloud. If and only if this stage has been executed successfully, the process will continue with labor-intensive testing in the second stage. The idea here is to only invest testing resources if the core functionality passes the automated test, enabling the testers to focus on advanced scenarios. This stage might include test cases such as performance testing, usability testing, or compatibility testing. Combining the two approaches yields a powerful QA strategy for mobile apps [7]. Conclusion – Doing Testing Right Used in the right manner, testing can be a powerful tool in the fight against the fragmented Android landscape. The crucial component of an effective testing strategy is to define custom-tailored test cases for the application at hand and define a workflow or process that streamlines testing. Testing a mobile app is a major challenge, but it can be solved efficiently with a structured approach, and the right set of tools and expertise. 66 Testing Experience – 24/2013 [2] http://techcrunch.com/2013/04/03/android-activations-tweak/ [3] http://www.handsetdetection.com/blog/where-in-the-world-areandroid-devices-showing-up-infographic/ [4] http://testobject.com/blog/2013/08/find-bugs-automatically-withrandom-testing-in-continuous.html [5] http://developer.android.com/guide/practices/screens_support.html [6] https://support.google.com/googleplay/android-developer/ answer/3131213?hl=en [7] http://testobject.com/blog/2013/11/a-testing-process-that-fits-yourmobile-app.html ◼ > about the author Erik Nijkamp (erik.nijkamp@testobject.com) is the CEO of TestObject GmbH (based in Hennigsdorf, a suburb of Berlin). TestObject specializes in QA solutions for the mobile sector and offers a cloud-based app testing service which radically simplifies UI testing offering test automation with an intuitive tests recorder and ready to use tests for any mobile app. As the product owner, he focuses on the strategic alignment of TestObject’s business solutions. During his time in Silicon Valley (IBM Research US) and consulting (IBM Deutschland GmbH) he gained highly valuable experiences in the high-tech sector.

×