Mobile Apps Testing Android IOS Blackberry Windows
Introduction A mobile application is a software application designed to run on smartphones, tablet computers and A mobile application, also known as other mobile devices. They are mobile app, is a term used to available through application describe software that runs on distribution platforms, which are smartphones and mobile phones. typically operated by the owner of Mobile applications are designed to the mobile operating system, such as educate, entertain, or assist the Apple App Store, Google consumers in their daily lives. Play, Windows Phone Store and BlackBerry App World. Some apps are free, while others have a price.
Cont …A mobile application will normally be structured as a multi-layered application consisting of user experience, business, and data layers. Whendeveloping a mobile application, you may choose to develop a thin Web-based client or a rich client. Ifyou are building a rich client, the business and dataservices layers are likely to be located on the device itself. If you are building a thin client, the business and data layers will be located on the server..
Figure 1 illustrates common rich client mobile application architecture withcomponents grouped by areas of concern
Types of Mobile Application Web application• Web apps live on a remote server and are delivered to users over the Internet through a browser. Any website component that performs a function for the user can be considered a Web app, so even Googles search engine counts. Web apps can be easier for IT to manage, because they all run on the same platform.
Native Application Native apps are applications developed for a specific platform or device. Because native apps interact directly with the operating system, they often perform better than Web apps. Native apps can also take advantage of the specific features of an OS or device, which sets them apart from Web apps.
Hybrid Application Most apps are hybrid apps, which combine native and Web app features. Hybrid apps work whether or not devices are connected to the Internet, work with a devices file system and integrate with Web services. For example, online banking apps store some data on devices, but users access them over the Web.
Legacy Application Legacy apps were built on older technologies but are still important to business processes. The challenge for IT comes in keeping that old technology working while converting it for new, more efficient platforms. Fortunately, legacy apps often use languages and OSes with open or standard programming interfaces, which can make them easier to update.
Challenges in Mobile Application TestingUnlike the PC-based environment, the mobile environmentcomprises a number of devices with diverse hardware (physicalproperties), software configurations (platform, OS etc.) andcommunication intricacies. You can’t certify that thefunctionality on one device, will work on all other devices withsame properties.This diversity in mobile computing environments presentsunique challenges in application development, deploymentand quality assurance, thus requiring unique testing strategiesfor different platforms.
Cont..Few challenges are:Operations with memory control of memory intensive leaksThe intensity of power consumption, sensitivity to the chargerThe use of disk space, stability in the limited space on the drive, logging, work with memorycardsSupport for various screen resolutions, portrait and landscape modesDifferent standards compliance for different platform (Symbian Signed certification, JavaVerified testing, etc.)
Functional testing Functional Testing ensures that the mobile application is behaving as per the requirements. Majority of the testing is driven by the user interface and call flows. However the specialized devices with emerging technologies like smart phones having good processing power, support multi-tasking applications with functional and background processing. This has multi-fold increased the functional testing. The complexity increases when this basic functional testing is combined with other types of testing, like port, laboratory, performance, and memory leakage testing as mentioned below. Due to unavailability of actual connectivity and required infrastructure, the mock/prototype and simulated testing further add the complexity.
Port Testing 1. Device Testing: •The testing is done on different devices with same platform. •There are chances that the base This type of testing is done to test platform/OS version can change. But it is the same functionality on mostly avoided. However, in case the OS version gets change for same different devices with different platform, then major testing is platforms. It can be classified into recommended in which the complete 2 categories: testing is done again for the variable OS version. •Based on the device properties, specific testing scenarios are identified in which testing is done. This is termed as minor testing.
Cont… 2. Platform Testing: • Platform testing is done on devices of different platform (OS). For example, functionality testing on device with J2ME to device with BREW platforms. Here because of variable OS architecture, the same application is implemented differently, so complete thorough testing has to be done.
Laboratory Testing This testing is done specifically at carrier side where the complete wireless network is simulated as similar as the real environment. The actual network problems, which might arise in the real environment, can be identified and resolved in this simulated laboratory. This is very important when application is using voice and/or data connection to perform some functions. Although simulators and other test tools do support simulation of such type of networks. But essentially they are not reliable enough to provide the complete assurance about the real-time network issues. Hence it is not advisable to launch a mobile application based on testing conducted only on simulators.
Performance TestingPerformance testing means the behavior of the application under certain conditions like low resources(Memory/Space), multi-user simultaneous access to application servers; time taken in data transfer is high.The applications performance may be affected from two sides; Server side and Client side.The major performance issues on the server side are:Variations in Response timesStreaming resource intensive multimedia packetsDelays/Drop in delivery of packetsApplications crashIn-efficient use of resources
Cont…The major performance To avoid these issues weissues on the client side should do the followingare: • Use extensible load testing tools• Applications behave differently on which support custom protocol various platforms and handsets • Include performance testing early• High Memory and CPU in the cycle consumption • Using stubs to simulate external• Mobile application too slow to components load • Record work flows using mobile• Application causing battery drain clients or simulators out
Memory Leakage TestingWhen a program does not correctly manage its memory allocations i.e. loses track of a piece of memory that wasallocated and hence will never be released to OS, it is said to leak memory. As result it diminish the performance of anapplication or even fails it.Mobile devices have huge constraint on memory in terms of installation and runtime memory. Although memory on thedevices has grown leaps and bounds but the constraint remains the same. This is due to the fact that along with growthin device memory, its consumption is also increases with background applications, sync process, multi-threading etc.With scarcity of memory the device behaves very weird. In some cases, the device runs very slow and inother, application crashes. This behavior is very random and is very hard to diagnose.A tester has real hard time to do memory leakage testing. With improvements, the memory instance can be fetched atany time and with verifying the trend, at least an alarm/warning can be raise to verify the memory usage.Memory usage mostly increases due to application allocate memory for creating objects , but not releasing the object orlocking the shared memory for specific usage, and hence not releasing it to the pool.New programming languages although have their own way of memory management like garbage collector in java, butprogrammers need to be very careful in dealing with explicit memory management as in C or C++.
Interrupt/TAPI Testing While running a mobile application, there are interruptions such as incoming calls, text messages, low memory or low battery notifications. The mobile application should be able to handle these interruptions gracefully without causing a disruption.
Cont… Different kinds of interrupts are: Incoming and Outgoing SMS and MMS Incoming and Outgoing calls Battery Removal Cable Insertion and Removal Network outage and recovery Media Player on/off Device Power cycle
Cont… Testing the application for interrupt handling is a very important aspect of mobile testing as these interrupts are a very common phenomenon that may occur anytime during the application run. The interrupts must be handled properly so that they do not cause interruption in the application functionality. The application must be in a suspended state during the interruption but must resume continuing its normal function after the interruption is over. The hardware interrupts are on the highest priority and the software interrupts can be prioritized accordingly
Usability testing Essentially the application usability plays a crucial role for success in market. The usability testing is done to evaluate the application for achieving its specified goals, efficiency and acceptance from the users. Usability testing is usually conducted by the test users in the test environment. Users are given the tasks to accomplish and their feedback about the problems faced, ease of use is recorded. This is the most accurate system to evaluate the usability of the product as it gives direct input on how real users use the system. The major goals of the usability testing are Performance, Accuracy, Recall and Emotional Response. The different methods used for usability testing are Hallway testing, Remote usability testing, Expert Review and Automated Expert Review
Installation Testing Installation testing is one of the most important parts of testing activities. The mobile applications are either pre-installed on the device or can be downloaded from any store. The installation testing is not necessary if the application is pre-installed. But in case user is downloading and installing an application on the device, this testing must be done to ensure a smooth installation without any difficulties. This testing focuses on each step of the installation process that a user has to follow. Usually the methods followed for downloading an application are ‘Over the Air’, ‘through cable’, SD Card, and Bluetooth. This testing covers installation, upgrade, and uninstallation of the application.
Security TestingMobile device security will become more and more important as the userbase grows, so it is essential to test the security of your mobile webapplications, sensitive data storage, and how your application behaves undervarious device permission schemes.A. Most mobile devices assume one user; they do not have multiple accounts.(Except in the case of multiple email addresses configured per device, whichshould be tested especially if your application deals with the device’s Contactsor Email functions). But in general there is no concept of userswitching, different profiles, or permissions based on user level.B. It is up to the user whether or not they configure a password/unlockpattern for their device at all.C. Outside communication of any sensitive data should be done over SSL/TLSbecause it can’t be assumed that the user will configure their Wi-Fi to besecure.
Stress TestingMobile device applications have much less overall device memory and power availableso must handle them very efficiently.Stress testing is a must to find exceptions, hangs, and deadlocks that may go unnoticedduring functional and user interface testing.1. Load your application with as much data as possible to try to reach its breaking point2. Perform the same operations over and over again3. Perform the repeated operations at varying speeds – very quickly or very slowly4 Leave your application running for a long period of time, both interacting with thedevice and just letting it sit idle, or performing some automatic task that takes a longtime.(E.g. – Slideshow)5. Randomly send screen taps and keystrokes to your application.6. Have multiple applications running on your device so you can switch between yourapplication and other device applications often.
Localization If you’re focusing on multiple regions/geography, you need to localize your mobile app. What if one of your users is based in China and he doesn’t know anything about English? Having an option to use application in local language will help you gain more popularity amongst local consumers. Hence, it’s better to translate application user interface and adapt graphics for a specific culture/locale. The process can also include translating any help content associated with the application. Again, having multilingual app will add few extra hours in testing your app but trust me it’s worth! It’s good to have larger audience than nothing!
Certification Testing In order to get the compliance certificate for the application, it has to be tested against the various testing guidelines provided for various platforms. Each platform has its own criteria for certifying the application. Following platform specific points should be taken care during certification:
Cont…. • Need to have valid Certificate for Signing the application • Application’s .apk file should be in compliance with http://www.android.com/us/developer-content- policy.htmlAndroid: • Proper versioning should be followed in case of updated build publication • Application can be published on the Android Market web site with valid market account credentials • Android market will scan and test your application and can report errors if any • No specific Test suites available as of now
Cont…iPhone: •Application should have Unique name •Application should have specific category •Need URL where you can have your application’s feedback •Need to follow apple HIG (Human Interface Guidelines) from Apple(http://developer.apple.com/library/ios/documentation/userexperience/conceptual/mobilehig/MobileHIG.pdf”) •Strictly main picture of your application should be in 320×480 or 320×460 size •App store will scan and test your application for various compliance •Application may get rejected under following cases like •Indecent Material •Unexpected delays while running application •Copying existing functionality
Cont…Symbian• Need to have .PKG file for building .SIS file• Application Read Me text file• Test Results for “Universal Tests” and “Capability” Test criteria• There are mainly 3 types of Signing: Open Signing, Certificate signing and express signing.Windows Mobile• Should comply the various Marketplace requirements mentioned under http://msdn.microsoft.com/en- us/windowsmobile/dd569132
Various Platforms As the desktop computer runs on different platforms or operating systems like Windows, Linux, and Mac OS; similarly the mobile device runs on different OS or platforms. The brief details of the various mobile platforms or OS are as follows:
Cont… iOS: iOS from Apple Inc. is a closed source Android: Android from Google Inc. is an open proprietary platform. The applications made onsource Linux derived platform. Till date we have 7 this platform are popular for the rich user releases of Android, the latest being 2.3. This interface it provides. iPhone and iPad are the mostplatform is growing by leaps and bounds and in a popular running products using this platform. year from 2009 to 2010 the market share rose Third party applications were not supported until 850% from 1.8% to 17.2%. the 2.0 release in July 2008.
Cont…Blackberry OS: Blackberry OS from RIM (Research inMotion) is a closed source proprietary platform. Thisplatform majorly focused on ease in use and wasmajorly targeted for business purposes and did notprovide many multimedia supporting features. RIM’sfuture strategy focuses more on the newly acquiredQNX where they have already launched a blackberryplaybook with running version of QNX and expect thefirst lot of smartphones to be launched in early 2012.
Cont…BREW / BREWMP: BREW is a platform run by Qualcomm which is used by some mobilemanufacturers and service providers. However people using the phone do not come to knowthat the platform is BREW as there is no branding involved from BREW side. This platform justruns in the background with custom “skins” of mobile manufacturers or operators on the top.Nowadays in many of the phones where BREW was used, it has been replaced by BREWMP.Bada: Bada is a closed source proprietary platform lately introduced by Samsung electronics toreplace the feature phones to smartphones. The first device to run on bada is “Wave”. With thisdevice Samsung has also launched an app store containing more than 3000 applications.Windows7: Windows 7 is again a closed source proprietary platform introduced by Microsoftwith a completely new UI inspired by Microsoft’s “Metro design language”.
Carriers Examples: Good examples of A mobile carrier is one that mobile carriers are Verizon provides connectivity services to Wireless, AT&T, Vodafone, Airtel , mobile phone subscribers. Aircel and so on.
Emulator and Real Device Mobile emulators are powerful tools for developing mobile applications and are widely used for both manual and automated mobile application testing. On the other hand, since mobile applications are used on real handsets and not emulators, testing on real devices during the QA process is required to ensure the highest level of application quality.
Mobile Emulators: Pros Price - Mobile emulators are free and provided as part of the SDK with each new OS release. Simple - Just download the software, install on your PC and youre ready to go. Multiple emulators can be run in a simple and straightforward manner. Fast – Since emulators are simple client software that runs locally on your PC, they have less latency than real devices connected to the local network or in the cloud.
Mobile Emulators: ConsIncreased Risk - With emulators, by definition, you are not testing on the same platformand network used by your users. This means that even if all goes well, you cannot be 100%sure it will actually work on a real device ("false-positives").Hardware and Software Differences – Emulators are typically a "plain vanilla" version ofthe OS and often do not reflect the specific hardware and software features of eachsupported device. In addition, as time passes following an OS version release, theemulators are not updated to reflect new devices available in the market.Different network environment - In terms of network configuration, mobile emulators runon the PC (through any personal firewall), connect to the LAN and access the Internet viayour corporate firewall. Using real handsets, the network is connected to the radiointerface and from there to the Internet. These differences could affect applicationbehavior.
Cont…. No way to test network interoperability – It is important toDifferences in Computing Resources test the impact of network-related – Depending on the processing events (e.g., incoming call, text power of the PC running the message, etc.) and different networkemulator and the type of handset or technologies Smartphone being used for (e.g., HSPDA, WCDMA, UMTS, LTE) testing, performance on the on mobile app behavior. Sinceemulator may be unrealistically good emulators are not connected to the or bad. mobile network, they do not support interoperability testing.
Real Devices: Pros Reliability - Testing on real handsets always gives you accurate results (no false positives or false negatives). Interoperability testing - Real device testing is typically performed in a live network. True user experience - Testing on real devices is the only way to truly understand the user experience, taking into account the CPU, memory, screen size, etc. for a given device. Performance testing - Easier to expose performance defects with real handsets, as well as defects which are the result of the handset itself or its environment.
Real Devices: ConsLogistics and costs - Best practices call for testing on up to 30-40 devices inyour market and replacing ~30% of those each quarter to stay up to date.The efforts involved in procuring and managing these devices are significant.Cumbersome for development - In the initial development stages, realhandsets are harder to connect to IDE than emulators, which can slow downthe debugging process.Security issues - If you are using devices connected locally to yourworkstation, you will need to make sure USB ports are open. Inaddition, mobile devices can easily be stolen, allowing unauthorized accessto your internal network resources.