For mobile applications, automated testing is a wise choice due to the many OS platforms and devices they run on; we offer a strategy for setting mobile test automation goals and selecting the right tool sets.
Selecting the Right Mobile Test Automation Strategy: Challenges and Principles
• Cognizant 20-20 InsightsSelecting the Right Mobile Test AutomationStrategy: Challenges and PrinciplesUnderstanding the various options and choosing the tools andautomation processes best suited to testing both mobile Web andnative mobile apps can reduce complexity and speed time to value. Executive Summary through test automation. As with conven- tional applications, there is an increased need The mobile phone has transitioned from its tra- to automate mobile applications testing to ditional role as a mere communication medium improve scale and efficiency. A well-orchestrated to that of a multipurpose personal gadget. Tech- automation strategy enables hardware manufac- nological advancements and the proliferation turers and software developers to reduce effort of devices across operating systems (Apple iOS, and accelerate time to market. Android and Windows Mobile) and platforms have made it more challenging for hardware manufac- Mobile Technology’s Growth Promise turers and application developers to develop and roll out new products. The growth of mobile devices is evidenced by the following: For vendor peace of mind, mobile applica- tions must be tested to ensure they run on key • Globally, the user base of location-based services has grown from 96 million in 2009 platforms and across a multitude of networks. to more than 526 million in 2012, according to Despite the pressures of short mobile develop- Gartner Research.1 ment cycles, quality testing of applications across operating systems, device platforms and networks • Worldwide mobile advertising and content is a necessary but daunting task to ensure revenue is forecast to reach $67 billion this long-term success in what is a highly fragmented year.2 and competitive global market. Moreover, non- • IDC predicts that smartphone shipments will functional testing — including usability, security reach 978 million in 2014.3 and adaptability — is as critical as functional testing. Effective testing can help device makers • By 2016, smartphones and tablets will put power in the hands of a billion global and application developers collect appropriate consumers, according to Forrester metrics that improve product quality. Research, Inc.4 This white paper explores industry best practices • Mobile device makers are providing on mobile testing and demonstrates effective customized platforms to increase their share ways of managing mobile application quality of the customer’s wallet. cognizant 20-20 insights | september 2012
Two Key Challenges in Mobile consider a test case that is created and needs toApplication Test Automation be executed on three different platforms — iOS, Android and Blackberry. In this example, the testMobile Platforms and Technology Coverage case is expected to be executed on 13 differentThe primary factor that determines an automation devices and in 10 different combinations, whichtool’s success is its ability to work across platforms leads to 130 rounds of testing. Designing a testand technology stacks. The following challenges execution matrix of this scope is both difficult andinfluence automation success: time-consuming (see Figures 1 and 2).• Device Diversity: Mobile Testing Industry Opportunities >> Multiple platforms and browsers. In the growing mobile market, the need for spe- >> Rendering differences. cialized mobile testing is growing. The following >> Mobile devices with varied application run three opportunities contribute almost 80% of times. total market demand.• Network Challenges: Mobile Automation Testing >> Multiple network types (e.g., GSM/GPRS/ There are only a handful of tools available to suc- Wi-Fi/Wi-Max). cessfully automate testing. The ROI promised by >> Different speeds of connectivity across automation testing highlights the need for more options. geographies. >> Multiple network operators with custom- Functional Testing ized network features. Mobile functional testing always involves a non-• Hardware Challenges: heuristic component that cannot be automated. >> Limitations in processing speed. Mobile Performance Testing >> Limitations of mobile memory size. Compared with a desktop/notebook environment, >> Differences in device communication mobile device system resources (e.g., processing protocols (e.g., WAP/HTTP). power, memory, etc.) are limited. Hence, perfor- mance testing of mobile applications is crucial.Test Execution Challenge Therefore, this domain is growing and has greatTest execution across different devices and brows- potential.ers is a huge, challenging effort. For example,Real-time Scenario for a Smartphone-based App: Sample Test Execution Mobile Applications Use Case 1 iPhone Mobile Test Lab Use Case 2 Wi-Fi Blackberry Use Case n Android Emulators Real Device Device Anywhere iOS Native Android Native Blackberry Native Opera Devices/Browsers Opera Mini Sky Fire Firefox 4 Firefox 5 Firefox 6 Dolphin 3,4&5 (2.1,2.2,2.3, 3.0,4.0) 5,6 ,7,8 & 9 Mobile 11.1 iphone3 iphone4 iphone5 Nexus One Moto Droid Samsung Captivate HTC Incredible Nexus S Moto Droid X Curve 8330 Tour 9630 Bold 9650 Bold 9000 TouchFigure 1 cognizant 20-20 insights 2
Smartphone Platforms and Browser Combinations iOS Android Windows Phone Blackberry OS Symbian Current Version 6.0 Current Version 4.2 Current Version 7.5 Current Version 7.0 Symbian 4 Key Versions Key Version 6.5 Key Version Key Version 2.3, 3.0, 4.0 (iPhone 4.0, iPad 2) 6.x & 7.x series Open Source Closed Source Open Source Closed Source Closed Source Symbian Apple Google Microsoft RIM Multitasking Multitasking Multitasking Multitasking Multitasking Safari Web Browser Chrome/Webkit Browser Internet Explorer Webkit Browser Webkit BrowserFigure 2Mobile Automation: What and Why? • User agent-based automation: This approach leverages user agent add-ons that come withMobile applications are becoming increasingly popular browsers such as Mozilla Firefoxsophisticated, significantly increasing the require- and Google Chrome. The user agents helpment for functional testing (see Figures 3 and render the specific Web content that would be4). To tackle this, test organizations are increas- displayed on the device onto a regular desktopingly exploring alternatives to traditional manual browser. The functionality can be automatedtesting. Automated testing is a highly effective by using popular tools such as QTP, Seleniumapproach to mobile app QA that can provide or RFT — each of which supports all desktopsignificant business returns, provided it is imple- browsers.mented by using the right tools and architecture,factoring in cross-platform challenges. • Cloud-based automation: Cloud-based test automation vendors support Web-based QAThe following specific types of testing also need across platforms. Cloud-based automatedto be automated: solutions are highly effective as they are offered as a unified package.• Cross-platform compatibility testing: This is necessitated by the growing number of • Lab-based automation (devices/simulators): handsets and platforms. This method involves setting up tools in a test environment and leveraging simulators• User experience testing: Most testing orga- or physical devices to automate testing using nizations have limited experience with design popular open-source tools such as Selenium. and execution of usability tests. Web driver–supported device browsers and• Field or network testing: Testing must be simulators are best suited here. performed in a geographically distributed envi- ronment to account for a variety of network Approaches for Native and Hybrid types. Mobile Applications• Structural challenges: Integrating mobile Cloud-based Approach testing toolsets into existing IT systems is a Cloud-based test automation is one popular crucial hurdle. means for achieving native and hybrid types of test automation. The method is useful whenTest Automation Mechanisms long-term automation is required across a setfor Mobile Apps of predefined devices. Commercial tool add-onA mobile Web site can be automated in one of the mechanisms such as HP QTP and IBM RFT are alsofollowing three manners: available for test engineers who are familiar with cognizant 20-20 insights 3
Three-tier Mobile Application Testing Framework Backend Middleware Device UI Application Layers Admin Services Mobile Client Enterprise Data Communication Mobile Web Framework/ Services Mobile Website Application Server Data Integration Testing Device Management Installation Functional Types of Testing SOA Testing Testing Testing Testing Integration Testing Security Testing Compatibility Device Testing Proﬁling Authentication Testing UI Testing Interruption Fully Automatable Partially Automatable Not AutomatableFigure 3industry-wide products. The ability to automate in enabling automation in a closed environment,from anywhere is a great advantage: All that for continuous automation or to achieve testis needed is connectivity to the cloud lab. The regression functionality. They also cover nonfunc-“automation-from-anywhere” feature is a distinct tional scenarios that handle mechanisms largelyadvantage when using tools such as Perfecto relevant to mobile apps.Mobile and Device Anywhere. Open Source/Platform-specific ToolsLab-based Approach Device platform-specific tools reliability is a highThere are multiple tool sets with different capa- priority. Platform vendors supply tools such asbilities available for organizations interested in Robotium for Android and iOS instruments fora lab-based approach. Tools such as EggPlant, iOS to provide this facility. However, to leverageJamo, See Test, ZAP, SilkTest and Test Quest this effectively, it requires additional program-provide industry tool add-ons. They are useful ming skills.Potential Areas of Automation in End-to-End Mobile Application Testing 100 90 80 Percentage of tests performed 70 using automated tools 60 50 40 30 20 10 0 Mobile Functional Mobile Mobile Cross- Security Testing SOA/ Package Enterprise Automation Testing Performance Browser Middleware Application Application Testing Testing Testing Testing Testing (e.g., ERP) TestingSource: Cognizant; based on 350-plus projects performed across all vertical markets and ouranalysis of industry trends.Figure 4 cognizant 20-20 insights 4
Cognizant’s Tools Web. Consider different connectivity aspects for automation execution runs.We have our own platform to easily automatetesting of mobile native and Web apps automation In addition, the following recommendations willacross platforms. be helpful in informing your test automation journey:Our platform ensures single type scripting thatworks cross-platform and thus guarantees greaterreusability of scripts. The platform also offers • Perform tool feasibility tests, because mobile technologies and platforms vary.an easy interface for testers to auto-generatescripts. For instance, scripts can be automated • Select tools that support both platformwithout any programming or scripting language simulators and devices, because you can mixknowledge. It also comes with a facility to and match devices and simulators to optimizeautomate as applications are created to support runs on different platforms.the Agile development methodology. • Aim for automation in nonfunctional areas — interruption and hardware scenarios thatMobile Application Testing Tools include battery state changes.Selection CriteriaThere are many successful techniques that can • Always optimize on platform support: In some cases, there may be a need for one or moremake testing applications on mobile devices tools to perform automation.more effective and efficient. It is essential toensure functional and nonfunctional compliance • Identify multiple device support and versionsof mobile applications on devices and mobile support.automation. It is also key to establish labs for • Determine the use of utilities and reusabletesting mobile applications. functions that add value to automation.Some basic criteria for mobile application testing • Ensure ROI per each mobile platform andinclude the use of emulators, multiple browsers, duration of execution as part of the tool con-different operating systems and the unique types sideration process.of testing required for GUI, field and interrup- • Look for avenues for early automation, becausetion testing. There are roughly 10 key principles most mobile projects are Agile.that need to be considered while selecting theright tool. • Integrated execution with the test manage- ment tool and application lifecycle manage- ment is an important success factor for a tool.11 Key Principles in Selecting the Right Tools forMobile Testing • Look for data-driven automation support:It is good practice to start with a proof of concept Iterations in the execution will increase(PoC) on critical platforms before narrowing coverage and ROI.down to a tool or tool set. One tool may not • Good candidates for automation are testachieve cross-platform automation goals. When cases that have repeated business logic,test automation tools are evaluated, organiza- functional flows or data-driven tests: Smoketions must not overlook how they integrate with test and build acceptance testing, regressiontest management environments. One starting testing, synchronization testing, compatibilitypoint is to identify reusable scenarios across testing, GUI testing and testing non-dynam-platforms and group them accordingly. This facili- ic UI components, as well as cross-platformtates the creation of an appropriate, time-saving functional testing.framework. Moving ForwardFor native apps, choose critical test cases that can The principles and success factors of testbe run across releases as automation candidates automation for mobile applications are muchand ensure that there is a cross-platform reus- different than those traditionally applied to con-ability and multi-browser compatibility. Always ventional applications. The attributes of devices,keep in mind different form factors while deter- environment, tool sets and test coverage addmining test coverage, especially for the mobile cognizant 20-20 insights 5