Smart Phone Testing - Are You Smart Enough?


Published on

Explore industry best practices on mobile testing and effective ways of managing mobile application quality. Learn about different aspects of mobile app testing, how to accelerate any mobile testing cycle, and how to establish a mobile testing lab without breaking the bank.

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Smart Phone Testing - Are You Smart Enough?

  1. 1. Smart Phone Testing – Are You Smart Enough?<br />May 2011<br />Pradeep Kumar<br />Mobile Testing Center of Excellence, Cognizant<br />
  2. 2. 1<br />Agenda<br />Emerging mobile trends – Smart phone app’s and OS share<br />Challenges in Mobile App dev and QA<br />Smart phone – industry share<br />Different smart phone OS and browsers<br />Industry patterns on smart phone app<br />Mobile application types – Native, Web and hybrid<br />Testing types and ways of conducting testing<br />Best practices<br />Emulators and it’s Pro’s and con’s<br />Mobile lab strategy and recommendations<br />Cognizant specific tools<br />Case studies <br />
  3. 3. 2<br />Polling Question<br />Apple iphone / ipad<br />Which of these smart phones do you think will be strategic by 2012?<br />
  4. 4. 3<br />Exploring Mobile Trends<br />Percent of Web Browsing <br />Gartner's list of top ten consumer mobile<br />applications for 2012: <br /><br /> 1: Money Transfer2: Location-Based Services3: Mobile Search4: Mobile Browsing5: Mobile Health Monitoring6: Mobile Payment7: Mobile Policy handling – Insurance, Quick Quote8: Mobile Advertising9: Mobile Instant Messaging10: Mobile Music<br />
  5. 5. 4<br />Top Challenges of Developing and Testing of Mobile Applications<br />Source: IDC’s Software Developer’s Collaborative Study<br />
  6. 6. Mobile Platforms to target for the Future<br />Iphone / Ipad<br />5<br />Current version 4G<br />Current Version 2.3<br />Current Version 7<br />Symbian^4<br />Current version 6.0<br />Closed Source<br />Closed Source<br />Open Source<br />Open Source<br />Closed Source<br />Apple<br />Microsoft<br />Google<br />Visual Multitasking<br />RIM<br />No –Multi tasking<br />Restricted Multi-tasking<br />Multi-tasking<br />Multi-touch interface<br />Multi-tasking<br />Safari web browser<br />Multi touch interface<br />Chrome/ Webkit Browser<br />Mozilla Browser<br />Webkit Browser<br />IOS<br />Windows<br />RIM<br />Android<br />Symbian<br />
  7. 7. 6<br />Polling Question<br />Banking<br />What type of industry will take most use of smart phones apart from social media<br />Insurance<br />Retail<br />Healthcare <br />& LifeScience<br />Banking<br />
  8. 8. 7<br />Some of the patterns across industry in the smartphone<br />
  9. 9. 8<br />Mobile Application types<br />Mobile<br />Applications<br /><ul><li>Third party applications that are installed in the device
  10. 10. Requires network connectivity.
  11. 11. Google Chat / Messenger, shopping application on iPhone
  12. 12. Applications that resideon the device and are accessible at any point of time
  13. 13. May require network connectivity
  14. 14. Messaging, Telephony, Multimedia, PIM, etc…</li></ul>Native<br />Applications<br />Hybrid <br />Applications<br />Web Based Applications<br /><ul><li>Accessible through mobilebrowsers
  15. 15. Interfaces with external systems/servers
  16. 16. Wireless Application Protocol (WAP) or Hyper Text Transfer Protocol (HTTP)
  17. 17. Mobile Banking, Reservation system</li></li></ul><li>9<br />Ways of testing in Mobile<br />Emulators <br />Virtual Mobile Testing Tool<br />Emulators <br />15%<br />15%<br />Sanity Test <br />UAT <br />Exhaustive Functional Test <br />Compatibility Test <br />Compatibility Test <br />Compatibility Test <br />Interoperability<br />Interrupt Test <br />Functional Test <br />Regression Test <br />Free downloadable <br />Available for all platforms<br />Actual devices used for validating the results <br />Wide Range of mobile handsets available for testing ( Any Application,<br />Any platform Any carrier N/W)<br />DeviceAnywhere, Keynote, Perfecto<br />Live Carrier Network<br />
  18. 18. 10<br />Ways of deploying Mobile Apps<br />
  19. 19. 11<br />Approach to Testing Mobile Applications<br />Planning<br />Design<br />Develop<br />Execution<br />Requirement Understanding & Analysis<br />Configure Test Environment<br />Identification of Scenarios<br />Perform<br /><ul><li>Function Testing
  20. 20. Test Automation
  21. 21. Specialized Testing
  22. 22. Performance Testing</li></ul>Determine utilization of Emulator / Device<br />Test Data preparation<br />Identification of Tools<br />Create Test Strategy<br />Create Test Scenario<br />Procuring Emulator / Device <br />C2Auto<br />Prepare Test Cases<br />Robotium<br />Automation Assessment<br />Defect Reporting<br />Define Framework for Automation<br />ROI Calculator<br />UI Automation<br /><ul><li>Test Log
  23. 23. Test Report
  24. 24. Defect Report
  25. 25. Test Scenario
  26. 26. Test Cases
  27. 27. Test Data
  28. 28. Test Strategy/Plan
  29. 29. Automation Assessment Report
  30. 30. Automation Framework</li></ul>Deliverables<br />
  31. 31. 12<br />Mobile Test Automation<br />Benefits<br />Reusability of scripts is maintained.<br />Speed-up Test execution which reduces time.<br />Higher Reproducibility for errors through automated testing.<br />Ease to maintain the portability of scripts across device.<br />Less manual interception, hence less error prone which gives higher ROI.<br />
  32. 32. 13<br />iPhone Automation Framework Approach<br />Features<br />Key Benefits<br />Architecture<br />Instrument<br />Tune up<br />Script can be easily executed on real device and simulator<br />Ease of Script using JavaScript<br />Ability to perform UI level automation on iPhone applications<br />Collection of JavaScript library functions<br />Eases the test script creation<br />Easy to install and use <br />Improves reusability<br />UI Automation<br />UI Automation<br />Assertion<br />Test Script <br />Imports <br />tune up<br />Pre-defined <br />Reusable <br />Functions<br />iPhone<br />
  33. 33. 14<br />Android Framework Approach<br />Architecture<br />Features<br />Key Benefits<br />Robotium Implement<br />Robotium implements Activity Instrumentation of Android Testing Framework in a simple manner similar to Selenium<br />The test script is a java file which implements roboitum’s class ‘solo’<br />The test scripts can be run on actual device or emulator using Android Junit Framework in eclipse platforms<br />It can also be run using the Instrumentation test runner through ‘adb’<br />Offers additional functions which are not present in Activity testing<br />It is open source and easy to develop scripts.<br />Improves test productivity, and can be executed on device and Emulators.<br />Android Instrumentation Testing<br />Robotium Test Scripts<br />Instrumentation Test Runner using adb (Non-eclipse)<br />Android Junit Eclipse<br />Application Under Test<br />Emulator<br />Device<br />
  34. 34. 15<br />Comparison of Different Mobile Testing Tools<br />Detailed Comparison4<br />
  35. 35. 16<br />Polling question<br />YES<br />NO<br />Do you have plans of setting up an inhouse lab?<br />
  36. 36. 17<br />Mobile Lab Strategy<br /><ul><li>PCs , MAC machines & different mobile platforms connected to approriate server for Applications.
  37. 37. Necessary Wi-Fi router, SIM cards, with internet connection.
  38. 38. Necessary firewall and Ethernet switch for networking to specific WAN/Networks.
  39. 39. The networking could be an isolated network or public network depending based on need
  40. 40. Support of different Mobile MNO’s for network connectivity.</li></ul>Environment Considerations<br />Typical Mobile LAB<br /><ul><li>Controlled Wireless with limited range with security protected environment.
  41. 41. Having firewalls and switches within lab premises.
  42. 42. All internet access should be monitored with limited connectivity.
  43. 43. Antivirus and security software on devices for enhanced security.
  44. 44. Authorised entry with usage of swipe/similar means for usage of the LAB depending on need.
  45. 45. Entry and exit recording in physical /electronic means to keep track
  46. 46. Restricting usage of personal mobile phones, USB drives or other media .</li></ul>Network security<br />Physical security<br />
  47. 47. 18<br />iPhone<br /><ul><li>Mac OS</li></ul>Environment<br />Standard Emulator<br /><ul><li>iPhone Emulator (SDK)</li></ul>Web based Emulator<br /><ul><li>
  48. 48. Simulator for quickly testing your iPhone web applications
  49. 49. iPhoney
  50. 50. Simulates web browsing experience in the visual shell of an iPhone
  51. 51. iBBDemo – Black baud iPhone Browser Simulator
  52. 52. Renders web kit targeted html including the custom -web kit CSS extensions, effectively giving you a compelling demo/test platform for iPhone Web content from the comfort of a Windows desktop</li></li></ul><li>19<br />Android<br />Android<br /><ul><li>Windows</li></ul>Official Emulator<br />Link-<br />Features<br /><ul><li>The emulator has a simulated GSM modem and SIM card.
  53. 53. User can create multiple instances of simulator and can simulate events like calling between instance, messaging etc.
  54. 54. Each running emulator instance includes a console facility that lets you dynamically query and control the simulated device environment.
  55. 55. It contains all the default applications available with android.</li></ul>Drawbacks<br />Not much user friendly, uses the command-line method to invoke most of the operations like network emulation, telephony emulation, using ‘Android Debug Bridge’ etc.<br />Standalone Emulator<br />Link-<br />Features<br />It comes as a standalone application, no need to download the complete SDK.<br />Drawbacks<br />It lacks many features available in the official version.<br />It just shows how an application will look like in an android environment.<br />
  56. 56. 20<br />Testing Approach for iPad using Simulator<br />Using Simulator<br />The iPad Simulator can be used to test iPad applications on the MAC desktop. <br />Requirements for the Simulator:<br /><ul><li>MAC (Minimum OS Snow Leopard ver 10.6.2 or above)
  57. 57. Software Development Kit (Minimum version 3.2)</li></ul>Benefits :<br /><ul><li>Free ware</li></ul>Limitations:<br /><ul><li>Wireless Services Provided for the Device is not available, hence it uses the Desktop’s network connection.
  58. 58. Memory Leak issues and Performance issues cannot be detected, as it uses MAC desktop’s memory.
  59. 59. Automation is not possible in Simulator.
  60. 60. Dependency on Mac Desktop to launch the simulator.</li></ul>Note: The simulators are iBBDemo2, iPadPeaK<br />
  61. 61. 21<br />M-Webchecker<br />A desktop tool which would check the compatibility of web page for mobile <br />browsers in accordance to W3C standards<br />Features<br />Key Benefits<br />Architecture<br />Evaluates a web page on various parameters and provides a comprehensive ‘Mobile Compliance Report’<br />Provides a detailed information on each and every parameter of a web page to be rendered on a mobile device<br />Renders the web page on an iPhone simulating a real experience of browsing on a mobile device<br />Provides the HTML source code of the requested Web Page<br /><ul><li>Provides the mobile web compatibility percentage with respect to W3C mobile and cognizant standards
  62. 62. Points out the errors in a webpage and recommends changes facilitating faster time to market
  63. 63. Helps developers to do quick profiling during web page design and quick compliance test for Test Engineers
  64. 64. Simulates how a page would be rendered on a mobile browser
  65. 65. Checks for HTTP response headers, encoding, media type, doc type, page size, external resources, Embedded Scripts, Caching, valid markup, style sheet usage and Table usage</li></li></ul><li>22<br />M-WebChecker –Mobile Compliance Testing<br />
  66. 66. 23<br />Automated Web Compliance Testing of Website<br />iPhone<br />Blackberry<br />Android<br />Windows phone<br />
  67. 67. 24<br />Snapshots of the results<br /><ul><li>The Webpage is 77% mobile compliant
  68. 68. A revised mobile edition of the webpage is a requisite
  69. 69. Improvements have to be made to make the web page render legibly on the mobile browsers
  70. 70. The critical areas to be improved are
  71. 71. Content format and markup
  72. 72. Tables for Layout has to be avoided. </li></li></ul><li>25<br />M-Jammer<br />Jammer, Cognizant’s Mobile Interrupt Simulating Tool, can be used for testing the AUT when the mobile application is stressed with multiple interrupts.<br />Features<br />Key Benefits<br /><ul><li>Works without any knowledge of the application, thus stands independent and executes on Native, Hybrid or any Web Application.
  73. 73. Generates various alerts such as Call-Incoming/Outgoing, SMS, Alarm, User Notifications
  74. 74. Interacts with the Connectivity Manager such as Wi-Fi and Bluetooth and controls them.
  75. 75. Sneaks in Vibration and Orientations by controlling the Hardware
  76. 76. Controls the brightness of the screen to enhance the usability for UI developer.
  77. 77. Works as a standalone or interacts with the Server App on Wi-Fi or through USB.
  78. 78. User has flexibility to operate independently on the mobile thus performing Mobile Automation or to configure, initiate and control the Automatic test execution from the Server or to individually trigger / control the mobile scenarios.
  79. 79. Helps to push customized SMS for testing
  80. 80. Detects the AUT hidden bugs
  81. 81. Aids to find Mean Time for Failure</li></ul>Architecture<br />
  82. 82. 26<br />M-TestLocale<br />Jammer, Cognizant’s Mobile Interrupt Simulating Tool, can be used for testing the AUT when the mobile application is stressed with multiple interrupts.<br />Features<br />Key Benefits<br /><ul><li>Works without any knowledge of the application, thus stands independent and executes on Native, Hybrid or any Web Application.
  83. 83. The predefined locations provided can be set as "My Position" as latitudes/longitudes.
  84. 84. In built map facility provided to choose the desired location.
  85. 85. The identified location can be set for a pre-defined time period and can be rolled back when desired.
  86. 86. Helps in testing applications that accesses location-based services without being in the actual network from offshore.
  87. 87. Facilitates testing across multiple locations virtually.
  88. 88. Testers can validate the multi-location data with a high flexibility and reliability. </li></ul>Architecture<br />
  89. 89. 27<br />Mobile App Testing – a Quick Recall<br />Mobile Testing Specifies <br />– Not to be missed<br /><ul><li>Understand the network landscape and device landscape before testing to identify bottlenecks.
  90. 90. Conducting testing in uncontrolled real-world test conditions (field-based testing) is necessary, especially for a multi-tier mobile application.
  91. 91. Select the right automation test tool for the success of the testing program. Rules of thumb for an ideal tool are:
  92. 92. One tool should support all desired platforms.
  93. 93. The tool should support testing for various screen types, resolutions, and input mechanisms such as touchpad and keypad.
  94. 94. The tool should be connected to the external system to carry out end-to-end testing.
  95. 95. Use the Weighted Device Platform Matrix method to identify the most critical hardware/ platform combination to test. This method will be very useful especially when hardware/ platform combinations are high and time to test is low.
  96. 96. Check the end-to-end functional flow in all possible platforms at least once.
  97. 97. Conduct performance testing, GUI testing, and compatibility testing using actual devices. Even though these tests can be done using emulators, testing with actual devices is recommended.
  98. 98. Measure performance only in realistic conditions of wireless traffic and user load.</li></ul>Installation<br />Un-installation<br />Network Test Cases<br />Voice Call Handling<br />SMS Handling<br />Unmapped keys<br />Application Logo<br />Splash<br />Low Memory<br />User Key Actions<br />Continual Keypad Entry<br />Visual Feedback<br />Exit Application<br />Charger Effect<br />Low Battery<br />Removal of Battery<br />Battery Consumption<br />Application Start/ Restart<br />Application Side Effects<br />External incoming Communication<br />–Infrared / BTH<br />
  99. 99. 28<br />Type of Testing<br />Accuracy <br /><ul><li>Validate system results inline with specification
  100. 100. Standard Protocols usage like Location Protocol, Messaging Protocols, Data Synchronization Protocols</li></ul>Interoperability<br /><ul><li>Mobile OS - Apple OS, Android OS, Windows Mobile OS, Symbian OS, Palm OS
  101. 101. Browsers - Safari, IE, Mozilla, Chrome, Webkit</li></ul>Security<br /><ul><li>User profiles, Authentication & Authorization</li></ul> - Native application needs UserID and password for authentication <br /> - Web application needs authentication of the device (service provider) and user credentials<br /> - Application accessibility based on users authorization <br /><ul><li>Data loss, Privacy issues</li></ul> - How data is stored on device<br /><ul><li>Vulnerability Assessment - Security patches, weakness in application code, password strength/construct</li></ul>Fault Tolerance<br /><ul><li>Stable Communication</li></ul> - Connection is suspended, the system needs to re-establish the communication and recover the data directly affected.<br />Recoverability<br /><ul><li>Crash recovery & transaction interruptions </li></ul> - Effective application recovery post unexpected interruption/crash. - Application transaction handling during power failure ( i.e. battery dies or manual shutdown)<br />
  102. 102. 29<br />Thank You<br /><br /><br />