Mobile Application Testing: Challenges and Best Practices


Published on

With the rapid rise of mobile devices including smartphones and tablets, many organizations are rolling out mobile apps to extend the reach of their traditional web applications. Although the methodology for mobile application testing is fundamentally the same as that of traditional web and desktop application testing, mobile apps testing presents some unique challenges and issues including coverage of a myriad of mobile devices, usability testing, integration of mobile testing with web interface testing, mobile app performance, and security issues. Jimmy Xu describes these issues and current best practices for solving them. Jimmy introduces the latest technologies and tools in mobile application test automation, mobile application usability, performance, and security testing. In a non-technical and easy-to-understand approach that does not require previous mobile app testing or development experience, Jimmy uses a real world mobile app project to illustrate the challenges—and solutions—of mobile app testing.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mobile Application Testing: Challenges and Best Practices

  1. 1.     BW4 Session  6/5/2013 10:15 AM                "Mobile Application Testing"       Presented by: Jimmy Xu CGI, Inc.                   Brought to you by:        340 Corporate Way, Suite 300, Orange Park, FL 32073  888‐268‐8770 ∙ 904‐278‐0524 ∙ ∙
  2. 2. Jimmy Xu CGI Jimmy Xu has been in the IT industry for more than sixteen years with various companies including JDA Software/i2, IBM/DWL, and CGI. Jimmy has many years of experience designing, developing, and testing enterprise applications, including web and mobile apps, for government, manufacturing, financial, healthcare, and telecom clients. He has deep technical expertise in Java, Linux, Android, iOS, and other enterprise application platforms and a good mastery of various enterprise software development and testing methodologies. Jimmy holds CISSP, CSSLP, and CSTE certifications; has published an eBook on software security; and has been a conference presenter on software development, testing, security, and performance.  
  3. 3. 4/18/2013 Mobile Application Testing: Challenges & Best Practices Better Software 2013, Las Vegas, USA Jimmy Xu, CISSP, CSSLP, CSTE June 5, 2013 Agenda • • • • • • Mobile Testing Overview Mobile Test Automation Mobile User Experience Test Mobile Performance Test Mobile Security Test Questions and Answers 2 1
  4. 4. 4/18/2013 Mobile Testing Overview Mobile Testing Challenges - Multiplicity Mobile application testing will usually need to cover a multiplicity of mobile devices with different capabilities, features, and limitations. 4 2
  5. 5. 4/18/2013 Mobile Testing Challenges - Usability Usability is a greater quality issue for mobile applications than for web or desktop applications as mobile users demand better user experiences. 5 Mobile Testing Challenges - Integration When mobile Applications are developed as add-ons to desktop web applications, requirements, use cases, and test cases will usually need to be synched up. 6 3
  6. 6. 4/18/2013 Mobile Testing Challenges - Performance Mobile applications add additional user load to enterprise application infrastructure: • Communication with enterprise servers can be limited and unreliable in bandwidth • Mobile device hardware resources are usually not as powerful as on desktop/laptop computers. 7 Mobile Testing Challenges - Security Mobile applications expose enterprise data to a larger potential attack surface. 8 4
  7. 7. 4/18/2013 Different Types of Mobile Applications Native Apps Web Apps Hybrid Apps Apps developed to run natively on mobile devices Optimized web pages correctly scales content for the device screen and are optimized for mobile browsers Compatible web pages are compatible with mobile browsers but take no extra steps to optimize the mobile viewing experience Combines native UI elements with access to web content within a web content–viewing area. Mobile Testing Perspectives Comprehensive Scope of MobileTesting • • • • • • • • • • • Native apps Best Practice Mobile web apps Hybrid apps Smart phones, tablets, Internet TVs Android, Windows 8, iOS, Blackberry OS Performance Security Usability Accessibility Compliance Testing Tools Regression Increase • Quality of Testing • Test Coverage • Release confidence Reduce Mobile Testing Team • Time to Market • Testing Resources • Defect Resolution Time • Overall Testing Costs Automation 10 5
  8. 8. 4/18/2013 Mobile Testing Strategy • Functional tests on • • • • primary devices User experience tests only on primary devices or on all devices Automate regression test cases Execute regression test cases on other devices Execute security and performance tests on simulators / real devices 11 Achieving Excellence in Mobile Testing Mobile testing services should drive 20% - 40% reduction in costs while improving speed to market, productivity and quality. Flexible delivery using “best shore” global delivery network provides riskbalanced delivery, optimized offsite leverage and agility in meeting demands Lifecycle Quality Management Frameworks that embody Lifecycle Quality Management approach leading to defect prevention, lower costs, and improved quality Automation Innovation Best Shore Global Delivery Model Deep domain expertise enabling business risk mitigation and improved effectiveness Innovative automated test service framework with trusted tools vendors resulting in faster time to market, improved efficiency, increased productivity and lower costs Domain Focused Solutions 12 6
  9. 9. 4/18/2013 Mobile Test Automation Mobile Test Automation Options Manual test on simulators Manual test on real devices Cloud-based test Cloud-based automated test 14 7
  10. 10. 4/18/2013 Manual Test on Simulators • Simulators not working exactly the same as real devices • Not possible to test SMS, email, and phone call services on real networks • Test logs and video recording of test sessions possible • Online peer sharing of test sessions possible • Inexpensive investment in infrastructure • Manual execution and expensive to run same test multiple times • Suitable for unit tests by developers Manual Test on Real Devices • Test your apps on exactly the same real devices & same live networks as for production use • No test logs and video recording of test sessions available • Very difficult for online peer sharing of test sessions • Test infrastructure can be expensive if testing on multiple devices + carrier networks is needed • Manual execution and expensive to run same test multiple times • Suitable for SIT & UAT when # of combinations of devices / networks is small CGI RESTRICTED AND CONFIDENTIAL 8
  11. 11. 4/18/2013 Cloud-Based Test • • • • • • • Remote access to multiple real devices & live networks Screenshots, test logs and video recording of test sessions available for defect analysis Online peer sharing of test sessions to promote offshore/onshore collaboration and agile testing Monitoring of real-time device performance & user experience Test infrastructure can be expensive depending on # of devices + carrier networks needed Manual execution and expensive to run same test multiple times Suitable for SIT & UAT with large number of devices / networks combinations but small number of test cases Cloud-Based Automated Test • • • • • • • Remote access to multiple real devices & live networks Screenshots, test logs and video recording of test sessions available for defect analysis Online peer sharing of test sessions to promote offshore / onshore collaboration and agile testing Monitoring of real-time device performance & user experience Script once, test automatically on multiple devices Test infrastructure can be expensive depending on # of devices + carrier networks needed Suitable for SIT & UAT with large number of devices / networks combinations and large number of test cases Data Interface Action 9
  12. 12. 4/18/2013 Separation of Actions, Data, Interfaces Actions 1) Enter value 2) Click Button Interface SystemUtil.Run "notepad","","","" Window("Notepad").WinEditor("Edit" ).Type "yes" Window("Notepad").WinEditor("Edit" ).Type micCtrlDwn + "s" + micCtrlUp Window("Notepad").Dialog("Save As").WinEdit("File name:").Set "test" Window("Notepad").Dialog("Save As").WinButton("Save").Click SystemUtil.Run ,"","",""Browser("Google").Page("G oogle").WebEdit("q").Set "mba"Browser("Google").Page("Googl e").WebEdit("q") .SubmitBrowser("Google").Page("mba - Google Search").SyncBrowser("Google").Clo se Data • Select a scripting tool • Select an execution platform • Separate actions, data, and interfaces 19 Script Once, Test Many • • • • Search for texts and images on the screen, regardless of their locations, sizes, and colors Record and play Keyword-based scripting Port scripts to any devices 10
  13. 13. 4/18/2013 Mobile Test Management Automation • • • • • • • Use ALM for mobile test management automation Manage mobile test statements, test cases, and test scripts Manage mobile test execution Manage mobile test defects lifecycle Integration with mobile test environment (simulators, real devices, or cloud) Integration with mobile development IDE/SDK Real-time traceability, KPIs, metrics, reporting CGI RESTRICTED AND CONFIDENTIAL Mobile User Experience Testing 11
  14. 14. 4/18/2013 Mobile User Experience Guidelines General Guidelines • Mobile Web Best Practices (MWBP) • Accessibility (WCAG) Device-Specific User Interface Guidelines • iOS user interface guidelines • Android user interface guidelines • Blackberry user interface guidelines App-Specific User • Domain specific Interface • Organization specific Guidelines Mobile Device Characteristics • • • • • • • • • • • Display screen size Keyboard entry limitation Pointing device limitation Network bandwidth limitation Battery life limitation Device memory limitation Display color limitation Color contrast limitation File format rendering limitation Browser variations Support for HTML, HTML5, CSS, JavaScript, Flash, Java differences • • • • • • • People interact with one app at a time Single window with no visible components Display orientation Gestures for user-device interaction Voice recognition Location services Text message, email, and phone call services 12
  15. 15. 4/18/2013 “One Web” Principle Some services have a primarily mobile appeal One Web: same services to all users / devices Some services and information are more suitable for and targeted at particular user contexts Some services have a complementary desktop and mobile appeal Some services have a primarily desktop appeal Testable Statements – iOS Orientation Change Example iOS orientation change guideline Testable Statements • Think twice before preventing your app from running in all orientations. • If your app only runs in one orientation: • Launch your app in your supported orientation, • Avoid displaying a UI element that tells people to rotate the device. • Support both variants of an orientation. • The iOS version of App XYZ must support both landscape and portrait orientation • The iOS version of App XYZ must support both variants of the landscape orientation by rotating content 180 degrees. • App XYZ is not required to rotate its content when the iOS device is held with the Home button on the top. 26 13
  16. 16. 4/18/2013 Test Cases - iOS Orientation Change Example Test Case • Download and install App XYZ from App Store onto a real iPhone 5 device • Start App XYZ from iPhone 5 home screen • Navigate to different screens of App XYZ • While on any screen, do the following: • • • • Hold iPhone 5 device in landscape with home button on the right, and expect App XYZ’s content to be displayed in a top-down, left-right order Then hold the device in landscape with home button on the left, and expect App XYZ’s content to be rotated 180 degrees Then hold the device in portrait with home button on the bottom, and expect App XYZ’s content to be rotated 45 degrees Then hold the device in portrait with home button on the top, and expect App XYZ’s display of content not to be changed. 27 Mobile Performance Test 14
  17. 17. 4/18/2013 Performance Engineering Domains Load Stress Models the expected production usage of an application by simulating multiple users accessing the application's services concurrently. It is the most fundamental performance test to understand response times and error rates. Tests the system’s stability when the load is raised beyond normal usage patterns. This test determines at what load an application fails, and how it fails. Useful for determining headroom for capacity planning Reliability Also known as endurance testing, this determines the ability of an application to perform its required functions under stated conditions for an extended period of time Volume Most often used to measure an application's throughput with respect to batch or message processing where user response times are not relevant Scalability By comparing to baselines, determines how linear the application's infrastructure can scale to support increased work load Testing Additional Load from Mobile Apps • • Estimating additional load from mobile apps Simulating requests from mobile apps with load test scripts • • Selecting a primary load testing tool Custom coding to support HTML5, JavaScript / Ajax, REST, SOAP Measuring server response time and throughput Monitoring usage of server system resources • • 15
  18. 18. 4/18/2013 Testing, Tuning & Profiling Mobile Apps • Measuring mobile user experience • Identifying bottlenecks in your mobile apps • Performing root cause analysis • Improving mobile user experience • Minimizing mobile app footprint • Profiling function calls Monitoring Device System Resources Measuring footprint of mobile apps on mobile devices Collecting & reporting system usage metrics CPU busy & idle time and background processes Memory & disk space usage Network bandwidth, throughput, and data usage Batter life & power consumption rate Connection interruptions, performance jitters and degradations of mobile networks 16
  19. 19. 4/18/2013 Mobile Security Test Secure SDLC REQUIREMENTS OPERATE/ MAINTAIN Vulnerability scanning should be regularly performed during the maintenance phase on both the application and infrastructure to ensure no new security risks have been introduced and that the level of security is still intact. DEPLOY MENT Application should be tuned and hardened at all layers of the platform stack to minimize infrastructure software mis-configuration vulnerabilities. TESTING Security testing should be performed during SIT to simulate application abuses and ensure any vulnerabilities uncovered are properly addressed as “security bugs.” GOVERNANCE Security Polices, Guidelines, Standards, Procedures, Metrics created & enforced by organizations Should be defined according to governance rules for authentication, authorization, non-repudiation, data confidentiality, integrity, accountability, session management, transport security, privacy, etc. PLANNING & DESIGN Should take into consideration network, server, middleware, database and programming platform vulnerabilities, leveraging techniques such as threat modeling and risk analysis. DEVELOPMENT Static code analysis should be performed to ensure secure coding guidelines are followed and coding vulnerabilities are minimized. 17
  20. 20. 4/18/2013 Device Based Attacks • • • • • • • • • • • Misplaced or lost smart phones / tablets Mobile devices not password protected Unencrypted credentials, insecure storage, or cached data Misconfigured certificate and proxy settings Mobile devices may have unauthorized modifications Malware apps downloaded from app stores or jail breaking Security software often not installed to scan for Trojans, spyware, malware, and spams Invoking classes, services, activities from insecure sources Out of date operating system versions Out of date software utilities Shared mobile app IDs and data Server & Network Based Attacks • • • • • • Data transmissions via Wi-Fi Hot Spots not always encrypted Bluetooth communications in "open" or "discovery" mode NFC offers no protection against eavesdropping Internet connections usually not protected by firewalls Man-in-the-middle attacks Weak authentication schemes Internet 18
  21. 21. 4/18/2013 Abuse Cases, Misuse Cases, Attack Scripts • Misuse cases refer to use cases when the actor initiates unintentional but potentially harmful actions. • Abuse cases refer to use cases when the actor initiates intentionally harmful actions. • Attack scripts refer to scripts developed to automate penetration such as dictionary attack and ‘fuzzing’. Mobile Security Testing Environment & Tools Static source code analysis tools Profiling tools • Mobile software development kit (SDK) • Debugging tools • Simulators • Tracing tools • System data dumping tools • Decompiling tools Penetration test tools • Exploring tools • SQL querying tools • Sniffing tools • Fuzzing tools Proxy tools • Tools that intercept and manipulate the traffic between mobile devices and servers • Automation tools 38 19
  22. 22. 4/18/2013 Vulnerability Severity • Identified vulnerabilities will be flagged with a CVSS severity level Assignment of CVSS score based on: High Severity: CVSS base score of 7.0-10.0 Medium Severity: CVSS base score of 4.0-6.9 • • • • Low Severity: CVSS base score of 0.0-3.9 • • Questions The primary impact on the confidentiality, integrity, and availability of the protected system/resources The derivative impact on loss of life and/or properties The percentage of the impacted area within the total environment How easy it is to exploit the vulnerability How easy it is to remediate the vulnerability How confident the testing team is about the existence of the vulnerability ? 20
  23. 23. 4/18/2013 Our commitment to you We approach every engagement with one objective in mind: to help clients succeed 21