Mobile Applications Testing: From Concepts to Practice

11,286 views

Published on

As applications for smartphones and tablets become incredibly popular, organizations encounter increasing pressure to quickly and successfully deliver testing for these devices. When faced with a mobile testing project, many testers find it tempting to apply the same methods and techniques used for desktop applications. Although some of these concepts transfer directly, testing mobile applications presents its own special challenges. Max Saperstone says if you follow the same practices and techniques as you have before, you will miss critical defects. Learn how to effectively test mobile applications, and how to add more structure and organization to generate effective test ideas to exploit the capabilities and weaknesses of mobile devices. Max shares first-hand experiences with testing mobile applications and discusses how to address various challenges. Work on real problems on your own device and learn firsthand how to be productive while testing mobile applications.

Published in: Technology

Mobile Applications Testing: From Concepts to Practice

  1. 1.       rial      Presented by:  Max Saperstone      rought to you by:          340 Corporate Way, Suite   Orange Park, FL 32073  888‐2 MB  AM Tuto 4/7/2014  8:30 AM          “Mobile Applications Testing:   From Concepts to Practice”      Coveros, Inc.        B   300, 68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com 
  2. 2.         or almost a decade, Max Saperstone has been a test engineer focusing on test automation ng       Max Saperstone Coveros, Inc.   F and the continuous integration/continuous delivery process. Max specializes in open source tools—Selenium, JMeter, AutoIT, Cucumber, and Chef. He has led several testing automation efforts, including developing an automated suite focused on web-based software to operate over several applications. Max also headed a major project developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while developing a system to keep test data “ageless.” He is currently developing a new testi architecture for SecureCI to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.
  3. 3. Mobile Application Testing Tutorial Student Handbook February 2014
  4. 4. Copyright 2014 Coveros, Inc. All rights reserved. Copyright © 2014 by Coveros, Inc. All rights reserved. No part or parts of this Coveros, Inc. documentation may be reproduced, translated, stored in any electronic retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without prior written permission of the copyright owner. Coveros, Inc. 4000 Legato Rd Suite 1100 Fairfax, VA 22033 Phone: 703-349-6109 www.coveros.com
  5. 5. Mobile Application Testing Tutorial Welcome Copyright 2014 Coveros, Inc. All rights reserved. Page 1 Welcome Slide 1 Welcome to the Mobile Testing for Test Professionals training course. In this course, you will learn all about how to apply Mobile Testing to your Software Testing efforts. 1© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Application Testing Tutorial
  6. 6. Mobile Application Testing Tutorial Course Trainers Copyright 2014 Coveros, Inc. All rights reserved. Page 2 Course Trainers Slide 2 Slide 3 2© Copyright 2013 Coveros, Inc.. All rights reserved. Trainer Jeffery Payne Jeffery Payne is CEO and founder of Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Prior to founding Coveros, Jeffery was Chairman of the Board, CEO, and co-founder of Cigital, Inc. Under his direction, Cigital became a leader in software security and software quality solutions, helping clients mitigate the risk of software failure. Jeffery is a recognized software expert and popular speaker at both business and technology conferences on a variety of software quality, security, and agile development topics. He has also testified before Congress on issues of national importance, including intellectual property rights, cyber-terrorism, Software research funding, and software quality. 3© Copyright 2013 Coveros, Inc.. All rights reserved. Trainer Alan Crouch Alan Crouch is a Senior Software Security Specialist with Coveros, Inc. a Virginia- based firm focusing on agile, software quality, and application security. Alan has worked closely with federal agencies and private companies to advise, audit, and support IT security and governance teams. In addition to his cybersecurity experience, he has a strong background in software engineering, test analysis, test automation, and security testing. With a passion for software and security, Alan has focused his career on building secure software and developing better software security practices.
  7. 7. Mobile Application Testing Tutorial Course Trainers Copyright 2014 Coveros, Inc. All rights reserved. Page 3 Course Trainers Slide 4 Slide 5 4© Copyright 2013 Coveros, Inc.. All rights reserved. Trainer Max Saperstone Max Saperstone has been working as a Software and Test Engineer for almost a decade, with a focus on Test Automation and the CI/CD process. He specializes in open source tools, including the Selenium Tool Suite, JMeter, AutoIT, Cucumber, and Chef. Max has lead several testing automation efforts, including developing an automated suite focusing on web-based software to operate over several applications for Kronos Federal. He also headed a project with Delta Dental, developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while also developing a system to keep test data 'ageless.' He is currently developing a new testing architecture for SecureCI™ to allow testing of multiple interfaces, custom reporting, and minimal test upkeep. 5© Copyright 2013 Coveros, Inc.. All rights reserved. Trainer Richard Mills Richard Mills has more than 20 years of experience in software engineering specializing in software development with a concentration on pragmatic software process and tools. Rich has a specific focus in Agile development methods, static and dynamic software analysis tools, configuration management, engineering, automated software build management, and continuous integration. He currently works as a Senior consultant with Coveros, Inc. Helping his customers build software better, faster and more securely. As a consultant, Rich helps customers improve software delivery by coaching and mentoring in the Agile development methodologies, automating software build, test, deployment and employing secure development techniques. Prior to joining Coveros, Rich spent 16 years with Cigital, Inc. fulfilling a wide range of roles including consulting, training, and software engineering.
  8. 8. Mobile Application Testing Tutorial Course Trainers Copyright 2014 Coveros, Inc. All rights reserved. Page 4 Course Trainers Slide 6 6© Copyright 2013 Coveros, Inc.. All rights reserved. Trainer Jeffrey Pierce Jeff Pierce is a Managing Consultant with Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Jeff is a leader with over twenty years experience in information technology (IT) program/project management and client management in all phases of business investment and system development lifecycle. He has proven technical excellence in development and deployment of strategic web-based enterprise information systems utilizing cloud computing and continuous integration technologies. Jeff also has experience with strategy and roadmap development, policy and procedure development, program management office implementation, IT and business management, AGILE SCRUM and XP management, systems engineering, technical team building and leading large and small on-shore and off- shore full-time and contract staff with an outstanding record of accomplishment.
  9. 9. Mobile Application Testing Tutorial About Coveros Copyright 2014 Coveros, Inc. All rights reserved. Page 5 About Coveros Slide 7 Coveros is a consulting company that helps organizations build better software. We provide software development, application security, QA/testing, and software process improvement services. Coveros focuses on organizations that must build and deploy software within the constraints of significant regulatory or compliance requirements. The primary markets we serve include: DoD, Homeland Security & associated critical infrastructure companies, Healthcare providers, and Financial services institutions 7© Copyright 2013 Coveros, Inc.. All rights reserved.  Coveros helps organizations accelerate the delivery of secure, reliable software  Our consulting services: – Agile software development – Mobile application development – Application security – Software quality assurance – Software process improvement  Our key markets: – Financial services – Healthcare – Defense – Critical Infrastructure About Coveros Corporate Partners
  10. 10. Mobile Application Testing Tutorial Agenda Copyright 2014 Coveros, Inc. All rights reserved. Page 6 Agenda Slide 8 8© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda  Introduction to Mobile Testing  Mobile Web Applications  Native Mobile Applications  Unique Mobile Testing Considerations  Mobile Testing Platforms  Mobile Testing Techniques  Tools for Mobile Testing  Wrap-Up
  11. 11. Mobile Application Testing Tutorial Introduction to Mobile Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 7 Introduction to Mobile Testing Slide 9 Slide 10 9© Copyright 2013 Coveros, Inc.. All rights reserved. Introduction to Mobile Testing 10© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda  What is Mobile Application Testing?  Why is Mobile Application Testing Important?  What Do You Need to Know to Be a Good Mobile Tester?  Types of Mobile Applications  Mobile Application SDLC
  12. 12. Mobile Application Testing Tutorial Definition of Mobile Application Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 8 Definition of Mobile Application Testing Slide 11 Companies everywhere are rapidly building and deploying mobile applications. Testing these applications must be accurate, reliable, usable and rapid – keeping up with the pace of development and changing mobile platforms. Additionally MAT will look at testing areas that have typically only been tested by hardware engineers and will need to testing focus in order for mobile applications to be built, delivered, and executed. Title Month Year 11© Copyright 2013 Coveros, Inc.. All rights reserved. Definition of Mobile Application Testing  Mobile Application Testing is a process by which application software developed for mobile devices is tested for functionality, security, usability and consistency.  Although Mobile Application Testing is a relatively new aspect of software testing, all existing standard testing practices and techniques can be applied. What is Mobile Application Testing?
  13. 13. Mobile Application Testing Tutorial Mobile Application Testing Importance Copyright 2014 Coveros, Inc. All rights reserved. Page 9 Mobile Application Testing Importance Slide 12 App stores can reduce the time in the feedback loop, and customers can be utilized for early and frequently feedback. Without it, you may be developing an application your customers don’t want As with all software, code quality verification is important. Users are quicker than ever to abandon an application that doesn’t work perfectly the first and every time. Users have high expectations and are App stores also make it extremely easy to get software out to the public at a rapid pace. You want to ensure your company is putting their best foot forward to maintain a good reputation Having a consistent, predictable release process will ensure good software is released every time The extensive choice of browsers, operating systems, devices and applications make it possible for enterprises to develop new, interactive and highly personal ways for customers and employees to access products, services and applications. Mobile devices are at the same time sources of rich information, powerful entertainment and purchasing platforms, and important work tools. Companies that successfully implement mobile strategies engage both customers and employees in new ways, and create an environment of personal interaction that enhances sales and productivity. What makes mobile so attractive to consumers—instant access to a multitude of apps; a wide range of rapidly evolving devices and functions; a choice of operating systems; new, cloud-based services that require no storage on the device—also makes it tricky for companies to deploy effectively. The fourth annual World Quality Report—from Cap Gemini, Sogeti and HP—reveals that only 31% of organizations surveyed currently have formal processes in place to test their mobile applications, while just 18% of those have a focus on security 12© Copyright 2013 Coveros, Inc.. All rights reserved.  App stores reduce the user feedback loop.  User feedback is both early and frequent.  Usability and quality is king, when users are quicker than ever to abandon an application that doesn’t work the first time.  Users can get updates rapidly and expect seamless updates and integration.  Mobile devices are changing quickly and constantly getting new capabilities and features. Why Is Mobile Application Testing Important?
  14. 14. Mobile Application Testing Tutorial Mobile App Testing Difference Copyright 2014 Coveros, Inc. All rights reserved. Page 10 Mobile App Testing Difference What Do You Need To Be a Good Mobile Tester? 13© Copyright 2013 Coveros, Inc.. All rights reserved.  Mobile Application Testing Tools aren’t Mature or Available – Many tools are created for specific device/OS combinations – Cross-platform tool support is limited – No one tool will do  Mobile Application Testing Requires Specialized Skills – UI and usability focused – Strong testing background – Understand networking – Engineering/hardware focus – Constantly learning  Cloud Components  Cost of Entry  Automation Immaturity – Automation image based and time consuming – Automation difficult to leverage outside the framework What Makes Mobile App Testing Different? 14© Copyright 2013 Coveros, Inc.. All rights reserved. What Do You Need To Be a Good Mobile Tester?  History of Mobile Devices and Emerging Trends  Basic Software Testing Skills  Basic Understanding of the Mobile Software Lifecycle  Basic Understanding of Telecommunications  Awareness of Mobile Operating Systems/Platforms – Experience in virtualization for Linux, or VMWare, and data center operations  Experience with cloud infrastructure and SaaS  Knowledge of Mobile testing tools and frameworks  Knowledge and experience with test automation tools
  15. 15. Mobile Application Testing Tutorial Understand the Basics of Software Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 11 Understand the Basics of Software Testing Slide 15 This course is not meant to teach you the basics of software testing, it assumes you know them and can apply them. 15© Copyright 2013 Coveros, Inc.. All rights reserved. Understand the Basics of Software Testing  Knowledge of Software Testing basics is essential to testing any platform. Even if you just starting your career, you need to know: – Software Testing Goals and Objectives – Software Testing Principles – Software Testing processes, methodologies, and techniques  If you don’t have a strong foundation in software testing, we suggest you start simple and enroll in some basic training programs. What You Need to Know
  16. 16. Mobile Application Testing Tutorial Understand the Basics of Telecommunications Copyright 2014 Coveros, Inc. All rights reserved. Page 12 Understand the Basics of Telecommunications Slide 16 Understanding the basics of Telecommunications will always give you advantages when doing testing mobile applications. Title Month Year 16© Copyright 2013 Coveros, Inc.. All rights reserved. Understand the Basics of Telecommunications  Mobile App Testing involves networks, not just the app or device itself.  Can you identify and understand the differences in these technologies? – 2G, 3G, 4G, Edge, LTE, CDMA, GPRS, GSM, HSCSD, SIM, SMS, WAP, TCP, UDP, Wi-Fi  Telecom is global – …therefore, so is your app. What You Need to Know
  17. 17. Mobile Application Testing Tutorial Mobile Operating Systems/Platforms Copyright 2014 Coveros, Inc. All rights reserved. Page 13 Mobile Operating Systems/Platforms Slide 17 Slide 18 17© Copyright 2013 Coveros, Inc.. All rights reserved. Awareness of Mobile Operating Systems/Platforms  OS Design – Integration Points – Device Capability Usage – Security Features/Vulnerabilities  Device Operation – Standalone – Offline – Wi-Fi – Interruptions What You Need to Know 18© Copyright 2013 Coveros, Inc.. All rights reserved. Awareness of Mobile Operating Systems/Platforms  Device Design – Variable Screen Size/Resolution – Device specific Browser Limitations/Behaviors – Variable input/navigation methodologies  Touch Screen  Keyboard/virtual keyboard  Ball/Pointer  Voice  Menu Structures – CPU Constraints/Variability – Memory Constraints/Variability – Power utilization constraints – Keypad lock What You Need to Know
  18. 18. Mobile Application Testing Tutorial Emerging Trends Copyright 2014 Coveros, Inc. All rights reserved. Page 14 Emerging Trends Slide 19 19© Copyright 2013 Coveros, Inc.. All rights reserved. It’s important to stay on top of emerging trends! Staying on top of emerging trends will better help you understand what you may need to test for when your business wants to add or integrate a new piece of technology into your mobile application. In addition, it will make your products more competitive in the marketplace. Emerging Trends
  19. 19. Mobile Application Testing Tutorial Types of Mobile Applications Copyright 2014 Coveros, Inc. All rights reserved. Page 15 Types of Mobile Applications Slide 20 Slide 21 20© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Mobile Applications  Mobile Web Apps – Non-Responsive – Responsive  Native Apps – Device-Specific – Responsive – Hybrid Apps 21© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Web Applications  Web apps are really websites that, in many ways, look and feel like native applications. They are run by a browser and typically written in HTML5.  Users access them as they would access any web page: they navigate to a special URL. Types of Mobile Applications
  20. 20. Mobile Application Testing Tutorial Native Applications Copyright 2014 Coveros, Inc. All rights reserved. Page 16 Native Applications Hybrid Applications 22© Copyright 2013 Coveros, Inc.. All rights reserved. Native Applications  Native apps live on the device and are accessed through icons on the device home screen.  Native apps are installed through an application store and are developed specifically for one platform.  They can take full advantage of all the device capabilities and can work offline. Types of Mobile Applications 23© Copyright 2013 Coveros, Inc.. All rights reserved. Hybrid Applications  Hybrid apps are application that combines elements of both native and web applications.  Hybrid apps are installed through an application store but integrate with a large amount of content from Web-based services.  They can often function without an Internet connection, but may have limited capabilities when disconnected  The majority of apps fall into this category. Types of Mobile Applications
  21. 21. Mobile Application Testing Tutorial Mobile Application SDLC Copyright 2014 Coveros, Inc. All rights reserved. Page 17 Mobile Application SDLC Slide 24 - [Source: Harty, Julian. “Mobile Apps”] Title Slide 25 24© Copyright 2013 Coveros, Inc.. All rights reserved. The Waterfall Mobile Application SDLC Requirements Implementation •Design •Code Verification •Unit Tests •Internal Installation •System Tests Launch/Release •Pre-Publication •Publication Engagement •Search •Trust •Download •Installation •Upgrade •Downgrade Validation •Payment •Use •Feedback 25© Copyright 2013 Coveros, Inc.. All rights reserved. The Agile Mobile Application SDLC Launch Releases1-n Sprint 1Sprint 0 Sprint 2 End Game Engagement Requirements Design Implementation Testing Launch/Release Engagement/Validation
  22. 22. Mobile Application Testing Tutorial Mobile Web Applications Copyright 2014 Coveros, Inc. All rights reserved. Page 18 Mobile Web Applications Agenda 26© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Web Applications 27© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda – Overview – Types of Mobile Web Apps  Non-Responsive Applications  Responsive Applications – Testing – Technologies
  23. 23. Mobile Application Testing Tutorial What is a Mobile Web Application? Copyright 2014 Coveros, Inc. All rights reserved. Page 19 What is a Mobile Web Application? Exercise 28© Copyright 2013 Coveros, Inc.. All rights reserved. Overview What is a Mobile Web Application?  A mobile web application is an application that is accessed by users over a network such as the Internet or an intranet on a mobile device.  Mobile Web applications are generalized for multiple platforms and not installed locally but made available over the Internet through a browser.  e.g. Online Banking Services  Mobile Web Apps can be written with HTML, CSS and JavaScript 29© Copyright 2013 Coveros, Inc.. All rights reserved.  Check your HTML 5 capabilities  With your Mobile Device, go to: – http://webdirections.org/deviceTest/ Exercise
  24. 24. Mobile Application Testing Tutorial Advantages Copyright 2014 Coveros, Inc. All rights reserved. Page 20 Advantages Non-Responsive Apps 30© Copyright 2013 Coveros, Inc.. All rights reserved. Overview Advantages  You can use simple HTML, CSS, JS for most mobile applications  You can maintain and fix bugs in real time.  Quick upgrade and migration Disadvantages  You can’t sell your application  Must be responsive to multiple resolutions, operating systems and browsers  Struggle with performance and testability 31© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Mobile Web Apps Non-Responsive Apps  A non-responsive web application that responds the same way on a laptop’s web browser as it would on a mobile device.  More and more website’s are moving away from non-responsive apps, but there are many reasons why someone would choose to have a non-responsive app.
  25. 25. Mobile Application Testing Tutorial Responsive Apps Copyright 2014 Coveros, Inc. All rights reserved. Page 21 Responsive Apps Responsive Web Apps - Example 32© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Mobile Web Apps Responsive Apps  Responsive Web design is an approach that suggests that an app should respond to the user’s behavior and environment based on screen size, platform and orientation.  The practice often consists of flexible screens, grids, and layouts, images and an intelligent use of CSS media queries, automatically switching to accommodate for resolution, image size and scripting abilities. 33© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Mobile Web Apps Responsive Web Apps - Example 1920x1080 Screen Resolution – Windows 7 www.coveros.com
  26. 26. Mobile Application Testing Tutorial Responsive Web Apps - Example Copyright 2014 Coveros, Inc. All rights reserved. Page 22 Responsive Web Apps - Example Responsive Web Apps - Example 34© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Mobile Web Apps Responsive Web Apps - Example 1920x1080 Screen Resolution – Windows 7 www.coveros.com 35© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Mobile Web Apps Responsive Web Apps - Example 1136 x 640 Screen Resolution – iPhone5 www.coveros.com
  27. 27. Mobile Application Testing Tutorial Testing a Mobile Web App Copyright 2014 Coveros, Inc. All rights reserved. Page 23 Testing a Mobile Web App Slide 36 36© Copyright 2013 Coveros, Inc.. All rights reserved. Testing Testing a Mobile Web App  Testing a mobile web app is fairly similar to how you would test any other web application. In fact, you can often use the same tools.  Using emulators and user-agents you can often replicate resolutions and device types on your laptop.  However, there are some things you won’t be able to replicate (gestures, shakes, etc.), so don’t think that you can avoid testing on an actual mobile device.  For responsive web apps, you can often reuse tests suites for various resolutions, but no good test plan for a responsive app will have one set of identical tests for each resolution.
  28. 28. Mobile Application Testing Tutorial Copyright 2014 Coveros, Inc. All rights reserved. Page 24 This Page Intentionally left blank
  29. 29. Mobile Application Testing Tutorial Native Mobile Applications Copyright 2014 Coveros, Inc. All rights reserved. Page 25 Native Mobile Applications Agenda 37© Copyright 2013 Coveros, Inc.. All rights reserved. Native Mobile Applications 38© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda  Overview  Types of Native Apps – Device Specific Applications – Responsive Applications – Hybrid Applications  Testing  Technologies – OS API – APP Store – Device Capabilities
  30. 30. Mobile Application Testing Tutorial What is a Native Application? Copyright 2014 Coveros, Inc. All rights reserved. Page 26 What is a Native Application? Device-Specific Applications 39© Copyright 2013 Coveros, Inc.. All rights reserved. Overview What is a Native Application?  Native applications are developed for a specific platform and installed on a mobile device.  Native applications can be pre-installed or installed through an app store on demand.  e.g. Phone app vs. Instagram.  Native applications are built in a language for a specific mobile device 40© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Native Apps Device-Specific Applications  Device-Specific Applications are native apps built for not only one platform (e.g. iOS) but also a device (e.g. iPhone). Responsive Applications  Responsive Applications are platform-specific native apps that are built to respond to various screen size, resolutions and other built-in device capabilities based on the mobile device the application is installed upon.
  31. 31. Mobile Application Testing Tutorial Hybrid Applications Copyright 2014 Coveros, Inc. All rights reserved. Page 27 Hybrid Applications Testing a Native App 41© Copyright 2013 Coveros, Inc.. All rights reserved. Types of Native Apps Hybrid Applications  By a strict definition, Hybrid apps are technically native apps. However, they are part native apps, part web apps; they live in an app store and can take advantage of the many device features available but rely on HTML being rendered in a browser embedded within the app itself.  Hybrid apps allow an organization to have consistency between different device platforms.  They can also provide presence in the app store without having to build much of a native app.  They can provide some cross-platform development, because HTML components can be reused to reduce costs. 42© Copyright 2013 Coveros, Inc.. All rights reserved. Testing Testing a Native App  Testing a native app is a mix of software and hardware testing. A tester has to test the app in all the ways a user may use the app (not just standard user behavior).  Using emulators you can often replicate resolutions, gestures and device platforms on your laptop.  However, there are some things you won’t be able to replicate, so don’t think that you can avoid testing on an actual mobile device.  Since each of the selection points are determined by the developer, test that there is enough space to perform actions and that you are properly applying OS Gestures.
  32. 32. Mobile Application Testing Tutorial Testing a Hybrid App Copyright 2014 Coveros, Inc. All rights reserved. Page 28 Testing a Hybrid App Slide 43 43© Copyright 2013 Coveros, Inc.. All rights reserved. Testing Testing a Hybrid App  Arguably, testing a hybrid app is the most complicated to plan and strategize. A good tester must know how to test both native app and mobile web app components to test a hybrid app successfully.  You can test many components like you would a mobile web app (even the same tools).  Testing the components built with the native code, should be performed just like a typical native app.  However, don’t forget about your integration points!  There are native mobile browser functionality that can have features that can cause unintended behavior in the embedded browser.
  33. 33. Mobile Application Testing Tutorial Unique Mobile Testing Considerations Copyright 2014 Coveros, Inc. All rights reserved. Page 29 Unique Mobile Testing Considerations Agenda 44© Copyright 2013 Coveros, Inc.. All rights reserved. Unique Mobile Testing Considerations 45© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda  User Expectations  Network  Data Storage  Operating System  Jail Breaking  Hardware/Device Capabilities  Security
  34. 34. Mobile Application Testing Tutorial User Expectations Challenges Copyright 2014 Coveros, Inc. All rights reserved. Page 30 User Expectations Challenges Network Challenges 46© Copyright 2013 Coveros, Inc.. All rights reserved. User Expectations Challenges  Global Users – Global User Community  Internationalization/Localization requirements  Instant Feedback (online ratings)  Expectations – Intuitive UI – Performance – 60% of mobile app users will abandon app or site if load time > 3 seconds – Seamless across multiple devices – Seamless across various connectivity options (3g, 4g, Wi-Fi) – Security – Data utilization – Power utilization – Easy Installation and Launch Testing Mobile Applications 47© Copyright 2013 Coveros, Inc.. All rights reserved. Network Challenges  Networks – More than 400 mobile networks globally  Numerous Global Standards – NMT (Nordic Mobile Telephony)  FDMA (Frequency Division Multiple Access) – GSM (Global System for Mobile Communications)  TDMA (Time Division Multiple Access)  FDMA (Frequency Division Multiple Access) – UMTS (3GSM) (Universal Mobile Telecommunications System)  W-CDMA (Wideband Code-Division Multiple Access) – IS-95 (CDMA one) / IS-2000 (CDMA 2000)  CDMA (Code-Division Multiple Access) – WIMAX / LTE (Long Term Evolution)  OFDMA (Orthogonal Frequency Division Multiple Access) Unique Mobile Testing Considerations
  35. 35. Mobile Application Testing Tutorial Network Challenges Copyright 2014 Coveros, Inc. All rights reserved. Page 31 Network Challenges 48© Copyright 2013 Coveros, Inc.. All rights reserved. Network Challenges  Multiple Carriers – Verizon, AT&T, Sprint, T-Mobile, Clearwire, US Cellular, Leap Wireless, nTelos, etc. – Not all 4G Networks are created equal. 4G for Verizon is not the same as 4G for AT&T  Network Hopping – Cell to Cell Hopping – Roaming – Cell to Wi-F – 3G to 4G, Edge to 3G, etc. – International Networks  Near Field Communication  Bluetooth Unique Mobile Testing Considerations 49© Copyright 2013 Coveros, Inc.. All rights reserved. Network Challenges  Implementation variability and issues – Tunneling – Web Proxies  limiting site connectivity  Transcoding  http header information filtering  Several Bandwidth/Connection Modes – 1G, 2G, 3G, 4G – Wi-Fi, Wi-Fi/3G, Wi-Fi/4G) Unique Mobile Testing Considerations
  36. 36. Mobile Application Testing Tutorial Data Storage Challenges Copyright 2014 Coveros, Inc. All rights reserved. Page 32 Data Storage Challenges Operating System Challenges 50© Copyright 2013 Coveros, Inc.. All rights reserved.  Small Capacity  Data Privacy / Shared Storage  Data Encryption (or lack there of)  Cloud Data Storage  Shared Data Resources – Photos – Calendar – Video – Mail – Messages – Notifications Data Storage Challenges Unique Mobile Testing Considerations 51© Copyright 2013 Coveros, Inc.. All rights reserved. Operating System Challenges  App Store Integration  OS API  Fragmentation  Backwards Compatibility  No Industry-wide standards  Capability, Capacity and Performance Unique Mobile Testing Considerations
  37. 37. Mobile Application Testing Tutorial Operating System Challenges Copyright 2014 Coveros, Inc. All rights reserved. Page 33 Operating System Challenges 52© Copyright 2013 Coveros, Inc.. All rights reserved. Operating System Challenges  Interruptions are events that would interrupt normal operations  There are two types of Interruptions – Application Interruptions:  Installation, Launch, Run, Exit, Upgrade, Uninstall/Delete – Device/OS Interruptions  Incoming Call  Receiving Incoming Call  Receiving Message  Device Shutdown  Device Restore  Remove Battery  Camera Activated  Loss of Connectivity  Regain Connectivity Unique Mobile Testing Considerations
  38. 38. Mobile Application Testing Tutorial About Jailbreaking Copyright 2014 Coveros, Inc. All rights reserved. Page 34 About Jailbreaking Jail Breaking Challenges 53© Copyright 2013 Coveros, Inc.. All rights reserved. About Jailbreaking  Jailbreaking (iOS) or Rooting (Android) is the process of removing the limitations on mobile devices through the user of software or hardware exploits.  Jailbreaking permits root access to the operating system, allowing download of non-approved apps, extensions and themes not available otherwise.  Large populations of users utilize Jailbroken/rooted mobile devices.  Corporate policy may permit devices from being Jailbroken Unique Mobile Testing Considerations 54© Copyright 2013 Coveros, Inc.. All rights reserved. Jail Breaking Challenges  Providing escalated privileges by overcoming limitations in a computer system or device by deliberating overriding security, administrative and marketing restrictions by the creator  Non-Approved Applications  Malware, Worms – First iOS Worm, named iKree, was released to affect Jailbroken iPhones in 2009 Unique Mobile Testing Considerations
  39. 39. Mobile Application Testing Tutorial Hardware Challenges Copyright 2014 Coveros, Inc. All rights reserved. Page 35 Hardware Challenges Slide 55 Slide 56 55© Copyright 2013 Coveros, Inc.. All rights reserved. Hardware Challenges  Screen Resolution – Size – Clarity – Portrait vs. Landscape  Bluetooth / NFC  Camera  Processing Power  Speaker/Microphone  Flashlight  Accelerometer  Magnetometer Unique Mobile Testing Considerations 56© Copyright 2013 Coveros, Inc.. All rights reserved. Hardware Challenges  Touch Screen – Single Touch Inputs – Multi-Touch Inputs – Virtual Keyboard – Multiple Inputs Simultaneously  Physical Inputs – Keypads – Track Ball  GPS / Location Services  Power Consumption  Memory Usage – Launch, Background, Foreground, Exit – Simultaneous Applications – Memory Sweeping Unique Mobile Testing Considerations
  40. 40. Mobile Application Testing Tutorial Security Challenges Copyright 2014 Coveros, Inc. All rights reserved. Page 36 Security Challenges Slide 57 Slide 58 57© Copyright 2013 Coveros, Inc.. All rights reserved. Security Challenges  Encryption – Data at Rest  On Device  On Middle-Tier Servers  On Databases  Cached Content – Data in Transit  Privacy – Application Privileges  Features  Data  Capabilities – 3rd Party Application Integration  User Configurable Trust Settings Unique Mobile Testing Considerations 58© Copyright 2013 Coveros, Inc.. All rights reserved. Security Challenges  Operating System – Malware – Critical Bugs – Out of Date (Upgrading) – Support ends after two years typically  Network – No Firewalls – Few devices limit Internet connections – Unsecure Communication Channels  Bluetooth Discovery Mode  Unsecure Networks  Unauthorized Modification, Jail Breaking, Rooting Unique Mobile Testing Considerations
  41. 41. Mobile Application Testing Tutorial Mobile Testing Platforms Copyright 2014 Coveros, Inc. All rights reserved. Page 37 Mobile Testing Platforms Agenda 59© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Testing Platforms 60© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda  Introduction  Desktop Browsers  Mobile Emulation  Cloud Testing  Physical Device Testing  Remote-Device Testing  Recommendations
  42. 42. Mobile Application Testing Tutorial Choosing the Right Platform Copyright 2014 Coveros, Inc. All rights reserved. Page 38 Choosing the Right Platform What is possible with Desktop Browsers? 61© Copyright 2013 Coveros, Inc.. All rights reserved. Introduction Choosing the Right Platform  There are many different device platforms, that are often fragmented and grow at different paces  iOS, Android, Blackberry, Windows phone, Symbian, etc.  Multiple form factors and screen resolutions  Users get new devices every 24 months  There are often carrier restrictions on devices (beyond network considerations) 62© Copyright 2013 Coveros, Inc.. All rights reserved. Desktop Browsers What is possible with Desktop Browsers?  Desktop browsers can be extremely useful when developing and testing mobile web apps.  Desktop browsers like Chrome and Firefox have many tools to effectively use user-agents to simulate multiple mobile browser types with little or no cost.  Leveraging desktop browsers does not replace device testing but they can provide robust debugging options and can be used to simulate multiple screen sizes.
  43. 43. Mobile Application Testing Tutorial User-Agent Switcher for Chrome Copyright 2014 Coveros, Inc. All rights reserved. Page 39 User-Agent Switcher for Chrome Slide 63 https://chrome.google.com/webstore/detail/user-agent-switcher-for- c/djflhoibgkdhkhhcedjiklpkjnoahfmg/reviews?hl=en-US You can also add user-agent strings as necessary (highly recommended) Title Month Year 63© Copyright 2013 Coveros, Inc.. All rights reserved. Desktop Browsers User-Agent Switcher for Chrome Example: iPhone Agent on Google Chrome in Incognito Mode
  44. 44. Mobile Application Testing Tutorial What is Mobile Emulation? Copyright 2014 Coveros, Inc. All rights reserved. Page 40 What is Mobile Emulation? iPhone Emulator 64© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Emulation What is Mobile Emulation?  Emulators offer the less expensive testing option, but they have many limitations for mobile testing and may not give an accurate depiction of the mobile user experience.  However, as part of the mobile testing process the use of emulation has an important role. There is certain testing that you can conduct with emulators that provides value. For example, testing for screen size or visual form factor.  However, emulation is often an approximation of the rendering and can only get you so far.  Leveraging emulators is a positive step towards real device testing, however, not sufficient for comprehensive testing needs. 65© Copyright 2013 Coveros, Inc.. All rights reserved. Emulator iPhone Emulator
  45. 45. Mobile Application Testing Tutorial iPad Emulator Copyright 2014 Coveros, Inc. All rights reserved. Page 41 iPad Emulator Android Emulator 66© Copyright 2013 Coveros, Inc.. All rights reserved. Emulator iPad Emulator 67© Copyright 2013 Coveros, Inc.. All rights reserved. Emulator Android Emulator
  46. 46. Mobile Application Testing Tutorial What is Cloud Testing? Copyright 2014 Coveros, Inc. All rights reserved. Page 42 What is Cloud Testing? Sauce Labs 68© Copyright 2013 Coveros, Inc.. All rights reserved. Cloud Testing What is Cloud Testing?  Cloud Testing platforms allow testers and developers to run their applications on cloud-based emulators and simulators in virtual environments.  This eliminates the need to maintain emulator, simulators for all your testing/development teams.  You can easily test on all types of OS versions and device models.  Some cloud-based tools allow mobile teams to receive real- time alerts against any measurement criteria, allowing them to address issues before they impact end-users  There are often APIs that allow automated testing through these platforms. 69© Copyright 2013 Coveros, Inc.. All rights reserved. Cloud Testing Sauce Labs
  47. 47. Mobile Application Testing Tutorial What is Device Testing? Copyright 2014 Coveros, Inc. All rights reserved. Page 43 What is Device Testing? Crowdsourcing 70© Copyright 2013 Coveros, Inc.. All rights reserved. Physical Device Testing What is Device Testing?  Testing on a real, physical device with full functionality to the tester.  Physical device testing will always provide the most accurate testing results, but is also the most costly.  Without access to the real processor and hardware to understand the quirks that some real device have, testers may not discover some real life issues that do not present themselves in emulators 71© Copyright 2013 Coveros, Inc.. All rights reserved.  Crowdsourcing is the practice of obtaining testing services by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees, outsourcing or suppliers.  It combines the efforts of numerous self-identified volunteers or part-time workers, where each tester of their own initiative adds a small portion to the greater testing efforts.  Crowdsourcing is different from outsourcing in that the work comes from an undefined public rather than being commissioned from a specific, named group. Crowdsourcing
  48. 48. Mobile Application Testing Tutorial What are Remote Device Access (RDA) Service? Copyright 2014 Coveros, Inc. All rights reserved. Page 44 What are Remote Device Access (RDA) Service? Which Platform is Best for My Mobile Testing? 72© Copyright 2013 Coveros, Inc.. All rights reserved. Remote Device Access What are Remote Device Access (RDA) Service?  RDA services enable access to a live device over the Internet. As a Mobile Apps Tester, you should be aware of such services and should suggest your managers about the capabilities of such services.  Due to the large number of devices available in the market, it is not feasible to buy a new device every time. That makes RDA a much more cost effective solution. 73© Copyright 2013 Coveros, Inc.. All rights reserved. Recommendations Which Platform is Best for My Mobile Testing? Type of Testing Browser Emulator Device RDA Cloud Emulator Debugging x x x Feature Testing x x x Usability Testing x x x Exploratory Testing x x x x x Performance Testing x x x x x Automated Testing x x x UAT x x Security Testing x x x x x Pre-Prod Testing x x
  49. 49. Mobile Application Testing Tutorial Mobile Testing Techniques Copyright 2014 Coveros, Inc. All rights reserved. Page 45 Mobile Testing Techniques Agenda 74© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Testing Techniques 75© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda – Exploratory Testing – Device Access – Security Testing – Performance Testing – Compatibility Testing – Submission Guideline Testing
  50. 50. Mobile Application Testing Tutorial Mobile Testing Techniques Copyright 2014 Coveros, Inc. All rights reserved. Page 46 Mobile Testing Techniques Exploratory Testing 76© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Testing Techniques Test Management Exploratory Testing Service/API Testing Device Access Security Testing Back-End Testing Mobile UI Automation Performance Testing Submission Guidelines Testing Compatibility Testing 77© Copyright 2013 Coveros, Inc.. All rights reserved. Exploratory Testing  Exploratory Testing is still the most widely used mobile testing technique  Necessary for some validation  Frequent key in outsourcing and crowdsourcing scenarios  Advantages – Tester can assess performance of application under actual operating conditions  Determination/Evaluation of Performance  Determination/Evaluation of Ease of Use  Identification of unexpected behaviors and errors  Does it feel right? – Test-As-You-Fly  A Goddard Space Flight Center Technical Standard which essentially means that you test the system under test as it is intended to be used. Mobile Testing Techniques
  51. 51. Mobile Application Testing Tutorial Standard Mobile App States Copyright 2014 Coveros, Inc. All rights reserved. Page 47 Standard Mobile App States Interruption Testing 78© Copyright 2013 Coveros, Inc.. All rights reserved.  App Store – Upload – Download – Update Broadcasting – Monetization scenarios  Installation and Removal – Multiple Modes – Verification of App Logo and Splash  Operation – Start/Stop – Pause/resume – Credentials – Auto-Start  Operational Modes – Standalone – Offline – Wi-Fi – Cabled – Airplane Mode Standard Mobile App States Exploratory Testing 79© Copyright 2013 Coveros, Inc.. All rights reserved.  Incoming Call  SMS Received  Voicemail  Email  Social media notifications  Alarms  Calendar reminders  Battery low  Battery Critical  Power Sleep Cycle  Battery Removed  Battery Charging  Low Memory  Device Switched Off  Device Reboot  Network outage and Recovery  Bluetooth Connection request  Phone/System Updates  Data Transfer Cable Insertion  Camera Activation  Home Button  Search Button  Volume Control Changes  Screen Lock Interruption Testing Exploratory Testing
  52. 52. Mobile Application Testing Tutorial User Interface Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 48 User Interface Testing User Inputs 80© Copyright 2013 Coveros, Inc.. All rights reserved.  Layouts, alignment, colors, fonts – Views/Tables – buttons, Menus, Links, Pickers, Toolbars – Alerts/Messages – Themes, Color Schemes, Styles – Brightness Settings  UI Elements – Position – Size – Inputs – Actions  Images and Symbols – Clarity – Alignment – Density (load performance)  Error Message Displays  Consistency – Across the application – Across various devices User Interface Testing Exploratory Testing 81© Copyright 2013 Coveros, Inc.. All rights reserved.  Gestures – Tap – Press/Hold – Double Tap – Drag – Pinch – Spread – Swipe/Flick – Rotation – Shake – Tilt User Inputs Exploratory Testing http://uxdesign.smashingmagazine.com/2011/10/06/not-your-parents-mobile-phone-ux-design-guidelines-smartphones/
  53. 53. Mobile Application Testing Tutorial Accessibility and Usability Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 49 Accessibility and Usability Testing Slide 82 Slide 83 82© Copyright 2013 Coveros, Inc.. All rights reserved.  Usability – Efficiency – Accuracy – Recall – Emotional Response  Accessibility – Screen Size – Lighting Conditions and brightness settings – Multi-screen capability and compatibility – Touch Interface size Accessibility and Usability Testing Exploratory Testing 83© Copyright 2013 Coveros, Inc.. All rights reserved.  Workflows – Information Presentation – Ease of Task Completion – Collaborative Task Completion (Sequential and Parallel)  Application Integration and Interactions – Native Application/Device – Web Application/Device – Hybrid Application/Device/Web App Accessibility and Usability Testing Exploratory Testing
  54. 54. Mobile Application Testing Tutorial Mobility Copyright 2014 Coveros, Inc. All rights reserved. Page 50 Mobility Connectivity 84© Copyright 2013 Coveros, Inc.. All rights reserved.  Geo aware (GPS, Tower Triangulation)  Accelerometer (Motion, Gestures)  Magnetometer (Compass)  Traversing zones/cells/networks – GPS Signal fade/interference – WiFi Signal fade/interference – Cellular Signal fade/interference Mobility Exploratory Testing 85© Copyright 2013 Coveros, Inc.. All rights reserved.  Near Field Communication – Android Beam – Bluetooth – Wi-Fi  Networks – Variable Networks Data throughput support – Variable network connectivity protocol types  4g/3g/analogue  1G, 2G, 3G, 4G  Wi-Fi, Wi-Fi/3G, Wi-Fi/4G) – Signal Strength  No Network  Intermittent connectivity  Strong Signal  Varying signal strength – Trust level settings (Security/Data Exposure) Connectivity Exploratory Testing
  55. 55. Mobile Application Testing Tutorial Usability: Deadly Application Failures Copyright 2014 Coveros, Inc. All rights reserved. Page 51 Usability: Deadly Application Failures Testing Device Sensors 86© Copyright 2013 Coveros, Inc.. All rights reserved. Exploratory Testing Usability: Deadly Application Failures • Insufficient features and functionality • Consumption of resources (Battery, CPU, Storage) • Failure to operate well under sub-optimal connectivity situations • Slow Performance • Interoperability • doesn’t integrate with native device features • causes errors in other existing applications • Difficult to use 87© Copyright 2013 Coveros, Inc.. All rights reserved.  Rotation/Tilt – Not supported (user expectations problem) – Could cause screen res/layout issues (switching from Portrait to Landscape mode) – Other unexpected results  Movement/Shake Inputs – Not supported (user expectations problem) – Unexpected results  Does jogging on a treadmill cause song over-shuffling on iPod?  Geo-Location Capabilities – Verify location mapping correct – Verify location capabilities while in motion – Verify lack of freeze-ups while in motion – Verify re-location upon loss/re-gain of signal  Magnetometer/Rotation – Verify Compass direction correct  Light – Automatic brightness adjustments  Heat – Device function shutdown, reduced performance Testing Device Sensors Device Access Testing
  56. 56. Mobile Application Testing Tutorial Security Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 52 Security Testing Performance Testing 88© Copyright 2013 Coveros, Inc.. All rights reserved. Security Testing  Network Attacks – Traffic Analyzers  Server Side Attacks – API/Service Attacks – Back-end Server – Database Attacks  Client-Side Attacks – Cross Site Scripting – Data Privacy Exploitation – Buffer Overflows  Injection and Data Flow Analysis  Jail-breaking/Rooting  Penetration Testing Mobile Testing Techniques 89© Copyright 2013 Coveros, Inc.. All rights reserved. Performance Testing  Mobile apps are often supposed to scale to massive volumes. We have to delve into lots of different realistic scenarios for performance over more than just WI-FI.  Performance testing should cover real world mobile networks (Not all networks are created equal). – 4G LTE, 3G, 2G, EDGE, GPRS  Don’t forget to test middle-tier and back-end for performance  If you’re not testing your mobile app out in the real world like a client would experience, your performance testing is not complete Mobile Testing Techniques
  57. 57. Mobile Application Testing Tutorial Performance and Stress Copyright 2014 Coveros, Inc. All rights reserved. Page 53 Performance and Stress Compatibility Testing 90© Copyright 2013 Coveros, Inc.. All rights reserved.  Low resources – Optimization Modes – Power saving Modes – Variable Battery Levels (Full, Low, Critical, Charging) – Memory/CPU utilization  Launch, Background, Foreground, Exit  Simultaneous Applications utilization  Continuous Keypad entry (across various keypad controls/events)  Memory Sweeping – Storage Space  Installation  Application Utilization – Battery Consumption  Server Load – Stress testing  Multi-user  Multi-functional – Backups  Crowd-Source Testing  Network Testing – Internetwork – Network  Roaming – Wife  Cell Performance and Stress Mobile Testing Techniques 91© Copyright 2013 Coveros, Inc.. All rights reserved.  Testing for Compatibility:-Testing the compatibility of your application with native device features (i.e. To make sure your application is not hampering native device functionality)  Platform  Device  Other Applications  Application backwards compatibility  Browsers  Networks Compatibility Testing Mobile Testing Techniques
  58. 58. Mobile Application Testing Tutorial Submission Guideline Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 54 Submission Guideline Testing Slide 92 Just an excerpt from the Apple Guidelines: 2. Functionality 1. Apps that crash will be rejected 2. Apps that exhibit bugs will be rejected 3. Apps that do not perform as advertised by the developer will be rejected 4. Apps that include undocumented or hidden features inconsistent with the description of the app 5. will be rejected 6. Apps that use non-public APIs will be rejected 7. Apps that read or write data outside its designated container area will be rejected 8. Apps that download code in any way or form will be rejected 9. Apps that install or launch other executable code will be rejected 10. Apps that are "beta", "demo", "trial", or "test" versions will be rejected 11. iPhone apps must also run on iPad without modification, at iPhone resolution, and at 2X iPhone 12. 3GS resolution 13. Apps that duplicate apps already in the App Store may be rejected, particularly if there are many 14. of them 15. Apps that are not very useful or do not provide any lasting entertainment value may be rejected 16. Apps that are primarily marketing materials or advertisements will be rejected 92© Copyright 2013 Coveros, Inc.. All rights reserved. Submission Guideline Testing  Google Play Store – Quick review – Simple standards to meet – http://play.google.com/about/developer-content-policy.html  Apple App Store – Notoriously difficult standards – Long review time – Stringent Requirements – https://developer.apple.com/appstore/guidelines.html Mobile Testing Techniques
  59. 59. Mobile Application Testing Tutorial Submission Guideline Testing Copyright 2014 Coveros, Inc. All rights reserved. Page 55 17. Apps that are intended to provide trick or fake functionality that are not clearly marked as such 18. will be rejected 19. Apps larger than 20MB in size will not download over cellular networks (this is automatically 20. prohibited by the App Store) 21. Multitasking apps may only use background services for their intended purposes: VoIP, audio 22. playback, location, task completion, local notifications, etc. 23. Apps that browse the web must use the iOS WebKit framework and WebKit JavaScript 24. Apps that encourage excessive consumption of alcohol or illegal substances, or encourage 25. minors to consume alcohol or smoke cigarettes, will be rejected 26. Apps that provide incorrect diagnostic or other inaccurate device data will be rejected 27. Developers "spamming" the App Store with many versions of similar apps will be removed from 28. the iOS Developer Program Title Month Year
  60. 60. Mobile Application Testing Tutorial Android Core Quality Requirements Categories Copyright 2014 Coveros, Inc. All rights reserved. Page 56 Android Core Quality Requirements Categories Apple App Store Requirements Categories 93© Copyright 2013 Coveros, Inc.. All rights reserved.  Functionality – Permissions – Installation – Audio – UI and Graphics – User / App State  Performance and Stability – Stability – Performance – Media – Visual Quality  Google Play – Policies – App details page – User Support Android Core Quality Requirements Categories Submission Guideline Testing 94© Copyright 2013 Coveros, Inc.. All rights reserved.  Functionality  Metadata  Locations  Push Notifications  Game Center  iAds  Trademarks and trade dress  Media Content  User Interface  Purchasing and Currencies  Scraping and Aggregation  Device Damage  Personal Attacks  Violence  Objectionable Content  Privacy  Pornography  Religion, Culture, and ethnicity  Contests, sweepstakes, lotteries, and raffles  Charities and Contributions  Legal Requirements Apple App Store Requirements Categories Submission Guideline Testing
  61. 61. Mobile Application Testing Tutorial Tools Copyright 2014 Coveros, Inc. All rights reserved. Page 57 Tools Agenda 95© Copyright 2013 Coveros, Inc.. All rights reserved. Tools 96© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda  Introduction  Multi-platform Testing Tools  Android Testing Tools  IOS Testing Tools  Blackberry testing tools  Windows Mobile Testing Tools  Remote Device Access Services  Outsource/Crowd Source Testing Services
  62. 62. Mobile Application Testing Tutorial Multi-platform Test Tools Copyright 2014 Coveros, Inc. All rights reserved. Page 58 Multi-platform Test Tools Multi-Platform Test Tools 97© Copyright 2013 Coveros, Inc.. All rights reserved. Selenium 2 (WebDriver)  Can run on emulators and devices  Native support exists for Android, iPhone and iPad but is deprecated  Third party drivers have been developed to fill in these gaps – iOS-Driver – Selendroid – Appium  Support for all major browsers – Browsers downloaded and installed on devices to supplement native browsers can be fully tested with WebDriver. Multi-platform Test Tools Testing Tools 98© Copyright 2013 Coveros, Inc.. All rights reserved. Gomez, mobiReady, and W3C mobileOK Checker  Tests web based apps and evaluates their mobile performance.  Examines mobile best- practices and usability, and provides steps to correct  Web-based testing, no download or install needed Multi-Platform Test Tools Testing Tools
  63. 63. Mobile Application Testing Tutorial Android Tools Copyright 2014 Coveros, Inc. All rights reserved. Page 59 Android Tools Android Debugging Tools 99© Copyright 2013 Coveros, Inc.. All rights reserved.  Monkey Runner – Runs on simulator or device – Generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events. – Can run an automated start-to-finish test of an Android application. You provide input values with keystrokes or touch events, and view the results as screenshots.  Robotium – Support for native and hybrid applications – Can write function, system and acceptance test scenarios, spanning multiple Android activities – Integrates smoothly with Maven or Ant to run tests as part of continuous integration. Android Tools Testing Tools 100© Copyright 2013 Coveros, Inc.. All rights reserved.  Dalvik Debug Monitor Server (DDMS) – Port-forwarding services – Screen capture on the device – Thread and heap information on the device  Logcat – Process, and radio state information – Incoming call and SMS spoofing – Location data spoofing and more  TaskKiller – A utility application used to kill process – Useful to skip device reboot. Android Debugging Tools Testing Tools
  64. 64. Mobile Application Testing Tutorial iOS Tools Copyright 2014 Coveros, Inc. All rights reserved. Page 60 iOS Tools Blackberry Tools 101© Copyright 2013 Coveros, Inc.. All rights reserved.  UIAutomation – Lets you run an automated set of tests against an application – Can be used to perform automated tests on real devices and on the iPhone Simulator – It enables you to quickly track regressions and performance issues  NUnit – Unit level testing for iOS apps  Test Studio for iOS – Lets developers test native iOS apps as well as hybrid and Web apps – The tool can test common iOS controls and record test scripts for gestures such as taps, multiple finger gestures such as zooms, swipes and taps.  iPhone test automation – Fonemonkey – UISpec – Bromine (not recommended) iOS Tools Testing Tools 102© Copyright 2013 Coveros, Inc.. All rights reserved. Testing Tools  BlackBerry device simulator – “Fledge”  Jamo Solutions – “M-eux Test”  Borland Silk Mobile Blackberry Tools Max1
  65. 65. Mobile Application Testing Tutorial Windows Tools Copyright 2014 Coveros, Inc. All rights reserved. Page 61 Windows Tools Remote Device Access Services 103© Copyright 2013 Coveros, Inc.. All rights reserved.  MSDN Hopper Test Tool – Simulates input stress on Windows Mobile devices – Hopper will stress all applications that are available through the menu system by rapidly sending keystrokes and screen taps in a random fashion  MSDN Device Emulator – Mimics the behavior of Windows Mobile – Provides a virtual hardware platform that you can use to test applications on multiple virtual devices – Runs code compiled for ARM microprocessors  MSDN Cellular Emulator Windows Tools Testing Tools 104© Copyright 2013 Coveros, Inc.. All rights reserved.  DeviceAnywhere  Perfecto Mobile  Keynote  Paca Mobile Center  pCloudy Remote Device Access Services Testing Tools
  66. 66. Mobile Application Testing Tutorial Security Testing Tools Copyright 2014 Coveros, Inc. All rights reserved. Page 62 Security Testing Tools Performance Testing Tools 105© Copyright 2013 Coveros, Inc.. All rights reserved.  Kryptowire (Native App Testing Tool)  OWASP ZAP (Mobile App Testing Tool)  For more information and training on Security Testing Tools, check out the Security Testing Course from SQE.  Additional resources at owasp.org Security Testing Tools Testing Tools 106© Copyright 2013 Coveros, Inc.. All rights reserved.  NeoLoad  BlazeMeter  SOASTA  Mobitest Performance Testing Tools Testing Tools
  67. 67. Mobile Application Testing Tutorial Crowd Source Testing Services Copyright 2014 Coveros, Inc. All rights reserved. Page 63 Crowd Source Testing Services Outsourced Mobile App Testing Services 107© Copyright 2013 Coveros, Inc.. All rights reserved.  GetJar  Mob4Hire  uTest  Ubertesters  App Lover  Apphance  Beta Bait  Bugcrowd Crowd Source Testing Services Testing Tools 108© Copyright 2013 Coveros, Inc.. All rights reserved.  Coveros  Mobica  Intertek  RapidSoft  Belatrix  Ranorex  HP  TestQuest  Utopia Solutions  Wipro Outsourced Mobile App Testing Services Testing Tools
  68. 68. Mobile Application Testing Tutorial Wrap-Up Copyright 2014 Coveros, Inc. All rights reserved. Page 64 Wrap-Up Agenda 109© Copyright 2013 Coveros, Inc.. All rights reserved. Wrap-Up 110© Copyright 2013 Coveros, Inc.. All rights reserved. Agenda  Wrap-Up  Recommendations  Questions
  69. 69. Mobile Application Testing Tutorial Mobile Testing Books we recommend Copyright 2014 Coveros, Inc. All rights reserved. Page 65 Mobile Testing Books we recommend Questions? 111© Copyright 2013 Coveros, Inc.. All rights reserved. Mobile Testing Books we recommend  “Tap Into Mobile Application Testing” Kohl, Jonathan  “A Practical Guide to Testing Wireless Smartphone Applications” Harty, Julian  “Mobile Application Security” Dwivedi, Himanshu  “Mobile Developer’s Guide To The Galaxy” http://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Mobile_De vGuide_13.pdf 112© Copyright 2013 Coveros, Inc.. All rights reserved. Questions? Contact Information: training@coveros.com
  70. 70. Mobile Application Testing Tutorial Notes Copyright 2014 Coveros, Inc. All rights reserved. Notes

×