Mobile Testing Methodologies: Trends, Successes, and Pitfalls


Published on

In today's dynamic mobile marketplace—where new handsets and mobile operating systems are released every day—your ability to deal with these changes which impact your mobile product is vital. The mobile application lifecycle today must be short; must be of great quality; cover a myriad of handsets with different sizes, layouts, and enhanced capabilities; and, of course, cover as many operating systems as possible. This lifecycle requires a new methodology and approach. Eran Kinsbruner describes the mobile project challenges and provides real life examples of ways to overcome them. Take back the main mobile market trends and forecasts together with the key automation tools available for your use today. Learn the differences between the various mobile cloud and automation tools to help you select the right tool for your project. See how you can ramp up a successful mobile project, avoid the common pitfalls, and shorten the time to market—all while delivering a top-notch quality product.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mobile Testing Methodologies: Trends, Successes, and Pitfalls

  1. 1. T10 Mobile Testing 5/2/2013 11:15:00 AM Mobile Testing Methodologies: Trends, Successes, and Pitfalls Presented by: Eran Kinsbruner Perfecto Mobile Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Eran Kinsbruner Eran Kinsbruner is director of product marketing at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at
  3. 3. 4/16/2013 Mobile Testing Methodologies Trends, Successes, and Pitfalls StarEast - 2013 Enabling Enterprise mobility Extending Software Quality to Mobile Perfectomobile Mobile Testing Center of Excellence Group Agenda About me and my company Mobile Market Trends Mobile Quality Trends Mobile Quality Challenges Mobile Risk Management Mobile Testing Best Practices Perfecto Mobile 1
  4. 4. 4/16/2013 ∙ ∙ ∙ ∙ ∙ Founded 2006 80+ employees Office in Boston, London, Israel Over 600 customers Fortune 500 enterprises, Telcos, SMBs ∙ SaaS Platform + Licenses ∙ Recurring revenue model Mobile Application Quality across the mobile application lifecycle Monitoring Testing Automation ALM Integration Offshoring Continuous Integration Public Cloud Private Cloud Mobile Market Trends Perfecto Mobile 2
  5. 5. 4/16/2013 Global Mobile Market Stats Source: nielsen, 2012 Mobile OS market expected to grow (even more) FireFox OS PEAK KEON Tizen Windows Phone Samsung/Intel New Mobile OS’s RIM’s BlackBerry OS 10 Ubuntu Z10 & Q10 SailFish By Jolla Perfecto Mobile 3
  6. 6. 4/16/2013 Cost Of Mobile Quality Rises Mobile accounts for 11% of U.S e-Commerce Existing Mobile OS’s – End of 2012 Source:IDC Samsung accounted for 63 Million out of the 159 Million in Q42012 Perfecto Mobile 4
  7. 7. 4/16/2013 Worldwide smartphone vendors Market share Source: IDC Worldwide Mobile Phone Tracker, January 24, 2013 Market insights – Forecasts and trends Mobile based shopping expected to grow by 24% in 2013 (U.S) Perfecto Mobile 5
  8. 8. 4/16/2013 Market trends - 2013 Mobile OS battle continues •New open source competitors trying to influence the eco system •Windows Phone and BlackBerry OS 10 – critical year for their positioning in the market Mobile apps become more complex – impact testing and automation HTML5 and Mobile web expected to grow significantly EMM establishment – Processes, Apps, Tools, Roles (CMOO) Mini-tablets and “Phablets” will successfully blur the lines between device types Mobile Quality Trends Perfecto Mobile 6
  9. 9. 4/16/2013 World Quality report 2012-2013 2012Mobile Testing is Behind the Curve Source: CapGemini Mobile Environment Fragmented Dynamic Complex Business critical Source: Google, 2012 Perfecto Mobile 7
  10. 10. 4/16/2013 Countless screen resolutions Source: opensignalmaps, 2012 (Android only) Consumer Reaction – Less Tolerant than In Desktop Perfecto Mobile 8
  11. 11. 4/16/2013 HTML5 Trend Involves Risks To help protect y our priv acy , PowerPoint prev ented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content. Mobile Quality Challenges Perfecto Mobile 9
  12. 12. 4/16/2013 Mobile Introduces various challenges Performance Automation AGILE ALM Functional Native Hybrid Web Application Types On various browsers Sample Screen Sizes (Only Smartphone) QVGA: quarter VGA (240×320 pixels) HVGA: half VGA (320×480 pixels) WVGA: wide VGA (480×800 pixels) FWVGA: full wide VGA (480×854 pixels) nHD: one-ninth high definition (360×640 pixels) qHD: one-quarter high definition (540×960 pixels) Various Device/Tablets OEM’s Various OS Types and Versions 2.3.3, 4.0.3 4.3.3, 5.1 4.1.2, 4.2.2 6.1, 6.1.1 Mobile Security Threats Source: Techcrunch Perfecto Mobile 10
  13. 13. 4/16/2013 Mobile OS Regression bugs – What does it have to do with my Mobile Testing? Per Device version Pre-release testing (iOS6.1.1 Beta) Apple Bug Reporter Source: LINK Android defect database Mobile Web Objects – Automation Challenges Perfecto Mobile 11
  14. 14. 4/16/2013 Apple iOS – UI Guidelines Source: Source: eHIG/Introduction/Introduction.html Android OS – UI Guidelines Source: Perfecto Mobile 12
  15. 15. 4/16/2013 Risks Management Mobile Quality Risk Analysis Cost of quality in mobile grows and impacted by • Availability • TTM • Functionality and UX • New platform support (Mobile OS, Mobile devices) • Previous app version compatibility Perfecto Mobile 13
  16. 16. 4/16/2013 Mobile end-user Less Tolerant end- Listen to the market Obviously someone forgot to do their regression testing on a Galaxy S3 before releasing this version to market… Perfecto Mobile 14
  17. 17. 4/16/2013 Device governance A clear process as part of the “mobile project” lifecycle Define platforms and categories Define coverage Acceptable not to test - You can’t test everything Define change frequency Adapt to the market trends as they happen Define SLA for changes Define coverage as part of the ALM phase Coverage visibility reports to project managers and executives Device Governance is a risk management for mobile projects Device Governance – Define coverage You can’t cover everything But you should define a target, which is the acceptable risk iOS is not as simple as you think At least 3 HW platforms (3GS, 4, 4S, 5) (iPADs, iPODs) At least 3 SW platforms Perfecto Mobile prev major 15% prev minor 25% Other 5% Latest 55% 15
  18. 18. 4/16/2013 Enterprises OWNS the risk for Mobile quality Mobile Malware - Trojans, viruses and rootkits migrating from traditional online banking and designed specifically for the mobile marketplace. Researchers see an increase in mobile malware development - in pace with market growth. ThirdThird-Party Apps - Consumers love their smart phone and tablet applications, but often these apps come from third parties with questionable security practices. Or worse, the apps are created by fraudsters and loaded with malware. Unsecured Wi-Fi - The unsecured wireless network is a toll-free highway for fraudsters to gain Witollaccess to mobile devices, either to seize control of or gain access to account information. devices, information. User Behavior - Consumers are prone to download third-party apps, use unsecured wireless thirdapps, networks, open and click links in SMS text messages and e-mails, and lose their mobile devices. edevices. Mobile testing and Hardware (Payment card readers, healthcare units and more) Uncompromised necessity for testing mobile apps Should adhere to regulations (GLBA/FFIEC, RDC, FCRA, PCI DSS) around security, GLBA/FFIEC, RDC, FCRA, DSS) authentications, payment and more Usually support complex business work flows (deposit check via camera) Multi tier interactions (Databases, trading accounts, other services) High Availability and Monitoring Must address mobile end user actions Using Home and Back button during application run Network lose during transaction Roaming during user transaction Application behavior in various networks 3G, 4G, WiFi DSS Compliance test case example Perfecto Mobile 16
  19. 19. 4/16/2013 Mobile Is Mandated To Strict Regulations Source: LINK Source: Link Mobile Testing Best Practices Perfecto Mobile 17
  20. 20. 4/16/2013 This is not a mobile test lab! 10 Devices 3 Extra Devices Many Cables 3 Workstations One Chair Building Blocks Of Mobile Testing Strategy Choose the right tools and platform Understand your users device and platform choices Optimize device and platform testing Automate as much as possible using device agnostic scripting Use mobile cloud based solutions (Collaborative, Governed, Scalable, Cost Effective) Use CD/CI solutions as part of your mobile ALM Do not re-invent the wheel reExtend existing ALM tools to Mobile. Perfecto Mobile 18
  21. 21. 4/16/2013 World Quality report 2012-2013 2012- Source: CapGemini Building Blocks Of Mobile Testing Strategy Consider outsource to reduce mobile manual testing costs Test for performance before releasing to the market Make sure you do regression on older supported platforms Do not have your end users do your QA work Perform pro-active testing on new devices and mobile OS’s Monitor in production to assure end user experience and availability Listen to your end users and act upon their feedback (Markets have significant impact on your business and branding) Perfecto Mobile 19
  22. 22. 4/16/2013 (True) Mobile Cloud Avoid device management Real Devices + Emulators Real smartphones Manual + Automation ALM Integration Device agnostic Core level extension End-toEnd-to-End Mobile ALM Development Functional Real Tablets Performance Secure, Centralized Interactive Emulators Short learning curve Monitoring Perfecto Mobile 20
  23. 23. 4/16/2013 END Perfecto Mobile 21