Your SlideShare is downloading. ×
Mobile Testing in the Cloud
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Mobile Testing in the Cloud

260
views

Published on

Today, organizations are rapidly deploying mobile versions of their customer-facing and internal applications. With the prevalence of more agile-based approaches and the challenge of an …

Today, organizations are rapidly deploying mobile versions of their customer-facing and internal applications. With the prevalence of more agile-based approaches and the challenge of an ever-increasing diversity of devices and OS versions, testers are being asked to accomplish more testing in less time. Rachel Obstler shares how leading enterprises are improving the efficiency of their mobile testing using automation, and how they identify the right processes and tools for the job. Sharing some fascinating statistics from their recent mobile quality survey of more than 69,000 mobile app developers and QA organizations in the top US enterprises, Rachel dives into the challenges identified in the survey and shares five clear ways to improve your testing process: implementing a collaborative agile process, optimizing with a development tool that naturally facilitates testing, using a combination of real and emulated devices—and when to use them, and more.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
260
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. T16 Mobile Testing 5/8/2014 1:30:00 PM Mobile Testing in the Cloud Presented by: Rachel Obstler Keynote DeviceAnywhere Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. Rachel Obstler Keynote DeviceAnywhere Rachel Obstler is the senior director for Keynote's DeviceAnywhere platform. Rachel has more than ten years of experience in the wireless industry, working for companies ranging in size from startups to multinational corporations. She most recently managed the wireless data quality products at Telephia Inc. and has experience managing the products of various wireless technology companies, including Metawave Communication (smart antenna technology) and Enuvis (location technology). Rachel spent three years working with Lucent Technologies, where she managed and launched a billion dollar product line of CDMA Base Stations.
  • 3. 4/26/2014 1 Mobile Testing in the Cloud Helping QA organizations manage the challenges of a mobile-first world Rachel Obstler May 8, 2014 Contents The Mobile Imperative The Challenge of Mobile Quality Solutions Optimizing Your Device Testing Strategy Automation 1
  • 4. 4/26/2014 2 The Mobile Imperative DOWNLOADED IN 20131 102BILLION APPS 77BILLION ANTICIPATED REVENUE GENERATED THROUGH MOBILE APPS BY 20171 #1 PRIORITY: MOBILERETAILER TECHNOLOGY PRIORITIES2 2 OF 3 BANKSPREDICT 100% OF THEIR CUSTOMERS WILL USE MOBILE SERVICES BY 20173 1. Gartner; 2. Forrester, State of Retailing Online, 2014; 3. Metaforic UK 2 The Mobile Imperative 1. Econsultancy; 2. Forbes; 3. ZDNet ONLY 16%CONSUMERS SURVEYED WILLING TO GIVE A PROBLEMATIC MOBILE APP MORE THAN ONE ATTEMPT1 USERS EXPECT A MOBILE APP TO BE RESPONSIVE IN 3 SECONDS 3 A MOBILE FRIENDLY SITE MAKES 2 OF 3 USERSMORE LIKELY TO BUY A COMPANY’S PRODUCT OR SERVICE 2 3
  • 5. 4/26/2014 3 Survey: Quality Expectations of Mobile Websites & Apps Almost half of the respondents think mobile quality expectations are the same as desktop, and 35% think they are higher. 10% 49% 36% 5% Quality expectations for mobile applications and websites are lower than desktop Quality expectations for mobile applications and websites are the same as desktop Quality expectations for mobile applications and websites are higher than desktop Not sure Now when thinking about quality expectations of mobile websites and applications, which of the following is true? (1,314 respondents) 4 Challenge: Mobile = Devices Device platforms, fragmentation, and growth • Multiple OS’s, form factors and screen resolutions • Frequent device refresh New capabilities to test • Camera, GPS, direction, orientation, voice, etc. Network considerations • Multiple carriers, variable throughput and latency, disconnected use, network switching 5
  • 6. 4/26/2014 4 Scan & Certify Instrument Test IntegrateObtain Insight Manage Deploy Design & Develop Challenge: Mobile = Faster, More Iterative, Continuous Activity 6 Challenge: Mobile = Late to the Game Employee Skillsets HP QC/IBM RQM users HP UFT/QTP users Automation engineers/ programmers Manual testers 7
  • 7. 4/26/2014 5 Challenge: Mobile = Spread Across the Organization Organizations Multiple teams Geographically diverse employee base IT security requirements 8 Survey: The Mobile Testing Organization is Largely Decentralized How is mobile website and application testing organized within your company? (# of respondents in each column) All Verticals (1590) High Tech/IT/ Software (661) Fin Svcs & Ins (234) Telecom (151) Media (139) Retail* (57) Centralized testing group, mobile-specific 19.7% 20.3% 20.9% 19.9% 22.3% 17.5% Centralized testing group, not mobile-specific 12.6% 12.0% 12.8% 9.9% 18.7% 10.5% Individual QA groups within business units/divisions, mobile-specific 35.8% 38.1% 36.3% 44.4% 36.0% 36.8% Individual QA groups within business units/divisions, not mobile-specific 20.8% 22.5% 20.5% 13.9% 14.4% 17.5% Testing is predominantly outsourced 3.5% 2.6% 3.4% 2.6% 3.6% 8.8% Currently not testing any mobile apps or websites 7.5% 4.5% 6.0% 9.3% 5.0% 8.8% More testing groups are distributed, rather than in one centralized group. Individual and also centralized groups are more often mobile-specific. * Retail sample size <100. 9
  • 8. 4/26/2014 6 Survey: Challenges of Testing Mobile Websites and Applications Regarding challenges encountered when testing mobile applications and websites, please rate the following challenges on a 1 to 10 scale, where 1 is ‘Not at all challenging’ and 10 is ‘Extremely challenging’. (1,314 respondents) 1 (Not Challenging) 2 3 4 5 6 7 8 9 10 (Extremely Challenging) Availability of mobile testing experts 4.7% 4.3% 5.8% 7.2% 17.4% 11.3% 15.4% 17.0% 8.7% 8.2% Implementing the right testing method/process for mobile 3.5% 3.2% 5.3% 7.4% 17.7% 11.5% 18.0% 18.3% 8.4% 6.5% Availability of proper testing tools 2.7% 3.2% 5.3% 5.4% 16.3% 11.6% 16.4% 16.6% 11.3% 11.3% Access to mobile devices 6.5% 6.3% 4.7% 4.5% 14.2% 10.0% 13.5% 16.7% 9.9% 13.6% Having enough time to test 4.2% 3.4% 5.6% 5.9% 16.7% 12.3% 14.9% 14.9% 10.2% 11.9% 10 Survey: Important Functional Testing Features Q A Within functional testing, what are the main priorities for QA? Easy access to a variety of device models was rated the most important functional testing feature across all verticals. 11
  • 9. 4/26/2014 7 Optimize Your Device Testing Strategy Determine your priority devices Provide easy access to real devices Enable local and remote employees Provide a secure, enterprise-grade test environment 12 Use Data to Optimize Test Resources 13 Internal apps Customer apps Android Screen Size and Density Source: Google Android OS Market Share Source: Google
  • 10. 4/26/2014 8 Real Devices in the cloudYour computer, connected to the internet Your key presses and mouse clicks sent to the device The device’s screen sent back to your computer SaaS-based Real Device Solution 14 Different screen resolution / screen size Unreadable text, blurred images, misalignment of screen elements, and items that fall off the screen. Android Customizations On-screen and physical controls that function differently across devices Customized handling of inputs and events (e.g. Samsung Swype vs. default Android keyboard). Memory / CPU Low or insufficient memory or processing power. Impacts of other services running on the device. Mobile device characteristics that impact the quality of your application or website, but are not easily verifiable using an emulated phone or browser, include: Test on Real Devices? 15
  • 11. 4/26/2014 9 Devices Shared Devices 24X7 access to hundreds of smart devices Use for compatibility testing, “untrusted device” testing Private Devices 24X7 access to your devices Enables geographically diverse team Inside corporate firewall or external Local Devices Plug your smart devices on hand directly into your local computer Make use of existing assets, great for local teams Use a Combination of Local and Remote Devices 16 Deploy an Enterprise-grade Product 17 Access and Project Management Multiple levels of user permissions Control access to all assets, including devices, scripts and test results Create different groups to manage access Security Password protection features (enforce change; salted; disable save) SSL communication LDAP integration
  • 12. 4/26/2014 10 0 Automate! 18 Determine target test cases to automate Support agile processes (continuous integration) Find a cross-device testing tool Make use of existing (desktop) automation tools and processes where possible Provide multiple scripting options because your team is not one size fits all Test Automation is Not Yet Widely Used What percentage of your mobile web regression tests is currently automated? (# of respondents in each column) All Verticals (1176) High Tech/IT/ Software (520) Fin Svcs & Ins (183) Telecom (106) Media (96) Retail* (45) 0% 28.8% 27.1% 26.8% 17.9% 28.1% 35.6% 1-25% 28.7% 31.3% 25.1% 33.0% 29.2% 24.4% 26-50% 17.3% 19.0% 19.7% 15.1% 14.6% 17.8% 51-75% 9.8% 11.0% 8.2% 15.1% 10.4% 2.2% >75% 4.9% 3.1% 5.5% 5.7% 4.2% 11.1% Don't know 10.5% 8.5% 14.8% 13.2% 13.5% 8.9% Majority or organization have less than 25% of mobile web regression tests automated. * Retail vertical sample size <100. 19
  • 13. 4/26/2014 11 Mobile Apps – Same Story What percentage of your mobile native applications regression tests is currently automated? (# of respondents in each column) All Verticals (1172) High Tech/IT/ Software (519) Fin Svcs & Ins (183) Telecom (105) Media (95) Retail (45) 0% 30.4% 28.3% 27.9% 20.0% 32.6% 46.7% 1-25% 27.2% 29.7% 26.2% 34.3% 22.1% 17.8% 26-50% 15.4% 18.9% 15.8% 11.4% 8.4% 11.1% 51-75% 9.2% 8.7% 10.4% 13.3% 9.5% 6.7% >75% 4.6% 4.2% 3.8% 5.7% 6.3% 8.9% Don't know 13.1% 10.2% 15.8% 15.2% 21.1% 8.9% * Retail vertical sample size <100. 20 Determine Your Targets for Automation Goals Test Cases Frequency Methodology Basic acceptance, build acceptance Functional testing across all existing areas of product Compatibility across devices Very high value to automate! High value to automate Medium value to automate # of Devices Optimal Test Strategy Automated Testing Automated Testing Manualor AutomatedTesting Smoke Testing Regression Testing Compatibility Testing Deep testing of new areas of functionality Low value to automate ManualTesting New Feature Testing Not automated Exploratory Testing Exploration of functionality from customer viewpoint ManualTesting 21
  • 14. 4/26/2014 12 Test Continuous Integration Feedback Loop 22 Develop Build Customer Case Study: Speed Time to Market Each build of the native mobile retail investment app required a Build Acceptance Test with 300 hundred test cases across two devices (one Android and one iOS) Run manually, tests were taking two QA engineers two weeks (30 test cases per engineer per day) Engineering did not receive any build feedback until two weeks had passed QA team had QTP skills, and managed the process using QC Automated test cases using DeviceAnywhere and QTP 300 test cases executed across two devices (in parallel) now take 24 hours to run, vastly reducing time to market Manual testers freed up to test new or more complex features All results saved in QC BACKGROUND RESULT: Automation on real devices resulted in 10X Reduction in BAT time! CUSTOMER Director of QA at a major financial institution 23
  • 15. 4/26/2014 13 Device Fragmentation Less up front scripting cost Reduce ongoing script maintenance Object-Level Scripting Important Cross-Device Testing 24 Customer Case Study: Improve Product Quality The current mobile test process for the consumer mobile banking app involved outsourcing testing to a manual test house. The available budget afforded them enough person/hours to complete their 900 regression test cases on only one device Results and issues were managed using unwieldy word documents As usage picked up, issues in the field highlighted the need to ensure quality across a variety of mobile platforms and devices BACKGROUND RESULTS CUSTOMER VP of Engineering at a major bank Now the same 900 test cases are run across 20 devices, improving test coverage and quality All 20 devices tested in half the time it took previously to test one device, improving time to market Test results are automatically available online with screenshots (no more word docs, improved process) 25
  • 16. 4/26/2014 14 Survey Your Existing Test Processes ALM Tools (HP ALM, IBM Rational) Automation Tools such as HP QTP Open Source Tools like Selenium, Robotium Other tools 26 What desktop test tools is your organization currently using? Survey: Important Functional Testing Features by Vertical Integration with ALM (Application Lifecycle Management) tools were also highly rated for financial services companies. Having an ‘integration with open source tools’ (such as Selenium, Robotium, etc.) was rated very high for retail companies as they traditionally have been more e-commerce focused and would often be developing automation using open source web-based tools. Were there any other vertical-specific challenges that also rated high? Q 27
  • 17. 4/26/2014 15 Beginning and Advanced Scripting Options Enable All Employees 28 UI-Based Scripting For complex logic and large test suite design and execution Integration with Leading Test Tools Allow employees with existing skillsets to easily extend to mobile Java API Enables your QA engineers with programming skills to write test in Java Script Recording Point and click recorder enables anyone to immediately create reusable test scripts Automation Engineers UFT/QTP users Programmers New Scripters Challenges 29 Mobile = Devices Mobile = Faster, More Iterative, Continuous Activity Mobile = Late to the Game Mobile = Spread Across the Organization
  • 18. 4/26/2014 16 Solutions Challenge Solution(s) Mobile = Devices Use data to determine priority devices Provide access to real devices in the cloud Automate using a cross-device testing tool Mobile = Faster, More Iterative, Continuous Activity Determine high-value test cases to automate Continuous integration Automate using a cross–device testing tool Mobile = Late to the Game Make use of existing test tools where possible Provide a variety of scripting options for differing skill sets Mobile = Spread Across the Organization Provide access to local and remote real devices Enterprise-grade product to manage multiple teams 30 Make every digital interaction count MyKeynote® TESTANALYZE Mobile App Testing Mobile Web Testing Load Testing Data Visualization Competitive Intelligence Expert Analysis Mobile App Monitoring Website Monitoring Mobile Web Monitoring MONITOR 31
  • 19. 4/26/2014 17 Keynote: Most Experienced Cloud-based Mobile, App, Web & Network Performance Company on the Planet 1995: Founded 500 employees 4,000+ customers 7,000+ measurement devices in ~300 locations – most scaled on-demand infrastructure in world 700,000,000 performance measurements daily 730 Carrier Networks monitored in ~300 locations in 200 countries 1,500+ devices in 15 global locations Forbes ‘One of the Best 100 Companies in America’ 32 Sao Paolo Moscow Beijing Paris New York Integrating the World’s Largest Global Testing & Monitoring Network and the World’s Largest Real Device Cloud 33
  • 20. 4/26/2014 18 Enterprise Solutions on a Global Scale 3,000 customers in 130 countries Business operations in EMEA, APAC, Americas Global infrastructure Web performance testing & monitoring agents in 100 locations Mobile testing and monitoring agents in 250 locations 7,000 devices on network 270B measurements per year 34 Thank You Questions?