Your SlideShare is downloading. ×
0
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Mobile Test Automation: Lessons from the Trenches
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Mobile Test Automation: Lessons from the Trenches

1,046

Published on

The app market is estimated to reach almost $58 billion by 2014. Everyone wants a piece of this big pie and want to get their application to monetize quickly. Result - reduced focus on quality testing …

The app market is estimated to reach almost $58 billion by 2014. Everyone wants a piece of this big pie and want to get their application to monetize quickly. Result - reduced focus on quality testing and control with increased costs.

Solution: Automating the testing cycle will ensure effective quality checks and rapid time to market. But what testing processes should you automate and which tools should you use? What are some of the best practices to be followed to ensure a smooth automation cycle?

In this session you will learn:
Understand the new needs and challenges in mobile test automation...

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,046
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
49
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Everything is moving faster. But, to put things in the right perspective, let’s see what has happened in the browser world.This illustration shows what has happened in the past 6 years.Microsoft released 4 major versions of its IE. This is more or less 1 version every 18 months.Competitor browsers, including Firefox and Chrome, did the same.
  • Now, looking at the mobile world only for 2011.The scale has been updated from 6 to only 1 year.This illustration shows the Mobile device market dynamics, including: Android, iOS, Windows, BlackBerry, and other device releases in a single year.For example:Android has introduces three major OS versions, alongside new devices such as tablets, all in one year.This also includes several minor versions, which in many cases incorporated major changes to the device functionalities. The large number of devices making up the Mobile market is ever growing with additional devices constantly being introduced on a daily basis.New devices and OS versions must be supported, developed and tested, by an application in to keep it relevant.
  • Everything is moving faster. But, to put things in the right perspective, let’s see what has happened in the browser world.This illustration shows what has happened in the past 6 years.Microsoft released 4 major versions of its IE. This is more or less 1 version every 18 months.Competitor browsers, including Firefox and Chrome, did the same.
  • Jail broken or not?Market relevantScreen resolution coverageOS CoverageDevices, OS, Demographics, Geography
  • Android has 4 buttonsIphone has one btnBb touch vs non touch
  • One of the biggest challenges is going to be the hundreds/thousands of different device sizes/resolutions currently available in the market. Some of the main considerations when designing for this are:Screen SizeResolutionScreen Real EstateAspect RatioWe have been working on a mobile web app internally which I will share with you today. Here I am showing 2 Androids and an iPhone side by side, the screen sizes and pixels per inch ratios are different, and as you can see this affects what is displayed as well as where it is displayed.
  • One important consideration to keep in mind when developing for mobile is that you should really be validating your designs on the actual target devices.Emulators or viewing in a desktop browser often gives you results which are not a true representation of what you are actually going to getI have a few more examples to show you, here you can see a screenshot of an early version of the iAir app from a desktop browser. As you can see it doesn’t look very good. But when we look at the same URL from a Droid Charge you can see it appears as designed.These screenshots were taken from the same URL, and yet, as you can see the desktop browser gives me an incorrect representation of what the page will look like on a mobile device browser.
  • Transcript

    • 1. www. infostretch. com Mobile Test AutomationLessons from the TrenchesManish Mathuria Mohammad ShaCTO Director of QAInfoStretch Corporation Sabre Airline Solutions(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 2. www. infostretch. com Manish Mathuria CTO, InfoStretch Corporation LINKEDIN: manishmathuria EMAIL: manish.mathuria@infostretch.com BLOG: blog.infostretch.com(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 3. www. infostretch. com AGENDA Mobile MobileLandscape Automation –and Testing BestChallenges Practices QUESTIONS 1 2 3 4 5 Mobile Test Case Study – Automation Sabre Airline Tool Solutions Categories(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 4. www. infostretch. com MOBILE LANDSCAPE MOBILE TESTING CHALLENGES(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 5. www. infostretch. comMOBILE ON THE FAST TRACKDo you know how many babies are born each day?• 371,000 DAILYImpressive, until you realize the number iPhones sold• 377,900 DAILY ( Luke Wroblewski (@lukew) January 24, 2012 )• 515,000 DAILY ( Q2 2012 - BI)Or the number of Android phones activated.• 1.3M DAILY (Eric Schmidt – 9/5/12)And letʼs forget the growth of tablets. not• 277,800 DAILY (just iPad – 189k daily in Q2 – WSJ Oct 1/2012) (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 6. www. infostretch. comENTERPRISE MOBILITY EXPLODING (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 7. www. infostretch. comENTERPRISE MOBILITY EXPLODING (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 8. www. infostretch. com MOBILE - SHORTER LIFE CYCLE Short Cycle for MobileRisk ‘Traditional’Gap fromrequirements MobileMobile Dev.Traditional SW/PCShort CycleMobile dev. Time SW Cycles (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 9. MOBILE ALM PROCESS – www. infostretch. com CONTINUOUS QAMarket 6 years 2005-2011 Browsers Product Version Product Release timeline 6 years (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 10. www. infostretch. comMarket 1 year 2011 V2.3 V2.3.3 V3 V2.3.4 V3.1 V2.3.5 V3.2 V2.3.7 V2.3.6 Android V4 Product Version Product Release timeline 1 year - 2011 (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 11. www. infostretch. comWHAT DOES ALL THIS MEAN?No Matter What Industry – There is an app coming near you!It will be fast, high profile, high visibility (C level)It will test your Dev and QA TeamsIt will be here to stayThink process, devices, Automation, AgileHave Fun! (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 12. 17 www. infostretch. com www. infostretch. comMOBILE AUTOMATIONMOBILE AUTOMATION TOOL CATEGORIES Mobile HTML5 Based Native Platform Platform Independent Automation Automation Technologies Mobile Automation (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 13. www. infostretch. com TOOL CATEGORIES MOBILE AUTOMATION TOOL CATEGORIES Mobile HTML5 Based Native Platform Platform Independent Mobile Platform Independent Mobile Automation Automation Automation – Type A Automation – Type B Technologies • Drives HTML and JS • Drives GUI widgets • Leverages screen • Drives GUI widgets andWhat constructs and Controls on the coordinates, images and Controls on the Native UI • Recognizes Web Native UI Layer OCR to drive automation Layer Controls • Cross device • Ultimate control as • Cross platform scripting, • Good control over GUI and platforms good object level UI one script many devices. executing app • Cross browsers layer access • Has access to the whole • Object aware automationPros • Object aware device automation • Non Intrusive to App • Limited to Web and • Intrusive, often • Limited object awareness, • Intrusive will likely take a Cross Platform requires code access reliance on image and special build. HTML5 Apps • Written specifically OCR • May or may not requireCons for a device platform • Less effective for complex jail breaking • Limited to the automation • Limited to the context of context of the the application application (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 14. www. infostretch. com MOBILE AUTOMATION LESSONS FROM THE TRENCHES(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 15. www. infostretch. comLESSONS LEARNT How to Select Test Cases and Devices for 1 Mobile Automation? 2 Scripting Challenges and Dealing with Fragmentation 3 Dealing with special device conditions 4 Test Execution Challenges (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 16. www. infostretch. com 21 DEVICES SELECTION How do you identify the Devices for Mobile Automation?Objective: Maximum Coverage at Minimum Cost (&time)Type of the App User Personas Geography• Games • Business • Asia Pac• Social • Teen • Americas etc.• Business • SocialApp Functions Device Popularity Output:• Streaming Form Factor Device/OS/Test• Screen Res OS Matrix• GPS etc. ( 4 04 0 8 )27 21 11 1 0 0 | | i n fm@ ik e oi s tg @ cn f.o s tm t c h . c o m2 8 8|0 L a k e s i 0 e a k ev e dS uD r i v2 0 S,u i t e n t 0 0C lS aa ,t a A ( 8) 7 7- 7- 00 o ar nft n reti h co re | 288 d L Dri si e ite e 0 Sa 2 a , ar n C Clara, CA
    • 17. www. infostretch. com 22TEST CASE SELECTIONHow do you identify the test cases for Mobile Automation? Tool and Resource Capability • System/Peripheral Interaction • Interaction between multiple apps/ OS-App /Multi Domain • Location aware, peripheral, battery life, system alerts etc. Stability/Change • Best Practices for other GUI Automation apply • UI components that will change less. • Clear understood business processes Capability/ Complexity / Priority • Less graphically intensive , more data and business scenarios • Medium complexity test cases - > High - > Low • Smoke - > High - > Medium - > Low ( 4 04 0 8 )27 21 11 1 0 0 | | i n fm@ ik e oi s tg @ cn f.o s tm t c h . c o m2 8 8|0 L a k e s i 0 e a k ev e dS uD r i v2 0 S,u i t e n t 0 0C lS aa ,t a A ( 8) 7 7- 7- 00 o ar nft n reti h co re | 288 d L Dri si e ite e 0 Sa 2 a , ar n C Clara, CA
    • 18. www. infostretch. comScripting Challenges / Fragmentation (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 19. WIDGETS WILL RENDER www. infostretch. comDIFFERENT EVEN ON SAME OSAndroid 2.3Left: SamsungRight: Sony Ericson (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 20. www. infostretch. comInherent Device/Platform Differences (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 21. www. infostretch. com www. infostretch. comFORM FACTOR CHALLENGES Droid X2 Droid Charge iPhone 4 (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 22. www. infostretch. com www. infostretch. comACTUAL TARGET DEVICESTEST ON ACTUAL TARGET DEVICES Desktop Browser Droid Charge (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 23. www. infostretch. comDealing with Fragmentation www. infostretch. comDesign your test cases for exceptions! Browser Browser Go Make Confirm Sign On Text Verify Sign Off Open To Payment PaymentQTP/Selenium Make Payment Test case Address Browser Select Acct Schedule Sync Text Quick Links Nick Name ATM Search Open View Payment Browser Go ZIP ATM Make Confirm Sign On Sign Off Sync Image To Search Payment PaymentFramework Component Library Android Android iOS BB Torch BB 8520 LG CU620 LG VX9000 HTC Aria HTC DroidDevices Device Implementations for Sign On (10+) Device Implementations QTP Testcases Component ( 4 04 0 8 )27 21 11 1 0 0 | | i n fm@ ik e oi s tg @ cn f.o s tm t c h . c o m2 8 8|0 L a k e s i 0 e a k ev e dS uD r i v2 0 S,u i t e n t 0 0C lS aa ,t a A ( 8) 7 7- 7- 00 o ar nft n reti h co re | 288 d L Dri si e ite e 0 Sa 2 a , ar n C Clara, CA
    • 24. www. infostretch. comDealing with Special Device Conditions (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 25. What Special Conditions? www. infostretch. com App Misbehave in Special Conditions… • CPU Usage, RAM, Network Conditions, Other Apps, Environment conditions, Light Conditions Handy Tools • Android – system panel app / Task Manager • iPhone – System and Monitoring Tools InfoStretch Device Stressor • Remotely sets device in desired and specific conditions – CPU, RAM, Network, Light etc. • Accessible from Automated scripts. (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 26. www. infostretch. com Execution Best Practices(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 27. DEVELOP STRONG TEST RECOVERY www. infostretch. com www. infostretch. com Automation Tests will break due to • Device characteristics • Test script errors • Automation tool instability • Device instability • Data instability Invest in Robust Test Recovery System • Develop agents that can restart test components on devices e.g. if webdriver • Enhance test case recovery, capability to restart test case. Invest in Logging from the test cases • Details trace of the test • Immediately tells you the failures ( 4 04 0 8 )27 21 11 1 0 0 | | i n fm@ ik e oi s tg @ cn f.o s tm t c h . c o m2 8 8|0 L a k e s i 0 e a k ev e dS uD r i v2 0 S,u i t e n t 0 0C lS aa ,t a A ( 8) 7 7- 7- 00 o ar nft n reti h co re | 288 d L Dri si e ite e 0 Sa 2 a , ar n C Clara, CA
    • 28. CI INTEGRATION - REFERENCE www. infostretch. com www. infostretch. com ( 4 04 0 8 )27 21 11 1 0 0 | | i n fm@ ik e oi s tg @ cn f.o s tm t c h . c o m2 8 8|0 L a k e s i 0 e a k ev e dS uD r i v2 0 0 ,u S e n2 0 0C lS aa ,t a A ( 8) 7 7- 7- 00 o ar nft n reti h co re | 288 d L Dri si e ite e S it a ta , ar n C 33 Clara, CA
    • 29. INVEST IN www. infostretch. comCONTINUOUS INTEGRATION (CI)AUTOMATING THE AUTOMATION What is CI? • Automate execution of test cases • Integrated automated reporting, and notifications • Prominent tools – Jenkins, hudson Why CI? • There is no Automation until its Automated  • Well justified effort, easy ROI, pays off in Days • Automated trending and device comparisons. Invest in a Test Automation lab of devices • Dedicated automation devices (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 30. OUR CI INTEGRATION www. infostretch. com www. infostretch. com ( 4 04 0 8 )27 21 11 1 0 0 | | i n fm@ ik e oi s tg @ cn f.o s tm t c h . c o m2 8 8|0 L a k e s i 0 e a k ev e dS uD r i v2 0 0 ,u S e n2 0 0C lS aa ,t a A ( 8) 7 7- 7- 00 o ar nft n reti h co re | 288 d L Dri si e ite e S it a ta , ar n C 35 Clara, CA
    • 31. www. infostretch. com CASE STUDY(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 32. www. infostretch. comBUSINESS PROBLEMAutomated a mobile web based application that enables the ability forpassengers to book and manage travel on their iOS and Android mobiledevices through a simple and intuitive process flow. Development and Manual Testers defined 33 user scenarios test cases for automation to detect regressions. Test cases are categorized by a. Book My Flight b. Manage My Flight c. Check Flight Status d. Feature Function Test – integrated testing with the above 3 categories Automation starts as soon as Manual tests verify each functionality. Application compatibility testing with system notifications are manually tested. (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 33. www. infostretch. comTEST ENVIRONMENTAutomation Tool Selected: Selenium Remote Android and IOS web driver. Automated testslimited to the Sabre web application only. Selenium based solution does not supportautomated application compatibility i.e. system alerts – phone calls while application isrunning and Blackberry devices - Perfecto Mobile test automation can be used in this case. Test on Firefox first – as baseline, then move to mobile devices Webdriver installed on Sabre each mobile devicesSelenium Test Suites Selenium Test Suites Monitor for Test Code Jenkins CI Repository regression Selenium Test Suites (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 34. www. infostretch. com SOME TECHNICAL CHALLENGES• Element ID Discovery – – Test code dependent on locator id changes and UI flow changes especially in agile scrum development environment. Colocation is critical to promote communication. – Xpath not supported on all mobile devices.• Selenium Web driver – connection reset when executing whole test suites on iOS and Android devices on a single run. – Web driver not responsive at times causing timeouts. – Still maturing - monthly releases• On Android – Android Web driver – delay in populating in list box values. – Problems with launching web applications on Android emulator. – Compatibility across various devices – i.e. On Android 2.3 OS - Samsung Galaxy and Sony Ericson have different list-box implementations (see next slide)• On IOS – iOS Web driver does not always reset after test suite run and need device restart. – Installing Web driver on iOS devices requires “jail broken” device and valid provision profile on device. (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 35. www. infostretch. comLESSONS LEARNT• Automation team to be co-located with development team especially with agile scrum teams.• Establish policy to ensure locator ids backward compatibility between sprints.• Due to the variability of devices and web driver behaviors, architect your automation with this theme “when things change, change it in ONE place”.• Automation scripts should coordinate the highest level abstraction, (libraries) whenever possible to maintain agility and robustness across devices.• Agreement on list of devices / vendors to be supported. Not all devices will be have the same way.• Selenium iOS and Android web drivers are still evolving - expect to do debugging to workaround web driver issues. (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 36. www. infostretch. com QUESTIONS(408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA
    • 37. www. infostretch. comNEXT STEPS http://tinyurl.com/automationpresentation www.infostretch.com/starwest-session-mobile-automation.pdf email: info@infostretch call: (408) 727-1100 (408) 727-1100 | info@infostretch.com | 2880 Lakeside Drive Suite 200, Santa Clara, CA

    ×