Top Best Practices for Successful Mobile Test Automation

9,461 views

Published on

Mobile apps have brought a whole new set of challenges when it comes to testing: Fast paced development cycles with multiple releases per week, multiple app technologies and development platforms to support, tons of devices and form factors, and additional pressure from enterprise and consumers less patient with low quality apps. With these new challenges, come a new set of mistakes testers can make!

Fred has worked with dozens of mobile test teams to help them avoid common traps when building test automation for mobile apps and would like to share some best practices that could be useful to a lot of developers and testers starting with mobile test automation.


Best practices such as:
• When, what and where to automate?

• Picking the right wait to handle unreliable back-end calls or device performance

• Building testability in a mobile app

• Automating the automation (!)

• Mix and match performance testing and functional testing

In this talk, Fred will bring some real stories (struggles!) and how small changes in process made these mobile apps 10x more reliable!

Published in: Technology
0 Comments
22 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,461
On SlideShare
0
From Embeds
0
Number of Embeds
345
Actions
Shares
0
Downloads
0
Comments
0
Likes
22
Embeds 0
No embeds

No notes for slide

Top Best Practices for Successful Mobile Test Automation

  1. TOP BEST PRACTICES FOR SUCCESSFUL MOBILE TEST AUTOMATION @fredberinger
  2. SOMETHING FOR YOU TO TAKE BACK HOME
  3. MOBILE TEST AUTOMATION IS HARD
  4. EXPECTATION •  •  •  •  It is as difficult as developing a good mobile app It is a software project by itself It is not optional in mobile You will probably fail before you succeed
  5. #1 - SET THE RIGHT GOALS
  6. AUTOMATION OBJECTIVES •  •  •  •  •  Continuous feedback to fix bugs faster Enable new activities Increase test coverage Raise confidence Better customer support
  7. #2 – TRACK YOUR OBJECTIVES IF IT IS NOT MEASURED, IT DOESN’T EXIST
  8. SET YOUR OWN METRICS •  •  •  •  Turnaround time for fixes Customer satisfaction on app store Lower number of bugs on production EMET (Equivalent Manual Execution Test)
  9. ACTIONABLE TEST INTELLIGENCE •  •  •  •  •  •  Part of the automation framework Available and highly visible to everyone Establish a baseline and track ROI Take costs into account (Value=Benefit-cost) Too many metrics can hurt Rinse and repeat until you get it right
  10. #4 BUT NOT ALL THE TESTS!
  11. MISTER JENKINS IN YOUR FRIEND •  •  •  •  •  •  •  •  Automate your build Automate your app deployment Automate your environment deployment Automate your tear down Automate your test execution Automate your reporting Automate your metrics tracking Automate your communication
  12. #5 – AUTOMATE ASAP BUT BUILD THE FOUNDATION FIRST
  13. Careful planning •  Plan your Automation infrastructure •  Cloud, on-premise, devices connectivity, app deployment, data aggregation, reporting, etc. •  Pick your automation tool wisely •  Start small and grow •  Don’t hesitate to pause tests building activity
  14. #6 – PICK THE RIGHT TESTS TO AUTOMATE AND I CLEARLY DON’T HAVE A GOOD PICTURE FOR THIS SLIDE
  15. ANALYZE THE RISK OF FAILURE R(c)=P(c) * I(c) Probability •  Code Complexity •  Changed Areas •  Affected Interfaces •  New Technology •  Component Maturity Impact •  Financial •  Reputation •  Legal •  Security •  Loss of Customers
  16. #7 – WHERE TO RUN TESTS HINT: 0% USERS RUN YOUR APP ON A SIMULATOR
  17. SIMULATOR VS REAL DEVICES •  Pros •  Cheap •  Integrated with IDE •  Cons •  Not testing on actual platform. What if the test pass? What’s next? •  Network is different •  OS is different (stock) •  Can’t simulate real hardware (CPU, Memory, etc.) Not fit for mobile performance •  Pros •  Reproduce real gestures •  Real results, no false negative •  Can test under OEM customization •  Fit for mobile performance •  Cons •  Need to be managed (device Cloud helps!)
  18. UNDERSTAND YOUR MARKET 74% iOS7
  19. HAVING TROUBLE WITH ANDROID
  20. #8 – BUILD EFFICIENT TESTS SIZE MATTERS!!
  21. S.F.I.R.S.T.R •  •  •  •  •  •  •  Small – Easier to understand & Fix Fast – Parallel execution for faster feedback Independent – Can run any subsets in any order Repeatable – Tests get the same result every time Self-Checking – No human checking Timely – Should be written in parallel with dev Reusable – To avoid maintenance nightmares
  22. #8 – BUILD TESTABILITY ASK KINDLY …
  23. WHICH IS BETTER? //TiUIScrollView[@touchTestId='mfaSiteKeysignOn_securityPhraseWindow_ScrollView']/ TiUIScrollViewImpl/UIView/TiUIView[.//UILabel[@text='If you don't recognize your personalized security image, don't enter your password.']]/TiUITextField/TiTextField XPATH text=blah TEXT classname=TiTextField[1] INDEXED OBJECT id=mfaSiteKey-siteKey_Password_Input UNIQUE ID •  •  •  •  Build more reliable tests with unique IDs Lower maintenance cost Test early A non-optional investment
  24. # 9 - DATA IS YOUR FRIEND
  25. BE DATA-DRIVEN •  •  •  •  Increase test coverage FAST Easy to add, remove and configure tests Reduce the number of tests to maintain Separation of Tests and Datas
  26. #10 – USE RELIABLE WAITS FORGET ABOUT TIME DELAYS!
  27. USE RELIABLE AND INTELLIGENT WAITS •  Use object visibility to manage the flow of the test •  Never use time delays to manage back-end variability •  Never use time delays to account for device performance •  Spend time finding the right locator to wait on •  Run your test 100 times before claiming success
  28. #11 – LEVERAGE FUNCTIONAL TESTS FOR PERFORMANCE YOU GET THE PICTURE
  29. COVER ALL YOUR BASES No backend connection Native Apps CPU Battery Memory Transaction timing Web browser users CPU Stress tests Endurance tests Load test Battery Memory Transaction timing Mobile browser users Native app users Response time Mobile Web Apps Load Balancer Data traffic HTTP(S) UDP WebSocket Cache Web Server Web Server Shared web & mobile infrastructure App Server App Server Database
  30. #12 – BUILD THE RIGHT TEAM
  31. LEAN AND EFFICIENT •  Quality Software Engineers •  Automation Framework ie. CI, Cloud, Pyton, Perl, Web, everything •  Automation Architect •  What tests to automate? Identify reusable components, Metrics, ROI, everything •  Automation Engineer •  Build automated tests, everything •  Expert in automation and knowledge of mobile apps dev •  Test Engineer •  Build tests, everything
  32. o  First End-to-End Mobile App Test Platform •  First Cloud-Based Load Testing Solution •  First Global Test Cloud (17 Countries, 100 Cities) •  First Mobile Test Automation “Platform” •  First real time RUM for web and mobile o  Over 350 Global Corporate Customers •  10,000 Mobile Developers and Testers use TouchTest •  Over 1,000 Mobile and Web Apps are Tested with TouchTest o  Award Winning & Patented Technology •  Named by Wall Street Journal Top 50 Hottest Companies three years running •  Gartner Visionary Leader © 2012 SOASTA. All rights reserved. May 22, 2012 33
  33. ü  ü  ü  ü  ü  Precision record and Playback iOS, Android, Native/Hybrid/Mobile Web 100+ Built-in components Jenkins integration Free forever for 1 device TouchTest Lite Download it FREE today! http://www.soasta.com/free
  34. TOP BEST PRACTICES FOR SUCCESSFUL MOBILE TEST AUTOMATION QUESTIONS? twitter.com/fredberinger fberinger@soasta.com http://www.fredberinger.com

×