Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
TOP BEST PRACTICES FOR

SUCCESSFUL MOBILE TEST AUTOMATION

@fredberinger
SOMETHING FOR YOU TO TAKE BACK HOME
MOBILE TEST AUTOMATION IS HARD
EXPECTATION
• 
• 
• 
• 

It is as difficult as developing a good mobile app
It is a software project by itself
It is not o...
#1 - SET THE RIGHT GOALS
AUTOMATION OBJECTIVES
• 
• 
• 
• 
• 

Continuous feedback to fix bugs faster 
Enable new activities
Increase test coverage...
#2 – TRACK YOUR OBJECTIVES

IF IT IS NOT MEASURED, IT DOESN’T EXIST
SET YOUR OWN METRICS
• 
• 
• 
• 


Turnaround time for fixes
Customer satisfaction on app store
Lower number of bugs on pr...
ACTIONABLE TEST INTELLIGENCE
• 
• 
• 
• 
• 
• 


Part of the automation framework
Available and highly visible to everyone...
#4

BUT NOT ALL THE TESTS!
MISTER JENKINS IN YOUR FRIEND
• 
• 
• 
• 
• 
• 
• 
• 



Automate your build
Automate your app deployment
Automate your en...
#5 – AUTOMATE ASAP

BUT BUILD THE FOUNDATION FIRST
Careful planning
•  Plan your Automation infrastructure

•  Cloud, on-premise, devices connectivity, app deployment, data
...
#6 – PICK THE RIGHT TESTS TO AUTOMATE

AND I CLEARLY DON’T HAVE A GOOD PICTURE FOR THIS SLIDE
ANALYZE THE RISK OF FAILURE
R(c)=P(c) * I(c)
Probability
•  Code Complexity
•  Changed Areas
•  Affected Interfaces
•  New...
#7 – WHERE TO RUN TESTS

HINT: 0% USERS RUN YOUR APP ON A SIMULATOR
SIMULATOR VS REAL DEVICES

•  Pros
•  Cheap
•  Integrated with IDE
•  Cons
•  Not testing on actual platform.
What if the ...
UNDERSTAND YOUR MARKET

74% iOS7
HAVING TROUBLE WITH ANDROID
#8 – BUILD EFFICIENT TESTS

SIZE MATTERS!!
S.F.I.R.S.T.R
• 
• 
• 
• 
• 
• 
• 

Small – Easier to understand & Fix
Fast – Parallel execution for faster feedback
Indep...
#8 – BUILD TESTABILITY

ASK KINDLY …
WHICH IS BETTER?
//TiUIScrollView[@touchTestId='mfaSiteKeysignOn_securityPhraseWindow_ScrollView']/
TiUIScrollViewImpl/UIV...
# 9 - DATA IS YOUR FRIEND
BE DATA-DRIVEN
• 
• 
• 
• 

Increase test coverage FAST
Easy to add, remove and configure tests
Reduce the number of tests...
#10 – USE RELIABLE WAITS

FORGET ABOUT TIME DELAYS!
USE RELIABLE AND INTELLIGENT WAITS
•  Use object visibility to manage the flow of the test
•  Never use time delays to man...
#11 – LEVERAGE FUNCTIONAL TESTS FOR PERFORMANCE

YOU GET THE PICTURE
COVER ALL YOUR BASES
No backend connection
Native Apps
CPU
Battery
Memory
Transaction timing

Web
browser
users

CPU

Stre...
#12 – BUILD THE RIGHT TEAM
LEAN AND EFFICIENT
•  Quality Software Engineers

•  Automation Framework ie. CI, Cloud, Pyton, Perl, Web, everything

•  ...
o  First End-to-End Mobile App Test Platform
•  First Cloud-Based Load Testing Solution
•  First Global Test Cloud (17 Cou...
ü 
ü 
ü 
ü 
ü 

Precision record and Playback
iOS, Android, Native/Hybrid/Mobile Web
100+ Built-in components
Jenkins...
TOP BEST PRACTICES FOR

SUCCESSFUL MOBILE TEST AUTOMATION
QUESTIONS?

twitter.com/fredberinger
fberinger@soasta.com
http:/...
Top Best Practices for Successful Mobile Test Automation
Upcoming SlideShare
Loading in …5
×

Top Best Practices for Successful Mobile Test Automation

11,625 views

Published on

Mobile apps have brought a whole new set of challenges when it comes to testing: Fast paced development cycles with multiple releases per week, multiple app technologies and development platforms to support, tons of devices and form factors, and additional pressure from enterprise and consumers less patient with low quality apps. With these new challenges, come a new set of mistakes testers can make!

Fred has worked with dozens of mobile test teams to help them avoid common traps when building test automation for mobile apps and would like to share some best practices that could be useful to a lot of developers and testers starting with mobile test automation.


Best practices such as:
• When, what and where to automate?

• Picking the right wait to handle unreliable back-end calls or device performance

• Building testability in a mobile app

• Automating the automation (!)

• Mix and match performance testing and functional testing

In this talk, Fred will bring some real stories (struggles!) and how small changes in process made these mobile apps 10x more reliable!

Published in: Technology
  • Did you try HelpWriting.net?. They know how to do an amazing essay.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • What Is An Equalizing Charge, Why Is It Needed, And How to Apply One To a Lead Acid Battery.. ◆◆◆ https://bit.ly/2t1uc6e
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • How To Pick Out The Best Battery For a Solar Panel System, Battery Bank, or Off-Grid System ♣♣♣ http://t.cn/AiFAGBwT
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I've reconditioned 17 batteries with EZ Battery Reconditioning. I was curious about your program so I decided to try it out after hearing from a friend how well this worked for her. I'm so glad I did! I've reconditioned 17 batteries with EZ Battery Reconditioning, even an old car battery I thought was long gone. My son likes using your program too! This is a skill everybody should know! Thank you, Tom and Frank. ▲▲▲ https://tinyurl.com/rtswhls
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I've Saved Over $400 On Batteries! I can't believe how simple your reconditioning steps are! My old (and once dead) car batteries, cell phone battery, drill battery, camera battery and tons of other batteries are all reconditioned and working great again! Since starting your program I've saved over $400 on batteries! ★★★ http://t.cn/AiFAb0DL
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Top Best Practices for Successful Mobile Test Automation

  1. TOP BEST PRACTICES FOR SUCCESSFUL MOBILE TEST AUTOMATION @fredberinger
  2. SOMETHING FOR YOU TO TAKE BACK HOME
  3. MOBILE TEST AUTOMATION IS HARD
  4. EXPECTATION •  •  •  •  It is as difficult as developing a good mobile app It is a software project by itself It is not optional in mobile You will probably fail before you succeed
  5. #1 - SET THE RIGHT GOALS
  6. AUTOMATION OBJECTIVES •  •  •  •  •  Continuous feedback to fix bugs faster Enable new activities Increase test coverage Raise confidence Better customer support
  7. #2 – TRACK YOUR OBJECTIVES IF IT IS NOT MEASURED, IT DOESN’T EXIST
  8. SET YOUR OWN METRICS •  •  •  •  Turnaround time for fixes Customer satisfaction on app store Lower number of bugs on production EMET (Equivalent Manual Execution Test)
  9. ACTIONABLE TEST INTELLIGENCE •  •  •  •  •  •  Part of the automation framework Available and highly visible to everyone Establish a baseline and track ROI Take costs into account (Value=Benefit-cost) Too many metrics can hurt Rinse and repeat until you get it right
  10. #4 BUT NOT ALL THE TESTS!
  11. MISTER JENKINS IN YOUR FRIEND •  •  •  •  •  •  •  •  Automate your build Automate your app deployment Automate your environment deployment Automate your tear down Automate your test execution Automate your reporting Automate your metrics tracking Automate your communication
  12. #5 – AUTOMATE ASAP BUT BUILD THE FOUNDATION FIRST
  13. Careful planning •  Plan your Automation infrastructure •  Cloud, on-premise, devices connectivity, app deployment, data aggregation, reporting, etc. •  Pick your automation tool wisely •  Start small and grow •  Don’t hesitate to pause tests building activity
  14. #6 – PICK THE RIGHT TESTS TO AUTOMATE AND I CLEARLY DON’T HAVE A GOOD PICTURE FOR THIS SLIDE
  15. ANALYZE THE RISK OF FAILURE R(c)=P(c) * I(c) Probability •  Code Complexity •  Changed Areas •  Affected Interfaces •  New Technology •  Component Maturity Impact •  Financial •  Reputation •  Legal •  Security •  Loss of Customers
  16. #7 – WHERE TO RUN TESTS HINT: 0% USERS RUN YOUR APP ON A SIMULATOR
  17. SIMULATOR VS REAL DEVICES •  Pros •  Cheap •  Integrated with IDE •  Cons •  Not testing on actual platform. What if the test pass? What’s next? •  Network is different •  OS is different (stock) •  Can’t simulate real hardware (CPU, Memory, etc.) Not fit for mobile performance •  Pros •  Reproduce real gestures •  Real results, no false negative •  Can test under OEM customization •  Fit for mobile performance •  Cons •  Need to be managed (device Cloud helps!)
  18. UNDERSTAND YOUR MARKET 74% iOS7
  19. HAVING TROUBLE WITH ANDROID
  20. #8 – BUILD EFFICIENT TESTS SIZE MATTERS!!
  21. S.F.I.R.S.T.R •  •  •  •  •  •  •  Small – Easier to understand & Fix Fast – Parallel execution for faster feedback Independent – Can run any subsets in any order Repeatable – Tests get the same result every time Self-Checking – No human checking Timely – Should be written in parallel with dev Reusable – To avoid maintenance nightmares
  22. #8 – BUILD TESTABILITY ASK KINDLY …
  23. WHICH IS BETTER? //TiUIScrollView[@touchTestId='mfaSiteKeysignOn_securityPhraseWindow_ScrollView']/ TiUIScrollViewImpl/UIView/TiUIView[.//UILabel[@text='If you don't recognize your personalized security image, don't enter your password.']]/TiUITextField/TiTextField XPATH text=blah TEXT classname=TiTextField[1] INDEXED OBJECT id=mfaSiteKey-siteKey_Password_Input UNIQUE ID •  •  •  •  Build more reliable tests with unique IDs Lower maintenance cost Test early A non-optional investment
  24. # 9 - DATA IS YOUR FRIEND
  25. BE DATA-DRIVEN •  •  •  •  Increase test coverage FAST Easy to add, remove and configure tests Reduce the number of tests to maintain Separation of Tests and Datas
  26. #10 – USE RELIABLE WAITS FORGET ABOUT TIME DELAYS!
  27. USE RELIABLE AND INTELLIGENT WAITS •  Use object visibility to manage the flow of the test •  Never use time delays to manage back-end variability •  Never use time delays to account for device performance •  Spend time finding the right locator to wait on •  Run your test 100 times before claiming success
  28. #11 – LEVERAGE FUNCTIONAL TESTS FOR PERFORMANCE YOU GET THE PICTURE
  29. COVER ALL YOUR BASES No backend connection Native Apps CPU Battery Memory Transaction timing Web browser users CPU Stress tests Endurance tests Load test Battery Memory Transaction timing Mobile browser users Native app users Response time Mobile Web Apps Load Balancer Data traffic HTTP(S) UDP WebSocket Cache Web Server Web Server Shared web & mobile infrastructure App Server App Server Database
  30. #12 – BUILD THE RIGHT TEAM
  31. LEAN AND EFFICIENT •  Quality Software Engineers •  Automation Framework ie. CI, Cloud, Pyton, Perl, Web, everything •  Automation Architect •  What tests to automate? Identify reusable components, Metrics, ROI, everything •  Automation Engineer •  Build automated tests, everything •  Expert in automation and knowledge of mobile apps dev •  Test Engineer •  Build tests, everything
  32. o  First End-to-End Mobile App Test Platform •  First Cloud-Based Load Testing Solution •  First Global Test Cloud (17 Countries, 100 Cities) •  First Mobile Test Automation “Platform” •  First real time RUM for web and mobile o  Over 350 Global Corporate Customers •  10,000 Mobile Developers and Testers use TouchTest •  Over 1,000 Mobile and Web Apps are Tested with TouchTest o  Award Winning & Patented Technology •  Named by Wall Street Journal Top 50 Hottest Companies three years running •  Gartner Visionary Leader © 2012 SOASTA. All rights reserved. May 22, 2012 33
  33. ü  ü  ü  ü  ü  Precision record and Playback iOS, Android, Native/Hybrid/Mobile Web 100+ Built-in components Jenkins integration Free forever for 1 device TouchTest Lite Download it FREE today! http://www.soasta.com/free
  34. TOP BEST PRACTICES FOR SUCCESSFUL MOBILE TEST AUTOMATION QUESTIONS? twitter.com/fredberinger fberinger@soasta.com http://www.fredberinger.com

×