Mobile apps have brought a whole new set of challenges when it comes to testing: Fast paced development cycles with multiple releases per week, multiple app technologies and development platforms to support, tons of devices and form factors, and additional pressure from enterprise and consumers less patient with low quality apps. With these new challenges, come a new set of mistakes testers can make! Fred has worked with dozens of mobile test teams to help them avoid common traps when building test automation for mobile apps and would like to share some best practices that could be useful to a lot of developers and testers starting with mobile test automation. Best practices such as: • When, what and where to automate? • Picking the right wait to handle unreliable back-end calls or device performance • Building testability in a mobile app • Automating the automation (!) • Mix and match performance testing and functional testing In this talk, Fred will bring some real stories (struggles!) and how small changes in process made these mobile apps 10x more reliable!

1. TOP BEST PRACTICES FOR
SUCCESSFUL MOBILE TEST AUTOMATION
@fredberinger

2. SOMETHING FOR YOU TO TAKE BACK HOME

4. MOBILE TEST AUTOMATION IS HARD

5. EXPECTATION
•
•
•
•
It is as difficult as developing a good mobile app
It is a software project by itself
It is not optional in mobile
You will probably fail before you succeed

6. #1 - SET THE RIGHT GOALS

7. AUTOMATION OBJECTIVES
•
•
•
•
•
Continuous feedback to fix bugs faster
Enable new activities
Increase test coverage
Raise confidence
Better customer support

8. #2 – TRACK YOUR OBJECTIVES
IF IT IS NOT MEASURED, IT DOESN’T EXIST

9. SET YOUR OWN METRICS
•
•
•
•
Turnaround time for fixes
Customer satisfaction on app store
Lower number of bugs on production
EMET (Equivalent Manual Execution Test)

10. ACTIONABLE TEST INTELLIGENCE
•
•
•
•
•
•
Part of the automation framework
Available and highly visible to everyone
Establish a baseline and track ROI
Take costs into account (Value=Benefit-cost)
Too many metrics can hurt
Rinse and repeat until you get it right

11. #4
BUT NOT ALL THE TESTS!

12. MISTER JENKINS IN YOUR FRIEND
•
•
•
•
•
•
•
•
Automate your build
Automate your app deployment
Automate your environment deployment
Automate your tear down
Automate your test execution
Automate your reporting
Automate your metrics tracking
Automate your communication

13. #5 – AUTOMATE ASAP
BUT BUILD THE FOUNDATION FIRST

14. Careful planning
• Plan your Automation infrastructure
• Cloud, on-premise, devices connectivity, app deployment, data
aggregation, reporting, etc.
• Pick your automation tool wisely
• Start small and grow
• Don’t hesitate to pause tests building activity

15. #6 – PICK THE RIGHT TESTS TO AUTOMATE
AND I CLEARLY DON’T HAVE A GOOD PICTURE FOR THIS SLIDE

16. ANALYZE THE RISK OF FAILURE
R(c)=P(c) * I(c)
Probability
• Code Complexity
• Changed Areas
• Affected Interfaces
• New Technology
• Component Maturity
Impact
• Financial
• Reputation
• Legal
• Security
• Loss of Customers

17. #7 – WHERE TO RUN TESTS
HINT: 0% USERS RUN YOUR APP ON A SIMULATOR

18. SIMULATOR VS REAL DEVICES
• Pros
• Cheap
• Integrated with IDE
• Cons
• Not testing on actual platform.
What if the test pass? What’s
next?
• Network is different
• OS is different (stock)
• Can’t simulate real hardware
(CPU, Memory, etc.) Not fit for
mobile performance
• Pros
• Reproduce real gestures
• Real results, no false negative
• Can test under OEM
customization
• Fit for mobile performance
• Cons
• Need to be managed (device
Cloud helps!)

19. UNDERSTAND YOUR MARKET
74% iOS7

20. HAVING TROUBLE WITH ANDROID

21. #8 – BUILD EFFICIENT TESTS
SIZE MATTERS!!

22. S.F.I.R.S.T.R
•
•
•
•
•
•
•
Small – Easier to understand & Fix
Fast – Parallel execution for faster feedback
Independent – Can run any subsets in any order
Repeatable – Tests get the same result every time
Self-Checking – No human checking
Timely – Should be written in parallel with dev
Reusable – To avoid maintenance nightmares

23. #8 – BUILD TESTABILITY
ASK KINDLY …

24. WHICH IS BETTER?
//TiUIScrollView[@touchTestId='mfaSiteKeysignOn_securityPhraseWindow_ScrollView']/
TiUIScrollViewImpl/UIView/TiUIView[.//UILabel[@text='If
you don't recognize your personalized security image,
don't enter your password.']]/TiUITextField/TiTextField
XPATH
text=blah
TEXT
classname=TiTextField[1]
INDEXED OBJECT
id=mfaSiteKey-siteKey_Password_Input
UNIQUE ID
•
•
•
•
Build more reliable tests with unique IDs
Lower maintenance cost
Test early
A non-optional investment

25. # 9 - DATA IS YOUR FRIEND

26. BE DATA-DRIVEN
•
•
•
•
Increase test coverage FAST
Easy to add, remove and configure tests
Reduce the number of tests to maintain
Separation of Tests and Datas

27. #10 – USE RELIABLE WAITS
FORGET ABOUT TIME DELAYS!

28. USE RELIABLE AND INTELLIGENT WAITS
• Use object visibility to manage the flow of the test
• Never use time delays to manage back-end variability
• Never use time delays to account for device
performance
• Spend time finding the right locator to wait on
• Run your test 100 times before claiming success

29. #11 – LEVERAGE FUNCTIONAL TESTS FOR PERFORMANCE
YOU GET THE PICTURE

30. COVER ALL YOUR BASES
No backend connection
Native Apps
CPU
Battery
Memory
Transaction timing
Web
browser
users
CPU
Stress tests
Endurance tests
Load test
Battery
Memory
Transaction timing
Mobile
browser
users
Native app
users
Response time
Mobile Web Apps
Load
Balancer
Data traffic
HTTP(S)
UDP
WebSocket
Cache
Web
Server
Web
Server
Shared web &
mobile infrastructure
App
Server
App
Server
Database

31. #12 – BUILD THE RIGHT TEAM

32. LEAN AND EFFICIENT
• Quality Software Engineers
• Automation Framework ie. CI, Cloud, Pyton, Perl, Web, everything
• Automation Architect
• What tests to automate? Identify reusable components,
Metrics, ROI, everything
• Automation Engineer
• Build automated tests, everything
• Expert in automation and knowledge of mobile apps dev
• Test Engineer
• Build tests, everything

33. o First End-to-End Mobile App Test Platform
• First Cloud-Based Load Testing Solution
• First Global Test Cloud (17 Countries, 100 Cities)
• First Mobile Test Automation “Platform”
• First real time RUM for web and mobile
o Over 350 Global Corporate Customers
• 10,000 Mobile Developers and Testers use TouchTest
• Over 1,000 Mobile and Web Apps are Tested with TouchTest
o Award Winning & Patented Technology
• Named by Wall Street Journal Top 50 Hottest Companies three years running
• Gartner Visionary Leader
© 2012 SOASTA. All rights reserved.
May 22, 2012
33

34. ü
ü
ü
ü
ü
Precision record and Playback
iOS, Android, Native/Hybrid/Mobile Web
100+ Built-in components
Jenkins integration
Free forever for 1 device
TouchTest Lite
Download it FREE today!
http://www.soasta.com/free

35. TOP BEST PRACTICES FOR
SUCCESSFUL MOBILE TEST AUTOMATION
QUESTIONS?
twitter.com/fredberinger
fberinger@soasta.com
http://www.fredberinger.com