The document presents ten critical questions concerning mobile automation challenges, specifically emphasizing the difficulties in automation due to device fragmentation, lack of mature solutions, and differences from desktop automation. It outlines experiences and opinions from the speaker, Ran Ben Aharon, including insights on real devices versus emulators, testing methodologies, and the importance of robust tools for reliable results. The document also highlights the essential aspects of mobile testing beyond UI testing, covering areas like resource usage, connectivity, and crash detection.

1. My 10 Mobile
Automation
Questions

2. Hello!
I’m Ran Ben Aharon
Automation & Tools lead @ EverythingMe
http://ranbena.com
@ranbena

3. Track Record
▣ Netcraft
▣ Mobile Web dev
▣ Firefox OS
▣ HTML5IL + FEDs

4. I LUV
▣ Coding (JS + Python)
▣ Refactoring
▣ Pull Requesting
▣ Solving problems with Tools

5. <3

6. FIXED A TYPO
NOW APP CRASHES
IN PRODUCTION

7. RELEASE DAY
SCARED SHITLESS

8. DREAMED OF MAKING
MOBILE APPS
NOW I CAN’T SLEEP

9. ‘’
In order to move fast, you
need good brakes.
Joey Simhon
EverythingMe

10. Auto-Tools
aka Visibility

11. Community
Share, Discuss, Support, Network

12. Doing Mobile Automation?
55%
Attempted
Yep
17%Not yet
28%
Your answers

13. 10 Questions
I needed to answer in order to begin

14. 1.
Why is it so hard
to automate
mobile?

15. Y u so hard
▣ No widely used solution
▣ Hard to get advice
▣ Not everything fits
▣ Requires lots of attention
▣ Devices fragmentation

16. 24 devices
Distinct devices
8 sizes
Distinct screen dimensions
70% users
Latest OS adoption rate

17. 18,796 devices
Distinct devices (Aug ‘14)
150 sizes
Distinct screen dimensions
2.5% users
Latest OS adoption rate
via OpenSignal

18. 2.
How is it so
different from
desktop software
automation?

19. Different from desktop
▣ Not yet mature
▣ No common open source solution
▣ Device fragmentation
▣ Users interact differently
▣ Device resources are limited
▣ App store limitations
▣ Rapid deployment

20. 3.
Is there no solid
open source
solution out
there?
Why did we
have to go and
build our own?

21. No solid open source solution
▣ Maturity
▣ Single platform*
▣ Single dev language*
▣ No one-stop-shop
CI > Testing fw > UI simulation fw > Visibility
▣ Each app has it’s unique needs
□ None have ALL features
□ Concentrate on UI testing
* Excluding Appium

22. 4.
How do the big
boys do it?

23. The big boys
Ebay
Calabash
Mobile automation at Ebay Slides
Mozilla
Robocop for Android
Marionette for Firefox OS
Automated testing for Firefox OS
SoundCloud
Robotium for Android
Frank for iOS
Facebook
Selendroid for Android
Yelp
KIF for iOS
Espresso for Android
Zappos
KIF for iOS

24. 5.
Must test on real
devices?
Are emulators
and simulators
ok too?

25. Emulator
▣ Cheap
▣ Fast
▣ Multiple on 1 comp
▣ Sharable
▣ Mock battery,
sensors, network
▣ No hw fatigue
▣ Low maintenance
Real or Emulator?
Real device
▣ Reliable results
▣ Any model available
▣ No need for
resource emulation

26. 6.
Device labs on
premise or cloud
service?

27. Pros
▣ Focus on most
relevant devices
▣ No dependency
▣ No queue
▣ Real time
▣ Fast results*
On premise

28. Pros
▣ Wide range of devices
▣ Scale
▣ Zero maintenance
▣ Real locations*
▣ Real cell networks*
▣ Your test fw + CI
▣ Built-in stress testing*
▣ Manual testing*
Device cloud
To name a few
Appthwack
Appurify
Experitest
Keynote
Perfectomobile
SauceLabs
TestDroid
...

29. LessPainful

30. LessPainful

31. Google I/O ‘14

34. opendevicelab.com

35. 7.
Open source or
commercial
service?

36. Pros
▣ Free
▣ Community
▣ Your code
Open source
To name a few
▣ Appium
▣ Calabash
▣ Espresso
▣ Frank
▣ KIF
▣ Robotium
▣ Selendroid
▣ MonkeyTalk
▣ ...

37. Android
▣ UiAutomator
API 16+ (Appium)
▣ UiAutomation
API 18+ (Robotium)
▣ Instrumentation
API 1+ (Espresso,
Selendroid, Calabash)
User Action Simulation & Introspection
iOS
▣ UiAutomation
(Appium, Zucchini)
▣ UiSpec (Kif, Frank)

38. Magneto - Command your Droids
Magneto

39. Pros
▣ Working solution
▣ Robust reports
▣ Script recording
▣ Device cloud
integration*
Proprietary solutions
To name a few
▣ Experitest
▣ Jamo
▣ Keynote
▣ Ranorex
▣ SOASTA
▣ TenKod
▣ ...

40. 8.
Who writes test
scenarios? Should you?

41. ▣ Automation engineer?
▣ QA team?
▣ Developer?
▣ Product?
Who writes tests?

42. 9.
Where and when
should tests run?
Is there a
widely adopted
methodology?

43. Release Flow Stages
Dev
Review
Merge
Release
Spec RC

44. Release Flow Stages
Dev
Review
Merge
Release
Spec RC
Build Build Build Build

45. 10.
How do you
maintain result
trust?

46. ▣ Unreliable results
▣ Not part of release flow
▣ No visibility
▣ Cumbersome tools
Distrust and disengagement

47. ▣ Reliable results
▣ Integral part of release flow
▣ Full visibility
▣ Delightful tools
Trust and cooperation

48. UI Testing. Got it.
Anything else?

49. Much much more.
UI testing is just one aspect out of many.
This is where it gets even more interesting!

50. Much more than UI testing
▣ Broken connectivity simulation
▣ Offline mode
▣ Excessive resource usage
▣ AB testing / feature switches
▣ Non visual events assertions
▣ Jank detection
▣ API mocks
▣ Stats testing
▣ OTA updates
▣ Stress testing
▣ Image recognition
▣ Crash detection
▣ Purchase/advert flow
▣ Sensors
▣ OS event simulation

51. More info
▣ GTAC - 2014 vids
▣ SmashingMagazine article
▣ TestDroid blog

52. Thanks!
Any questions?
You can find me at
http://ranbena.com
ran@ranbena.com
@ranbena