The document discusses the importance of test automation in Android development, outlining the various types of tests and best practices for implementation. It highlights the challenges of building applications and the need to choose effective automation strategies to save time and resources. Additionally, different testing tools and frameworks are presented, along with their advantages and disadvantages.

1. Testing for Android
When, Where, and How to
Successfully Use Test
Automation
Trent Peterson | @tdpeterson | trent@appthwack.com

2. Who?

3. AGENDA
What is testing?
What’s the problem?
Why should I care?
When: Choosing our battles
Where: Best Practices
How: Tools & Frameworks
Q&A

4. INTRO TO TESTING

5. Types of tests
•
•
•
•
•
•
•
•
Functional
Non-functional
Integration
Unit
Regression
Smoke
Usability
Performance
• Stress
• Acceptance
• Load
• Sanity
• End-to-End
• System
• Security
And on and on…

6. Testing distilled for this discussion
Functional Testing
Non-functional Testing

7. THE PROBLEM

8. Building things is hard.

9. WHY SHOULD I CARE?

10. So what’s the big deal?
1,000,000+ apps in
2 crashes and 84% will uninstall
- Compuware as reported in TechCrunch

11. We’ve been here before.

12. CHOOSING OUR BATTLES

13. First, what’s our goal?
Save time!
BE REALISTIC
Save money!
• Increase regularity of testing
Shrink QA!
• Free up resources for other testing efforts
• Enable previously impossible tests
AUTOMATE
EVERYTHING
!

14. Deciding what to automate
How often will I test this?
Is this something a human is good at?
How much effort will automation require?

15. Some examples
AUTOMATE
MANUAL
• Granular functionality
(unit tests)
• Repetitive tasks
(update paths,
navigation, etc.)
• Performance
• UX
• Responsiveness and
“feel”
• New functionality

16. Our options (Machine)
JVM
Emulators
Real Devices
• FAST
• Unit tests
only*
• Mocks
• Cheap
• Catch layout
issues
• Simulate
calls/SMS
• SLOW
• Illusion of 1:1
with real
devices
• Exactly what end
customers use
• Performance data
• Fast
• Expensive:
Purchase,
maintain, etc.

17. Our options (Human)
QA
Crowd
End Users
• Pros
• Real
feedback
• Know the
product
• Expensive
• Real feedback
• Might know
product
• Expensive
(usually)
• Slow
• Don’t do this.

18. BEST PRACTICES

19. No barriers between Dev & QA

20. Automation is a software project

21. Automation is a tool, not a solution

22. Automate with precision

23. It’s all about data

24. Know your matrix

25. Abstraction is important
…but don’t obsess

26. TOOLS and FRAMEWORKS

27. Tool: Recorders
GOOD
BAD
• Fast
• Non-developers can
create tests
• Fragile
• Difficult to maintain
• Non-developers can
create tests
BEST PRACTICE
• Treat as a fancy spy

28. Tool: UI Exerciser Monkey
$ adb shell monkey
–p my.sweet.app –v 1000

29. Framework: Android Testing Framework
SUMMARY
• JUnit/Java
• Instrumentation provides hooks into Android
SDK to control Android-specific functionality
• Very basic functionality supported
• The foundation for most Android testing. Learn
it, use it, love it.

30. Framework: UI Automator
SUMMARY
• JUnit/Java
• Extends testing framework with UI selectors
and manipulators
• Handle/control system dialogs, etc
• 4.1+ only
• Excellent choice if you have the luxury of
supporting 4.1+.

31. Framework: Robolectric
SUMMARY
• JUnit
• Runs in JVM
• Mocks Android functionality with “shadow”
objects
• Super fast
• Not everything is mocked (also, mocks)
• Great for staying sane while developing

32. Framework: Robotium
SUMMARY
• JUnit extension
• Runs on emulators and real devices
• Oriented towards black-box testing and
ensuring real-world outcomes
• Limited to the app under test
• Great choice for developers going beyond
basic unit testing and testing UI.

33. Framework: Cucumber-based
SUMMARY
• Human-readable tests
• Runs on emulators and real devices
• Cross-platform
• Complex scenarios require development
• Behavior-driven design. Be careful!
• Great choice for simple flows and teams with
limited developer resources.

34. Framework: Appium
SUMMARY
• Supports many language adapters
• Runs on emulators and real devices
• Cross-platform
• New, and not all WebDriver concepts map
intuitively to native apps
• Great choice for teams happy with Selenium
and expanding to native.

35. Frameworks: Commercial
GOOD
BAD
• Support
• Added features and
integrations
• Longevity?
• Proprietary
• Often heavy and
difficult to maintain
• $$$
ADVICE
• Evaluate very carefully and acknowledge lock-in.

36. Tool: Spoon
Spoon
Instrumentation Tests

37. When choosing, ask…
What are we testing?
What are our automation goals?
Who will write the automated tests?

38. SHAMELESS PLUG

39. API
• Continuous
Integration
• IDE
• Command Line
Clients
• Browser-based
web app
AppThwack
•
•
•
•
100s of non-rooted devices
Parallel execution
Built-in compatibility tests
Support for all popular
automation frameworks
Results
• Interactive, realtime report
• JSON data via
API

40. Q&A
Trent Peterson | @tdpeterson | trent@appthwack.com