There are many tools, libraries and frameworks available for Android developers to test their applications. The jungle is huge and it's not easy to find the right ones. Some frameworks are good for unit testing, some are good for instrumentation testing, and some can be used for both. Some have great capabilities but annoying weaknesses. Some are good for testing UI, other allow you to make good mocks. We will look at many frameworks, the popular ones like Mockito, Robolectric, Espresso, and some other. Presented at GDG DevFest Minsk 2016.

1. Guide to the jungle of testing frameworks
Tomáš Kypta
@TomasKypta

3. Unit Testing Experience on Android

5. Android apps are difficult to test

6. Android apps were difficult to test

7. Types of Android tests

8. Types of Android tests
Instrumentation
tests
Local unit
tests

9. Android test code
• project sources
• ${module}/src/main/java
• instrumentation tests
• ${module}/src/androidTest/java
• unit tests
• ${module}/src/test/java
• full Gradle and Android Studio support

11. • the essential piece of both instrumentation and unit tests
• alone can be used only for pure Java code
• doesn’t provide any mocks or Android APIs

12. Instrumentation Tests

13. Instrumentation Tests
• running on physical device or emulator
• gradle connectedAndroidTest
• ${module}/build/reports/androidTests/connected/
index.html

15. Instrumentation Tests
Legacy instrumentation tests
or
Testing Support Library

16. Legacy Instrumentation Tests
• JUnit3
• Tests extend from TestCase
• AndroidTestCase
• ActivityInstrumentationTestCase2
• ServiceTestCase
• …
deprecated
since API
level 24

17. Testing Support Library
• JUnit4 compatible
• AndroidJUnitRunner
android {
defaultConfig {
testInstrumentationRunner
"android.support.test.runner.AndroidJUnitRunner"
}
}
dependencies {
androidTestCompile 'com.android.support.test:runner:0.5'
}

18. Testing Support Library
• JUnit test rules
• AndroidTestRule
• ServiceTestRule
• DisableOnAndroidDebug
• LogLogcatRule
• …
androidTestCompile 'com.android.support.test:rules:0.5'

20. • framework for functional UI tests
• part of Android Testing Support Library
androidTestCompile
'com.android.support.test.espresso:espresso-core:2.2.2'

21. @Test
public void sayHello() {
onView(withId(R.id.edit_text))
.perform(typeText(STRING_TO_BE_TYPED), closeSoftKeyboard());
onView(withText("Say hello!"))
.perform(click());
String expectedText = "Hello, " + STRING_TO_BE_TYPED + "!";
onView(withId(R.id.textView))
.check(matches(withText(expectedText)));
}

22. Problems
• testing on device is not isolated
• device state affects the result
• e.g. screen on/off might affect test result
onView(withId(R.id.my_view))
.check(matches(isDisplayed()));

23. Some annoyances
android.support.test.espresso.NoActivityResumedException:
No activities in stage RESUMED.
Did you forget to launch the activity. (test.getActivity() or similar)?

24. UI Automator

25. UI Automator
• APIs for building UI tests
• interaction with both your apps and system apps
• Android 4.3+
androidTestCompile
'com.android.support.test.uiautomator:uiautomator-v18:2.1.2'

26. UI Automator
• several parts
• API for information retrieval and performing
operations
• API for cross-app testing
• uiautomatorviewer tool

27. UI Automator
@RunWith(AndroidJUnit4.class)
public class AnActivityTest {
private static final String MY_PACKAGE = "com.example.helloworld";
private static final int LAUNCH_TIMEOUT = 5000;
UiDevice mDevice;
@Before
public void setUP() {
mDevice = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation());
mDevice.pressHome();
Context context = InstrumentationRegistry.getContext();
Intent intent = context.getPackageManager().getLaunchIntentForPackage(MY_PACKAGE);
intent.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TASK);
context.startActivity(intent);
// Wait for the app to appear
mDevice.wait(Until.hasObject(By.pkg(MY_PACKAGE).depth(0)), LAUNCH_TIMEOUT);
}
// do the testing
}

28. UI Automator
@RunWith(AndroidJUnit4.class)
public class AnActivityTest {
// a setup
@Test
public void testText() throws Exception {
UiObject textLabel = mDevice.findObject(
new UiSelector().packageName(MY_PACKAGE).text("Hello World!"));
UiObject button = mDevice.findObject(
new UiSelector().packageName(MY_PACKAGE).text("Change text"));
button.click();
UiObject2 textResult = mDevice.findObject(By.res(MY_PACKAGE, "txt_label"));
Assert.assertEquals("Hello Minsk!", textResult.getText());
}
}

29. Java API

30. • use with UI Automator
• direct manipulation with sensor values on Genymotion
devices
• allows to omit mocking sensor values
androidTestCompile 'com.genymotion.api:genymotion-api:1.0.2'

31. if (!GenymotionManager.isGenymotionDevice()) {
return; //don't execute this test
}

32. if (!GenymotionManager.isGenymotionDevice()) {
return; //don't execute this test
}
GenymotionManager genymotion =
Genymotion.getGenymotionManager(
getInstrumentation().getContext());

33. if (!GenymotionManager.isGenymotionDevice()) {
return; //don't execute this test
}
GenymotionManager genymotion =
Genymotion.getGenymotionManager(
getInstrumentation().getContext());
genymotion
.getRadio().call("555123456");

34. if (!GenymotionManager.isGenymotionDevice()) {
return; //don't execute this test
}
GenymotionManager genymotion =
Genymotion.getGenymotionManager(
getInstrumentation().getContext());
genymotion
.getRadio().call("555123456");
genymotion
.getNetwork().setProfile(Network.Profile.EDGE);

36. • for iOS, Android, Windows apps
• based on WebDriver protocol
• alternative to UI Automator
• aims to automate apps from any language and any test
framework
• …with access to back-end APIs and DBs
• Android 4.1+

37. Instrumentation tests
are
kinda
SLOOOOOW

38. Unit Tests

39. Unit Tests
• run on JVM
• mockable android.jar
• gradle test
• ${module}/build/reports/tests/${variant}/index.html

40. ...

42. • Helps rarely
• returns 0, false, null, …
Method ... not mocked.
android {
testOptions {
unitTests.returnDefaultValues = true
}
}

44. • mocking framework
• easy to use
• compatible with Android unit testing
testCompile 'org.mockito:mockito-core:2.2.11'

45. • can be used also in instrumentation tests
• needs dexmaker
androidTestCompile 'org.mockito:mockito-core:2.2.11'
androidTestCompile 'com.google.dexmaker:dexmaker:1.2'
androidTestCompile 'com.google.dexmaker:dexmaker-mockito:1.2'

46. @RunWith(JUnit4.class)
public class ContextManagerTest {
@Mock Context mAppContext;
}

47. @RunWith(JUnit4.class)
public class ContextManagerTest {
@Mock Context mAppContext;
@Before
public void setUp() {
MockitoAnnotations.initMocks(this);
}
}

48. @RunWith(JUnit4.class)
public class ContextManagerTest {
@Mock Context mAppContext;
@Before
public void setUp() {
MockitoAnnotations.initMocks(this);
}
@Test
public void testWithContext() {
…
}
}

49. @RunWith(MockitoJUnitRunner.class)
public class ContextManagerTest {
@Mock Context mAppContext;
@Test
public void testWithContext() {
…
}
}

50. @RunWith(JUnit4.class)
public class ContextManagerTest {
@Test
public void testWithContext() {
Context appContext = mock(Context.class);
}
}

51. @RunWith(JUnit4.class)
public class ContextManagerTest {
@Test
public void testWithContext() {
Context appContext = mock(Context.class);
Mockito.when(appContext.getPackageName())
.thenReturn(“com.example.app”);
…
}
}

52. • Mockito.spy()
• wrapping a real object
• Mockito.verify()
• verify that special condition are met
• e.g. method called, method called twice, …

53. Limitations
• final classes
• opt-in incubating support in Mockito 2
• anonymous classes
• primitive types
• static methods

55. • Can mock static methods
• Can be used together with Mockito

56. @RunWith(PowerMockRunner.class)
@PrepareForTest(Static.class);
PowerMockito.mockStatic(Static.class);
Mockito.when(Static.staticMethod())
.thenReturn(value);
PowerMockito.verifyStatic(Static.class);

58. • functional testing framework
• runs on JVM
• at first, might be difficult to use
• the ultimate mock of Android APIs
• provides mocks of system managers
• allows custom shadows

59. • possible to use for UI testing
• better to use for business logic
@RunWith(RobolectricTestRunner.class)
public class MyTest {
…
}

60. • Robolectric
• RuntimeEnvironment
• Shadows
• ShadowApplication
• ShadowLooper

61. Potential problems
• difficult to search for solutions
• long history of bigger API changes
• many obsolete posts

63. • cross-platform BDD framework
• human-like test definitions
testCompile ‘junit:junit:4.12'
testCompile ‘info.cukes:cucumber-java:1.2.5'
testCompile 'info.cukes:cucumber-junit:1.2.5'

64. • describe the desired behaviour
Feature: CoffeeMaker
Scenario: a few coffees
Given I previously had 3 coffees
When I add one coffee
Then I had 4 coffees

65. • create the
mapping
public class CoffeeMakerDefs {
CoffeeMaker mCoffeeMaker = new CoffeeMaker();
}

66. • create the
mapping
public class CoffeeMakerDefs {
CoffeeMaker mCoffeeMaker = new CoffeeMaker();
@Given("^I previously had (d+) coffees$")
public void hadCoffeesPreviously(int coffees) {
mCoffeeMaker.setCoffeeCount(coffees);
}
}

67. • create the
mapping
public class CoffeeMakerDefs {
CoffeeMaker mCoffeeMaker = new CoffeeMaker();
@Given("^I previously had (d+) coffees$")
public void hadCoffeesPreviously(int coffees) {
mCoffeeMaker.setCoffeeCount(coffees);
}
@When("^I add one coffee$")
public void addCoffee() {
mCoffeeMaker.addCoffee();
}
}

68. • create the
mapping
public class CoffeeMakerDefs {
CoffeeMaker mCoffeeMaker = new CoffeeMaker();
@Given("^I previously had (d+) coffees$")
public void hadCoffeesPreviously(int coffees) {
mCoffeeMaker.setCoffeeCount(coffees);
}
@When("^I add one coffee$")
public void addCoffee() {
mCoffeeMaker.addCoffee();
}
@Then("^I had (d+) coffees$")
public void hadCoffees(int coffees) {
Assert.assertEquals(coffees, mCoffeeMaker.getCoffeeCount());
}
}

69. • place definition and mapping at the same paths!
• ${module}/src/test/java/com/example/MyMapping.java
• ${module}/src/test/resources/com/example/
MyDefinition.feature
@RunWith(Cucumber.class)
public class RunCucumberTest {
}

70. Code Coverage

71. Code Coverage
• instrumentation tests
• JaCoCo
• EMMA
• obsolete
• unit tests
• JaCoCo

72. Instrumentation Tests & Code Coverage
• has to be explicitly enabled
• gradle createDebugCoverageReport
• ${module}/build/reports/coverage/debug/index.html
• ${module}/build/outputs/code-coverage/connected/$
{deviceName}-coverage.ec
• doesn’t work on some devices!!!
buildTypes {
debug {
testCoverageEnabled true
}
}

73. JaCoCo

74. JaCoCo
• enabled by default for unit tests
• gradle test
• generates binary report in build/jacoco
• ${module}/build/jacoco/testDebugUnitTest.exec
• it’s necessary to create a JacocoReport task to obtain a readable
report

77. Good tests
• run in any order
• run in isolation
• run consistently
• run fast
• are orthogonal

78. Rules of thumb
• prefer pure Java
• abstract away from Android APIs
• separate business logic and UI
• don’t write business logic into activities and fragments
• MVP, MVVM is a way to go
• try avoid static and final
• use dependency injection

79. Questions?