Oh you test?
cool test bro
Paul Blundell
@blundell_apps
blog.blundellapps.com github.com/blundell
AutoTrader
- Define different types of testing
- Clarify testing & naming
- Benefits of each type of testing
- Codez available to hel...
Test Types
- Unit Tests
- Integration Tests
- Functional Tests
- Acceptance Tests
- Mutation Tests
- Vendor Tests
- Fuzz T...
- Test a single block of code / function
- Test independently from collaborators
- Only test against interface definitions...
@Test
public void testGetLastPathSegmentReturnsFilename() {
SecureUrl secureUrl = new SecureUrl(
"http://test.com/some/ran...
- Daily development work
- Allow for continuous refactoring
- Test Driven Development
- Documentation for legacy developer...
Unit Tests - Where
- All building blocks
- All levels of an application architecture
- Help pinpoint a place of failure
- Highlight UI issues
- Guarantee combined system integrity
- Prove integration of components
Unit Tests - Won’t
- Interaction between two or more classes
- Use real objects
- Can use threads
- Can access database
Integration Tests - W...
@Test(expected = IllegalStateException.class)
public void testFindFragmentWithUnknownTagResourceIdThrowsError() {
Activity...
- Ensure parity of collaboration between objects
- Testing expected changes - db schema
- Testing environment integration
...
- All building blocks
- Interactions between layers
- Help pinpoint integration issues
Integration Tests - Where
- Highlight UI issues
- Show specific code block of failure
- Give instant feedback
- Hard to diagnose some failures
- Ful...
- Whole system, nearly end to end
- Don’t care about intermediary steps
- Slower to run
- Mostly controller driven
Functio...
public void testShowsFilmInformationAfterApiCallFinished() {
String expectedTitle = "Swimming Pool";
startFilmActivity();
...
- Evaluate expectation of sum of the parts
- Confirming feature completion
- TDD Keep you focused
- Confidence in features...
- Span whole architecture
- Touching live systems
- Above integrated components
Functional Tests - Where
- Show specific broken units of code
- Lack of knowledge of the details
- Give quick feedback for specific problems
Functi...
- Black box tests
- From user perspective
- Specialised form of functional tests
- Model the final complete system
Accepta...
public void testRotationInBrowseScreenMaintainsActionBar() {
swipeToBrowseScreen();
Activity activity = rotate(this);
asse...
- User interface is integrated
- TDD feedback loop
- Capturing differences across devices
- Screenshots for greater feedba...
- Span whole architecture
- Touching live systems
- Above integrated components
Acceptance Tests - Where
- Run very fast
- No feedback upon cause (just symptoms)
- Help day-to-day incremental improvements
Acceptance Tests - Won...
- Mutate the state of your code
- Insert faults in your software
- Unit tests are ran
- Tests fail “mutant is killed” thum...
Mutation Tests - Example
if (a == b) {
// do something
}
will be mutated to
if (a != b) {
// do something
}
Different
Muta...
- Find holes in your test suite
- Depends on % code coverage
- Confidence in your unit tests
- Complexity of your problem ...
- Outside of your code
- Around your test suite
- Not written yourself but configured
- Configuration is key
Mutation Test...
- Give any confidence in working software
- Show collaboration between objects
- Prove if the application actually works
-...
- can be unit, integration, functional
- learn how 3rd
party libraries work
- confirming understanding
- safety net for ou...
@Test
public void setAndGetActiveSessionAreTrustworthy() {
Session.setActiveSession(mockFacebookSession);
FacebookSession ...
- Incorporating 3rd party library
- Not beneficial to add retrospectively
- Replacing libraries with confidence
- Wanting ...
- Around your 3rd party libraries
- One off test
- Suite inside unit tests
Vendor Tests - Where
- Confirm your domain code
- Replace unit, integration or acceptance tests
- Adds overhead to development
Vendor Tests - W...
- Feeding your software random data
- Wait to see what breaks
- It is not logical
- Android = Application Exerciser Monkey...
adb shell monkey -p com.your.package.name -v 50000
Fuzz Tests - Example
// Allowing start of Intent { act=mubi.intent.acti...
- From the beginning
- Thin slice (UI)
- Continuous integration, faster feedback
- Identifying differences across Android ...
- Black box testing
- On top of your application (Android)
- Custom fuzz testing - input level
- Continuous integration
Fu...
- Prove application is running correctly
- Only reports crashes
- Highlights quality rather than bugs
- Not a replacement ...
Helpers / Libraries
- Brief overview
- What test types it benefits
- Any positives
- Any negatives
Mockito
//mock creation
List mockedList = mock(List.class);
//using mock object - doesn’t throw exceptions
mockedList.add(...
Fest
FEST is a collection of libraries, whose mission is to simplify software testing.
No more confusion about the order o...
“Robolectric is a unit test framework that de-fangs the Android SDK jar so you can test-drive the
development of your Andr...
“Robotium is an Android test automation framework. Robotium makes it easy to write powerful
and robust automatic black-box...
Use Espresso to write concise, beautiful, and reliable Android UI tests
Espresso tests state expectations, interactions, a...
The Monkey is a program that runs on your emulator or device and generates pseudo-random
streams of user events such as cl...
The uiautomator testing framework lets you test your user interface (UI) efficiently by
creating automated functional UI t...
Architecture Empower / Encumber
- Allows for separation of concerns
- Java only module
- Directed testing
- Encapsulating change
- Reuse
Modules / Modular...
- Readable & flexible CI builds
- Flavors, build types
- Swapping out whole components
- Greater range of testing flexibil...
- Test reliability
- Test speed
- Data integrity
Mock Services Empower
- Hexagonal architecture
- Insulates system from outside dependencies
- Empowers testing
- Disconnect from platform requir...
S.O.L.I.D Empowers
- Quality of your code reflects quality of your
tests
- Object Oriented programming lends itself to
quality testing
- TDD
...
- Framework lock in
- Google Play services
- Android code examples
- Activity life cycle
Android (frameworks) Encumber
- Believe in testing
- Believe in clean code
- Software craftsmanship
- TDD
- Be energetic
- Enjoy it
YOU Empower || Encum...
Monkey Runner
Spoon
Jmock
Hexagonal Architecture
Contract Tests
Others
Groovy Tests
Static analysis tools
Charles
cURL
Pos...
We are hiring developers
Online Blogs
Martin Fowler http://martinfowler.com/tags/testing.html
JB Rainsberger http://blog.thecodewhisperer.com/
Kevi...
Paul Blundell
Questions?
@blundell_apps
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Oh so you test? - A guide to testing on Android from Unit to Mutation
Upcoming SlideShare
Loading in...5
×

Oh so you test? - A guide to testing on Android from Unit to Mutation

3,266

Published on

Everyone knows you need testing, but what are the different types of testing, how will each type benefit you and what libraries are available to ease the pain? This talk will run through an explanation of each type of testing (unit, integration, functional, acceptance, fuzz, mutation...) explaining upon each level of an Android app, the testing involved, how this will benefit you and how it will benefit your users. It will also explain the architecture of a well tested app. Finally ending with some examples and libraries that ease your accessibility into testing and help with faster more descriptive feedback.

Published in: Education, Technology

Oh so you test? - A guide to testing on Android from Unit to Mutation

  1. 1. Oh you test? cool test bro
  2. 2. Paul Blundell @blundell_apps blog.blundellapps.com github.com/blundell AutoTrader
  3. 3. - Define different types of testing - Clarify testing & naming - Benefits of each type of testing - Codez available to help you - Some architecture hints & tips Abstract
  4. 4. Test Types - Unit Tests - Integration Tests - Functional Tests - Acceptance Tests - Mutation Tests - Vendor Tests - Fuzz Tests
  5. 5. - Test a single block of code / function - Test independently from collaborators - Only test against interface definitions - New behaviour : new unit test Unit Tests - What
  6. 6. @Test public void testGetLastPathSegmentReturnsFilename() { SecureUrl secureUrl = new SecureUrl( "http://test.com/some/random/url/filename.mp4"); String lastPathSegment = secureUrl.getLastPathSegment(); assertThat(lastPathSegment).isEqualTo("filename.mp4"); } Unit Tests - Example
  7. 7. - Daily development work - Allow for continuous refactoring - Test Driven Development - Documentation for legacy developers Unit Tests - When
  8. 8. Unit Tests - Where - All building blocks - All levels of an application architecture - Help pinpoint a place of failure
  9. 9. - Highlight UI issues - Guarantee combined system integrity - Prove integration of components Unit Tests - Won’t
  10. 10. - Interaction between two or more classes - Use real objects - Can use threads - Can access database Integration Tests - What
  11. 11. @Test(expected = IllegalStateException.class) public void testFindFragmentWithUnknownTagResourceIdThrowsError() { Activity activity = Robolectric.buildActivity(Activity.class).get(); FragmentManager fM = activity.getFragmentManager(); Resources resources = activity.getResources(); TaskFragmentFinder finder = new TaskFragmentFinder(resources); finder.findTaskFragment(fM, UNKNOWN_TASK_ID); } Integration Tests - Example
  12. 12. - Ensure parity of collaboration between objects - Testing expected changes - db schema - Testing environment integration Integration Tests - When
  13. 13. - All building blocks - Interactions between layers - Help pinpoint integration issues Integration Tests - Where
  14. 14. - Highlight UI issues - Show specific code block of failure - Give instant feedback - Hard to diagnose some failures - Full system confidence Integration Tests - Won’t
  15. 15. - Whole system, nearly end to end - Don’t care about intermediary steps - Slower to run - Mostly controller driven Functional Tests - What
  16. 16. public void testShowsFilmInformationAfterApiCallFinished() { String expectedTitle = "Swimming Pool"; startFilmActivity(); onData(is(instanceOf(Film.class))) .atPosition(FIRST) .onChildView(withId(R.id.film_text_view_title)) .check(matches(allOf(isDisplayed(), withText(equalToIgnoringCase(expectedTitle))))); } Functional Tests - Example
  17. 17. - Evaluate expectation of sum of the parts - Confirming feature completion - TDD Keep you focused - Confidence in features of system Functional Tests - When
  18. 18. - Span whole architecture - Touching live systems - Above integrated components Functional Tests - Where
  19. 19. - Show specific broken units of code - Lack of knowledge of the details - Give quick feedback for specific problems Functional Tests - Won’t
  20. 20. - Black box tests - From user perspective - Specialised form of functional tests - Model the final complete system Acceptance Tests - What
  21. 21. public void testRotationInBrowseScreenMaintainsActionBar() { swipeToBrowseScreen(); Activity activity = rotate(this); assertActionBarOpen(activity); } private void assertActionBarOpen(Activity activity) { assertTrue("Expected ActionBar open but was closed.", activity.getActionBar().isShowing()); } Acceptance Tests - Example
  22. 22. - User interface is integrated - TDD feedback loop - Capturing differences across devices - Screenshots for greater feedback Acceptance Tests - When
  23. 23. - Span whole architecture - Touching live systems - Above integrated components Acceptance Tests - Where
  24. 24. - Run very fast - No feedback upon cause (just symptoms) - Help day-to-day incremental improvements Acceptance Tests - Won’t
  25. 25. - Mutate the state of your code - Insert faults in your software - Unit tests are ran - Tests fail “mutant is killed” thumbs up - Tests pass “mutant survived” thumbs down Mutation Tests - What
  26. 26. Mutation Tests - Example if (a == b) { // do something } will be mutated to if (a != b) { // do something } Different Mutators: if (a == b) { // do something } will be mutated to if (true) { // do something } public int method(int i) { i++; return i; } will be mutated to public int method(int i) { i--; return i; }
  27. 27. - Find holes in your test suite - Depends on % code coverage - Confidence in your unit tests - Complexity of your problem is high Mutation Tests - When
  28. 28. - Outside of your code - Around your test suite - Not written yourself but configured - Configuration is key Mutation Tests - Where
  29. 29. - Give any confidence in working software - Show collaboration between objects - Prove if the application actually works - Help at all without unit tests Mutation Tests - Won’t
  30. 30. - can be unit, integration, functional - learn how 3rd party libraries work - confirming understanding - safety net for outside changes Vendor Tests - What
  31. 31. @Test public void setAndGetActiveSessionAreTrustworthy() { Session.setActiveSession(mockFacebookSession); FacebookSession session = Session.getActiveSession(); assertThat(mockFacebookSession).isEqualTo(session); } Vendor Tests - Example
  32. 32. - Incorporating 3rd party library - Not beneficial to add retrospectively - Replacing libraries with confidence - Wanting to ‘in house’ a library feature Vendor Tests - When
  33. 33. - Around your 3rd party libraries - One off test - Suite inside unit tests Vendor Tests - Where
  34. 34. - Confirm your domain code - Replace unit, integration or acceptance tests - Adds overhead to development Vendor Tests - Won’t
  35. 35. - Feeding your software random data - Wait to see what breaks - It is not logical - Android = Application Exerciser Monkey - Combinatory with other tools Fuzz Tests - What
  36. 36. adb shell monkey -p com.your.package.name -v 50000 Fuzz Tests - Example // Allowing start of Intent { act=mubi.intent.action.ON_BOARD cmp=com.mubi/.onboard.OnboardActivity } in package com. mubi :Sending Touch (ACTION_DOWN): 0:(1091.0,659.0) :Sending Touch (ACTION_UP): 0:(1085.1356,667.17145) :Sending Touch (ACTION_DOWN): 0:(467.0,404.0) :Sending Touch (ACTION_UP): 0:(472.5769,398.45746) // CRASH: com.mubi (pid 754) // Short Msg: java.lang.IllegalStateException // Long Msg: java.lang.IllegalStateException: Fragment WatchFragment{413d0d98} is not currently in the FragmentManager // Build Label: google/nakasi/grouper:4.1.1/JRO03H/405518:user/release-keys // Build Changelist: 405518 // Build Time: 1364293068000 // java.lang.IllegalStateException: Fragment WatchFragment{413d0d98} is not currently in the FragmentManager // at android.app.FragmentManagerImpl.saveFragmentInstanceState(FragmentManager.java:586) // at android.support.v13.app.FragmentStatePagerAdapter.destroyItem(FragmentStatePagerAdapter.java:140) // at android.support.v4.view.ViewPager.setAdapter(ViewPager.java:413) // at com.mubi.onboard.OnboardActivity.updateViewPager(OnboardActivity.java:139) // at com.mubi.onboard.OnboardActivity.onRetrieved(OnboardActivity.java:132) // at com.mubi.onboard.OnboardTaskFragment$3.run(OnboardTaskFragment.java:77) // at android.os.Handler.handleCallback(Handler.java:725) // at android.os.Handler.dispatchMessage(Handler.java:92) // at android.os.Looper.loop(Looper.java:137) // at android.app.ActivityThread.main(ActivityThread.java:5195) // at java.lang.reflect.Method.invokeNative(Native Method) // at java.lang.reflect.Method.invoke(Method.java:511) // at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:795) // at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:562) // at com.android.internal.os.ZygoteInit.main(Native Method) // at dalvik.system.NativeStart.main(Native Method) // ** Monkey aborted due to error. Events injected: 4191
  37. 37. - From the beginning - Thin slice (UI) - Continuous integration, faster feedback - Identifying differences across Android versions - Highlight refactoring effort Fuzz Tests - When
  38. 38. - Black box testing - On top of your application (Android) - Custom fuzz testing - input level - Continuous integration Fuzz Tests - Where
  39. 39. - Prove application is running correctly - Only reports crashes - Highlights quality rather than bugs - Not a replacement for unit, integration etc Fuzz Tests - Won’t
  40. 40. Helpers / Libraries - Brief overview - What test types it benefits - Any positives - Any negatives
  41. 41. Mockito //mock creation List mockedList = mock(List.class); //using mock object - doesn’t throw exceptions mockedList.add(“one”); //selective & explicit verification verify(mockList).add(“one”); Mockito is a mocking framework that tastes really good. It lets you write beautiful tests with clean & simple API. Mockito doesn't give you a hangover because the tests are very readable and they produce clean verification errors. - Unit Tests : mocking dependencies - Integration Tests : questionable use
  42. 42. Fest FEST is a collection of libraries, whose mission is to simplify software testing. No more confusion about the order of the "expected" and "actual" values. Our assertions are very readable as well: they read very close to plain English, making it easier for non-technical people to read test code. Regular JUnit: assertEquals(View.GONE, view.getVisibility()); Regular FEST: assertThat(view.getVisibility()).isEqualTo(View.GONE); FEST Android: assertThat(view).isGone(); - Unit Tests - Integration Tests - Functional Tests - Acceptance Tests : anywhere you do assertions improving readability
  43. 43. “Robolectric is a unit test framework that de-fangs the Android SDK jar so you can test-drive the development of your Android app. Tests run inside the JVM on your workstation in seconds. “ Robolectric @RunWith(RobolectricTestRunner.class) public class ViewingQueryTaskTest { @Test public void testAlwaysClosesCursor() { viewingQueryTask.run(); verify(mockCursor).close(); } } - Unit tests : for day to day activity including TDD - Integration tests : to handle Android - Positive : speed vs on device tests - Negative : the rabbit hole
  44. 44. “Robotium is an Android test automation framework. Robotium makes it easy to write powerful and robust automatic black-box UI tests. With the support of Robotium, test case developers can write functional and user acceptance test scenarios, spanning multiple Android activities.“ Robotium - Functional tests : check when you click a button the shared preferences are updated - Acceptance tests : ensure features have been completed and take screenshots for feedback public class EditorTest extends ActivityInstrumentationTestCase2<EditorActivity> { private Solo solo; public void setUp() throws Exception { solo = new Solo(getInstrumentation(), getActivity()); } public void testFileExtensionIsInMenu() { solo.sendKey(Solo.MENU); solo.clickOnText("More"); solo.clickOnText("Preferences"); solo.clickOnText("Edit File Extensions"); Assert.assertTrue(solo.searchText("rtf")); } }
  45. 45. Use Espresso to write concise, beautiful, and reliable Android UI tests Espresso tests state expectations, interactions, and assertions clearly without the distraction of boilerplate content, custom infrastructure, or messy implementation details getting in the way. Espresso public void testSayHello() { onView(withId(R.id.name_field).perform(typeText("Dave")); onView(withId(R.id.greet_button)).perform(click()); onView(withText("Hello Dave!")).check(matches(isDisplayed())); } - Functional tests : check when you click a button the shared preferences are updated - Acceptance tests : ensure features have been completed, doesn’t support screenshots as an API but you can still do it http://tiny.cc/disableAnim http://tiny.cc/espressoScreenshot
  46. 46. The Monkey is a program that runs on your emulator or device and generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events. You can use the Monkey to stress-test applications that you are developing, in a random yet repeatable manner. UI Exerciser Monkey adb shell monkey -p com.your.package.name -v 50000 - Fuzz tests : as already discussed allows you to test input events Don’t get the Application Exerciser Monkey & the Monkey Runner mixed up. App Ex Monkey – Fuzz Testing Monkey Runner – Android device control using Python scripts
  47. 47. The uiautomator testing framework lets you test your user interface (UI) efficiently by creating automated functional UI testcases that can be run against your app on one or more devices. UI Automator extend UIAutomatorTestCase UIObject & UISelector Min SDK 16 Jelly Bean adb shell uiautomator runtest MyTests.jar -c com.example.MyApp Tests are in the jar, and the package is the package of the app under test. - Acceptance tests : acts just like a user.
  48. 48. Architecture Empower / Encumber
  49. 49. - Allows for separation of concerns - Java only module - Directed testing - Encapsulating change - Reuse Modules / Modularity Empowers
  50. 50. - Readable & flexible CI builds - Flavors, build types - Swapping out whole components - Greater range of testing flexibility Gradle Empowers
  51. 51. - Test reliability - Test speed - Data integrity Mock Services Empower
  52. 52. - Hexagonal architecture - Insulates system from outside dependencies - Empowers testing - Disconnect from platform requirements Ports & Adapters Empower
  53. 53. S.O.L.I.D Empowers
  54. 54. - Quality of your code reflects quality of your tests - Object Oriented programming lends itself to quality testing - TDD - Static methods are evil Low Code Quality Encumbers
  55. 55. - Framework lock in - Google Play services - Android code examples - Activity life cycle Android (frameworks) Encumber
  56. 56. - Believe in testing - Believe in clean code - Software craftsmanship - TDD - Be energetic - Enjoy it YOU Empower || Encumber
  57. 57. Monkey Runner Spoon Jmock Hexagonal Architecture Contract Tests Others Groovy Tests Static analysis tools Charles cURL Postman Google sizes (small medium large)
  58. 58. We are hiring developers
  59. 59. Online Blogs Martin Fowler http://martinfowler.com/tags/testing.html JB Rainsberger http://blog.thecodewhisperer.com/ Kevin Rutherford http://silkandspinach.net/ Robert Martin https://sites.google.com/site/unclebobconsultingllc/blogs-by-robert-martin References More Info Android Fuzz Tests http://developer.android.com/tools/help/monkey.html Hardcore Fuzz Tests http://www.ibm.com/developerworks/library/j-fuzztest/index.html Vendor Tests https://www.youtube.com/watch?v=47nuBTRB51c#t=23m34s Mutation Tests http://pitest.org/ Google Testing Opinion http://googletesting.blogspot.co.uk/2010/12/test-sizes.html Mockito https://code.google.com/p/mockito/ Mocking Frameworks http://blogs.telerik.com/skimedic/posts/13-07-23/top-5-reasons-to-use-a-mocking-framework Android Fest https://github.com/square/fest-android Robolectric http://robolectric.org/ Robotium https://code.google.com/p/robotium/ Espresso https://code.google.com/p/android-test-kit/wiki/Espresso UI Automator http://developer.android.com/tools/help/uiautomator/index.html SOLID Principles http://www.its-on-the-internet-so-it-must-be-true.com/2012/06/solid-principles.html
  60. 60. Paul Blundell Questions? @blundell_apps
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×