[AnDevCon 2016] Mutation Testing for Android


Published on

Unit testing coverage is a great way to show us the amount of tested lines and branches of code, but is this really enough? The answer is "no" since unit testing coverage does not really fully measure the efficiency of the unit tests.

This is why there is a need for having techniques that show unit tests efficiency. Mutation testing is one of these powerful techniques. The main idea of mutation testing is to perform byte code modifications (mutations) to original Android app source code and then run app unit tests to check if they are strong enough to fail as a result of these mutations.

This session discusses mutation testing techniques, and demonstrates PIT as a powerful mutation testing tool for Android apps with demos.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Without unit testing, the developers life is difficult:
    The integration between the components is complex because we can have buggy components that are not unit tested at all. This means that the developers will stay nights
    and work hard for making these components working together.
  • The number of defects are un-manageable. One resolved defect can occur many times because there is no repeatable test case that
    ensures that a specific component defect will not happen again.
  • Mainly unit testing is about:
    For every piece of code that has some logic in your application, it should an equivalent another piece of code that unit test it.
    When you unit test some component, you should only care about this component (NOT about the other components which your component interacts with “Mocking is important here”).
    Read the remaining.
  • Repeatable here means that the unit tests can be executed for N number of times, and they are always produce the same output (No flaky tests).
    Incremental here means when a new component defects are discovered, the unit test code base must be updated.
  • Run the app.
    Explain the App MVB Architecture
    View -> RepoListFragment / View Interface.
    Presenter -> RepoListPresenter / Presenter Interface
    Main Interactor.
    RepoList Service.
    RepoListPresenter Test.
    WeatherPresenter Test.
    Explain test cases one-by-one.
    Run unit tests.
    Show how Jacoco is configured.
    Run: > ./gradlew clean jacocoTestReport Minimum (20 minutes) >>35 min?
  • http://pitest.org/quickstart/mutators/
  • Show how PIT is configured.
    Run: > ./gradlew clean pitest
    Fix test cases.
    Re-run again.

    The common causes of tests failing at the coverage stage are
    1. PIT picking up tests not included/are excluded in the normal test config
    2. Tests rely on an environment variable or other property set in the test config, but not set in the pitest config
    3. Tests have a hidden order dependency that is not revealed during the normal test run
    4. PIT doesn't like something in your tech stack - possibly a JUnit test runner

  • {2, 0, 2, 6}

  • [AnDevCon 2016] Mutation Testing for Android

    1. 1. Android Mutation Testing Hazem Saleh Mobile Architect @Nickelodeon
    2. 2. About  More than twelve years of experience in Java and mobile solutions.  Apache Open Source Committer.  Author of four technical books.  DeveloperWorks Contributing Author and DZone MVB.  Technical Speaker (JavaOne, ApacheCon, Geecon, JavaLand …etc).  JSR Experts Group Member.  An X-IBMer and Currently a Mobile Software Architect in Nickelodeon.
    3. 3. Agenda  Developers Life without Unit Testing.  Unit Testing 101.  Unit Testing in Android.  Code Coverage and Jacoco.  Code Coverage Demo.  Current challenges of traditional code coverage.  Mutation Testing 101.  What and Why PIT?  PIT Android Integration.  Mutation Testing Demo.  Quiz.  Q & A.
    4. 4. Developers Life without Unit Testing Complex integration between the system components.
    5. 5. Developers Life without Unit Testing Unmanaged number of new/regression defects especially when the system complexity increases.
    6. 6. Developers Life without Unit Testing Low application quality. Longer testing cycle. test fix test fix test fix testfix test fix test fix test fix
    7. 7. Unit Testing 101 A unit testing is a piece of code (usually a method) that invokes another piece of code and later checks the correctness of some assumptions. Unit testing helps in detecting BUGGY components in the early stages of the project. A test suite is a set of test cases, and a test case is a set of tests which verifies the system components.
    8. 8. AutomatedRepeatable Easy to run. Fast Easy to understand Incremental Unit Testing 101
    9. 9. Unit Testing in ANDROID  Generally, Unit tests in Android are implemented using: – JUnit. – Mockito (for mocking testable class dependencies).  However, there are some challenges: – Mocking Static methods. – Mocking final methods or classes. – Mocking private methods.  Thanks to PowerMock, the previous challenges are met.
    10. 10. Unit Testing in ANDROID Other challenge: –Unit testing classes with huge dependency on Android SDK is hard (such as activities / fragments). Thanks to Robolectric: –This challenge is also met: http://robolectric.org/ –Robolectric mocks all Android classes by Shadow classes. –Robolectric adds a great power to Android apps unit testing.
    11. 11. Code Coverage and Jacoco  Code Coverage represents the amount of source code which will be executed when test cases run.  In order to measure the amount of tested source code, there are popular coverage criteria: –Statement coverage –Function coverage –Branch coverage, which represents the amount of branches of each control structure (such as if) covered.
    12. 12. Code Coverage and Jacoco  JaCoCo is a free code coverage library for Java.  It includes the popular code coverage metrics: statement coverage, branch coverage, and method coverage.
    13. 13. Code Coverage and Jacoco  Fortunately, JaCoCo is fully Integrated with Gradle.  JaCoCo works fine with Android projects.  In order to configure JaCoCo with Gradle: –Enable code coverage for the build type(s) that you will be testing with. –Import JaCoCo in your gradle build file. –Configure JaCoCo with your Java sources.
    14. 14. Code Coverage and Jacoco Demo Sample URL: https://github.com/hazems/Dagger-Sample
    15. 15. Challenges  Traditional code coverage only measure the amount of executed code.  Traditional code coverage does not detect code faults.  Traditional code coverage does not show how strong the code is.
    16. 16. Mutation testing 101  Mutation testing is about seeding app source code with faults (mutations).  After seeding, unit tests then run.  If a unit test fails, then a mutation is killed (and means that your unit test is strong enough to face this mutation).  If a unit test succeeds, then a mutation is lived (and means that your unit test needs modification to be stronger).  In mutation testing, the quality of the test can be measured by the percentage of the killed mutations.
    17. 17. Mutation testing 101  Mutations can be for example: –Negate Conditionals Mutator (<= TO > and < TO >=) –Increment Mutator (i++ TO i--) –Invert Negatives Mutator (-I to I) –Math Mutator (+ to – and – to + and * to / and * to /) –Return Values Mutator (boolean  true to false) –Void Method Call Mutator (removing call).
    18. 18. What and Why PIT?  One of the mutation testing tools for Java.  It has advantages –Fast to execute (unlike most of the Java mutation testing tools). –Compatible with: •Ant •Maven •Gradle –Active. –Provides easy to read test reports.
    19. 19. PIT Android Integration  Gradle Plugin for PIT is available: http://gradle-pitest-plugin.solidsoft.info/ https://github.com/szpak/gradle-pitest-plugin  Gradle Plugin works perfect with Java projects.  However, for Android Gradle projects, you need to use this fork: https://github.com/koral--/gradle-pitest-plugin
    20. 20. PIT Android Integration  For using PIT in your Android apps, add PIT plugin to your top-level build.gradle file as follows buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle- pitest-plugin:0.0.3' } }  Apply plugin: 'com.android.application’ apply plugin: 'pl.droidsonroids.pitest'
    21. 21. PIT Android Integration  Configure PIT plugin basically as follows pitest { targetClasses = ['com.test.xyz.daggersample.presenter.*'] /* Specify target classes to be mutated */ excludedClasses = ['**Factory*'] /* Exclude the unnecessary classes */ pitestVersion = "1.1.0" threads = 4 /* Specify number of threads to execute mutation testing */ outputFormats = ['XML', 'HTML'] /* Specify output format */ }
    22. 22. PIT Android Integration  Check the mutation results by executing the following command: >./gradlew clean pitest
    23. 23. Mutation Testing Demo Demo
    24. 24. Quiz
    25. 25. Q & A Twitter: @hazems Email: hazems@apache.org