A overview of common test coverage metrics and measurement tools.

1. Utility of Test Coverage Metrics
in TDD
Virender Kumar
TDD Trainer, Consultant
vir_kum77@yahoo.com

2. Learning Objectives
• TDD Overview
• Common Test Coverage Metrics
• Significance of Test Coverage Metrics in TDD
• Test Coverage Measurement Tools

3. TDD
• Software development technique in which
tests are written prior to code
• If you can’t write test for what you are trying
to code, then you should not be thinking
about coding
• TDD reduces the gap between design and
feedback (performance achieved by
implementing the design)

4. Test Coverage Metrics
• Statement Coverage
• Branch Coverage
• Condition Coverage
• MCDC
• Data Flow Coverage
• Path Coverage
• Mutation Coverage

5. Mutation Coverage
• Mutation – small change in program
• Apply mutations to program to obtain set of
mutants M1, M2 … Mn
• Run test suite on each of mutants. Mutant is
killed if an error is detected
• Mutation Test Coverage = No. of mutants
killed / Total no. of mutants
• Has to be automated for any reasonable sized
program

6. Mutation Coverage
• May not be practical to consider all possible
mutants
• Use mutation operators (rules) to generate
mutants
• Value mutations – change values to reflect errors
in reasoning about programs
• Decision mutations – change relational operators
or parenthesisation in conditions
• Statement mutations – change whole line of code

7. Test Coverage Measurement Tools
• PIT (www.pitest.org)
• Jumble (http://jumble.sourceforge.net/)
• Clover
(https://www.atlassian.com/software/clover/
overview)
• Cobertura
(http://cobertura.github.io/cobertura/)
• EMMA (http://emma.sourceforge.net/)

8. PIT
• Measures line coverage and mutation
coverage
• Bytecode based mutant generation
• Test selection is automatic – based on line
coverage
• Can be integrated with Ant or Maven based
build systems
• Compatible with most mocking frameworks,
except Powermock and Jmockit

9. Observations
• TDD does not imply that the tests are perfect
• Test Coverage measurements are relevant to
TDD
• Mutation coverage tells whether something is
really tested
• Tools are available which make it practical to
measure mutation coverage

10. Future Work
• Collect more data on test coverage in TDD
• Effectiveness of mutation coverage vs. other
coverage metrics
• Comparison between various coverage
measurement tools
• Study effect of mocking frameworks on code
coverage