What Test Coverage
Mean to us?
Joseph Yao
Tuesday, July 9, 13
Who I am?
n Agile Coach at Odd-e
n Father, Husband...
n Software Craftsman
n Basketball, Board Game...
Tuesday, July 9...
Agenda
n Bias and Truth of Test Coverage
n Classical Test Coverage
n Mutation Test Coverage
n TDD and Test Coverage
Tu...
Bias of Test Coverage
Tuesday, July 9, 13
Truth of Test Coverage
I expect a high level of coverage. Sometimes
managers require one. There's a subtle difference.
-- ...
Truth of Test Coverage
Tuesday, July 9, 13
Classical Test Coverage
Tuesday, July 9, 13
Function Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Satisfied Test: foo(2, 2)
...
Function Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Tuesday, July 9, 13
Statement Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Satisfied Test: foo(2, 2)...
Statement Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Unsatisfied Test: foo(2, ...
Decision Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Satisfied Test: foo(2,...
Decision Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Unsatisfied Test: foo(...
Condition Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Satisfied Test: foo(2...
Condition Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Unsatisfied Test: foo...
Classical Test Coverage
Tuesday, July 9, 13
Classical Test Coverage
Tuesday, July 9, 13
Expectation of Automated Test
Tuesday, July 9, 13
Expectation of Automated Test
Any Possible “Bad” Code
Change can be Detected
Tuesday, July 9, 13
Mutation Test Coverage
Tuesday, July 9, 13
Why Mutation Test
Tuesday, July 9, 13
Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
Tuesday, Jul...
Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
int foo (int...
Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
int foo (int...
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Condition Cove...
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x...
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x...
Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x...
Other Mutators
n Negate Conditionals Mutator
n Math Mutator
n Increments Mutator
n Invert Negatives Mutator
n Inline ...
Test Driven Development
Tuesday, July 9, 13
TDD and Test Coverage
n You can most likely ignore Classical Test Coverage
n You can’t ignore Mutation Test Coverage
n ...
PokerHands Kata
Tuesday, July 9, 13
True Mutation
Tuesday, July 9, 13
Problematic Test
Tuesday, July 9, 13
False Mutation
Tuesday, July 9, 13
Reference
n Martin Fowler’s blog http://martinfowler.com/bliki/
TestCoverage.html
n Mutation Test Wikipedia http://en.wi...
Q & A
新浪微博:姚若舟
微信:yaoruozhou
土豆(Kata视频):姚若舟
Github: JosephYao
Tuesday, July 9, 13
Upcoming SlideShare
Loading in …5
×

What test coverage mean to us

1,303 views

Published on

Mutation test is a way to put a "mutation" into your code, run the test and then see if the test fails or not. A mutation is a change to production code which should make it behave in a different way. The idea is, if the code is just enough to pass the test, any mutation should make the test failed due to the behavior change. Although this statement is not always true, it should be correct in most cases.
By TDD, we should get just enough and the simplest code that passing all the test. But, how do we know this? Sometimes, since writing test and code is so interactive in TDD, we may not choose the smallest baby step to pass the test. This means we may write some code which is not driven by current test. On the other side, regardless doing TDD or not, we hope those unit tests can avoid future regression when we changing the code, right? To make it more clear, our expectation is that any "bad" code change (aka. mutation) can be detected by those tests.

Published in: Technology, Business
1 Comment
0 Likes
Statistics
Notes
  • Since it isn't referenced in the slides for anyone wondering, the tool being shown here is http://pitest.org.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total views
1,303
On SlideShare
0
From Embeds
0
Number of Embeds
242
Actions
Shares
0
Downloads
66
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

What test coverage mean to us

  1. 1. What Test Coverage Mean to us? Joseph Yao Tuesday, July 9, 13
  2. 2. Who I am? n Agile Coach at Odd-e n Father, Husband... n Software Craftsman n Basketball, Board Game... Tuesday, July 9, 13
  3. 3. Agenda n Bias and Truth of Test Coverage n Classical Test Coverage n Mutation Test Coverage n TDD and Test Coverage Tuesday, July 9, 13
  4. 4. Bias of Test Coverage Tuesday, July 9, 13
  5. 5. Truth of Test Coverage I expect a high level of coverage. Sometimes managers require one. There's a subtle difference. -- Brian Marick http://martinfowler.com/bliki/TestCoverage.html Tuesday, July 9, 13
  6. 6. Truth of Test Coverage Tuesday, July 9, 13
  7. 7. Classical Test Coverage Tuesday, July 9, 13
  8. 8. Function Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Satisfied Test: foo(2, 2) Tuesday, July 9, 13
  9. 9. Function Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Tuesday, July 9, 13
  10. 10. Statement Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Satisfied Test: foo(2, 2) Tuesday, July 9, 13
  11. 11. Statement Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Unsatisfied Test: foo(2, -1) Tuesday, July 9, 13
  12. 12. Decision Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Satisfied Test: foo(2, 2), foo(-1, 2) Tuesday, July 9, 13
  13. 13. Decision Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Unsatisfied Test: foo(2, 2) Tuesday, July 9, 13
  14. 14. Condition Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Satisfied Test: foo(2, 2), foo(2, -1), foo(-1, -1) T/F Tuesday, July 9, 13
  15. 15. Condition Coverage int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } T/F Unsatisfied Test: foo(2, 2), foo(-1, 2) T/F Tuesday, July 9, 13
  16. 16. Classical Test Coverage Tuesday, July 9, 13
  17. 17. Classical Test Coverage Tuesday, July 9, 13
  18. 18. Expectation of Automated Test Tuesday, July 9, 13
  19. 19. Expectation of Automated Test Any Possible “Bad” Code Change can be Detected Tuesday, July 9, 13
  20. 20. Mutation Test Coverage Tuesday, July 9, 13
  21. 21. Why Mutation Test Tuesday, July 9, 13
  22. 22. Redundant Code? int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } Tuesday, July 9, 13
  23. 23. Redundant Code? int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } Tuesday, July 9, 13
  24. 24. Redundant Code? int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z; } n Missing test to prove the sideEffect call is needed n If code is redundant, it can’t be learnt via any classical test coverage Tuesday, July 9, 13
  25. 25. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  26. 26. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>=0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  27. 27. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>=0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) Test to cover this Mutator: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, 0)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  28. 28. Conditional Boundary Mutator int foo (int x, int y) { int z = 0; if ((x>0) && (y>0)) { z = x; } return z; } int foo (int x, int y) { int z = 0; if ((x>0) && (y>=0)) { z = x; } return z; } Condition Coverage Satisfied Test: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, -1)) assertEquals(0, foo(-1, 2)) n Test not prevent “bad” code change to happen Test to cover this Mutator: assertEquals(2, foo(2, 2)) assertEquals(0, foo(2, 0)) assertEquals(0, foo(-1, 2)) Tuesday, July 9, 13
  29. 29. Other Mutators n Negate Conditionals Mutator n Math Mutator n Increments Mutator n Invert Negatives Mutator n Inline Constant Mutator n Return Values Mutator n Void Method Calls Mutator n Non Void Method Calls Mutator n Constructor Calls Mutator Tuesday, July 9, 13
  30. 30. Test Driven Development Tuesday, July 9, 13
  31. 31. TDD and Test Coverage n You can most likely ignore Classical Test Coverage n You can’t ignore Mutation Test Coverage n If not TDD, then you can learn a lot from both Classical and Mutation Test Coverage Tuesday, July 9, 13
  32. 32. PokerHands Kata Tuesday, July 9, 13
  33. 33. True Mutation Tuesday, July 9, 13
  34. 34. Problematic Test Tuesday, July 9, 13
  35. 35. False Mutation Tuesday, July 9, 13
  36. 36. Reference n Martin Fowler’s blog http://martinfowler.com/bliki/ TestCoverage.html n Mutation Test Wikipedia http://en.wikipedia.org/ wiki/Mutation_testing n Mutation Test Analysis Report http:// crestweb.cs.ucl.ac.uk/resources/ mutation_testing_repository/TR-09-06.pdf Tuesday, July 9, 13
  37. 37. Q & A 新浪微博:姚若舟 微信:yaoruozhou 土豆(Kata视频):姚若舟 Github: JosephYao Tuesday, July 9, 13

×