SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
5.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
5
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
6.
Coverage
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
6
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
7.
Coverage
7
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
8.
8
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
9.
9
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
Let’s mutate our code to see.
10.
Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
10
11.
Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
•Mutation operators = types of bugs
•Mutant = Program with one seeded bug
11
12.
Mutation analysis
Input : P, TS, Ops
Output : score, coverage
M <- generateMutants (P, OPs)
forAll (m in M)
run (TS,m)
if (one-test-fail)
then killed <- m
else alive <- m
score = killed / size(M)
12
14.
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
14
15.
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
15
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
16.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
16
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
17.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
17
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
18.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
18
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
19.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Mutation reveals bugs in the test suite
19
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Bugs in the test suite:
- Weak oracle
- Missing input
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
20.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
20
@Test
factorialWith5Test() {
long obs = fact(5);
assertEqual(120, obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
@Test
factorialWith1Test() {
assertEqual(1, fact(1));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
22.
Descartes
•Mutation operators: extreme mutation
•Active, open source development
•Pitest plugin
•Compared to PIT
• Less mutants
• Different type of feedback
• Same framework
22
23.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
23
24.
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
24
long fact(int n){
return 0;}
long fact(int n){
return 1;}
PIT : 7 mutants
Descartes : 2 mutants
39.
package com.spotify.apollo.request;
class TrackedOngoingRequestImpl extends ForwardingOngoingRequest {
public void reply(Response<ByteString> message) {
doReply(message);
}
private boolean doReply(Response<ByteString> msg) {
final boolean removed = requestTracker.remove(this);
if (removed) {
super.reply(message);}
return removed;
}
}
39
40.
package com.spotify.apollo.request;
class TrackedOngoingRequestImpl extends ForwardingOngoingRequest {
public void reply(Response<ByteString> message) {
doReply(message);
}
private boolean doReply(Response<ByteString> msg) {
final boolean removed = requestTracker.remove(this);
if (removed) {
super.reply(message);}
return removed;
}
}
40
public void reply(Response<ByteString> message) {}
private boolean doReply(Response<ByteString> msg) {return true;}
private boolean doReply(Response<ByteString> msg) {return false;}
@Test
shouldNotReplyIfNotTracked() {
tOR = new TrackedOngoingRequestImpl(o,t);
tracker.remove(tOR);
tOR.reply(Response.ok());
verifyNoMoreInteractions(ongoingRequest);
}
41.
41
public void reply(Response<ByteString> message) {}
private boolean doReply(Response<ByteString> msg) {return true;}
private boolean doReply(Response<ByteString> msg) {return false;}
package com.spotify.apollo.request;
class TrackedOngoingRequestImpl extends ForwardingOngoingRequest {
public void reply(Response<ByteString> message) {
doReply(message);
}
private boolean doReply(Response<ByteString> msg) {
final boolean removed = requestTracker.remove(this);
if (removed) {
super.reply(message);}
return removed;
}
}
@Test
shouldNotReplyIfNotTracked() {
tOR = new TrackedOngoingRequestImpl(o,t);
tracker.remove(tOR);
tOR.reply(Response.ok());
verifyNoMoreInteractions(ongoingRequest);
}
42.
42
package com.spotify.apollo.environment;
public class ApolloConfig {
public boolean enableMetaApi() {
return optionalBoolean(apolloNode, "metaApi").orElse(true);
}
43.
43
public boolean enableMetaApi() {return true;}
public boolean enableMetaApi() {return false;}
package com.spotify.apollo.environment;
public class ApolloConfig {
public boolean enableMetaApi() {
return optionalBoolean(apolloNode, "metaApi").orElse(true);
}
Covered by 8 test cases which all expect
enableMetaApi()to return true
44.
44
package com.spotify.apollo.environment;
public class ApolloConfig {
public boolean enableMetaApi() {
return optionalBoolean(apolloNode, "metaApi").orElse(true);
}
public boolean enableMetaApi() {return true;}
public boolean enableMetaApi() {return false;}
45.
Descartes
•Current
• Compatible with JUnit5 and latest Pitest
• Integrates with Maven and Gradle
•Future: Descartes and CI
• Incremental Descartes on a commit
• Descartes for pull requests
45
46.
Conclusion
•Mutation analysis
• Automatic generation of mutants
• Evaluate the test suite
•Bugs in test suites
• Oracle
• Input space coverage
• Testability
• Indirectly tested code
46