1. The Cake Your Test
Coverage is a Lie!
Chris Shepherd
© 2020 ThoughtWorks

2. Hi!

3. Photo by luis gomes from Pexels
Testing!

6. Photo by Min An from Pexels

7. ● Common testing strategies
● Exciting errors (and where to find them)
● Mutation testing and its history
● Stryker Mutator - A fun framework for finding
mutants
● Some other bits and bobs
So what's the plan, Stan?

8. © 2020 ThoughtWorks

9. © 2020 ThoughtWorks© NASA

10. © NASA

11. © NASA

12. © NASA

13. © NASA

14. © Photo by Rebekah Howell on Unsplash

15. © NASA

18. Manual,
Exploratory
Tests
Unit Tests
Service Tests
UI /
E2E Tests

19. Unit Tests
Service Tests
UI /
E2E Tests
Manual,
Exploratory
Tests

20. Unit Tests
Service Tests
UI /
E2E Tests
Manual,
Exploratory
Tests

21. Unit Tests
Service Tests
UI /
E2E Tests
Manual,
Exploratory
Tests

22. Manual,
Exploratory
Tests

23. Photo by Jake Davies on Unsplash

24. Photo by Giorgio Trovato on Unsplash
Static Type System
Unit Tests
Integration Tests
E2E Tests

25. Photo by Sanjay Shivakumar on Unsplash
Integrated Tests
Integration Tests
Implementation
Details

27. Photo by Somben Chea from Pexels
Manual,
Exploratory
Tests
Unit Tests :(

28. Photo by Sarah Kilian on Unsplash

29. The
Competent
Programmer
Hypothesis.

31. Mutation
Testing.

32. Problem One
The Oracle Problem

33. “In order to use testing to validate a
program it is necessary to assume the
existence of a test oracle which can be used
to check the correctness of test output.”
- William Howden, 1978

34. Problem Two
Photo by Gustavo Fring from Pexels
The Reliable Test Set Problem

35. ● Surfaced in 1971
● An answer to validating a Reliable Test Set
● Seed defects in your software to create Mutants
● Re-evaluate your tests to verify their efficacy
● Profit
In a Nutshell
Photo by Dollar Gill on Unsplash

36. Photo by Jean-Baptiste Burbaud from Pexels
The
Coupling
Effect
Hypothesis.

39. execute(operation, args) {
if (operation //= "ADD") {
this.toDoList.push(args);
}
return this.toDoList;
}
it("Adds Items", () /> {
const toDoList = new ToDoList();
const list = toDoList.execute("ADD", "Write the Tests");
expect(list).toEqual(["Write the Tests"]);
});

40. File % Stmts % Funcs % Lines
All files 100 100 100
utils.js 100 100 100

41. execute(operation, args) {
if (operation //= "ADD") {
this.toDoList.push(args);
}
return this.toDoList;
}
it("Adds Items", () /> {
const toDoList = new ToDoList();
const list = toDoList.execute("ADD", "Write the Tests");
expect(list).toEqual(["Write the Tests"]);
});
execute(operation, args) {
if (true) {
this.toDoList.push(args);
}
return this.toDoList;
}
<- Before
After ->

42. File % Stmts % Funcs % Lines
All files 100 100 100
utils.js 100 100 100

44. //.
if (this.toDoList[i] != item) {
return this.toDoList;
}
//.
Decision Mutations
//.
if (true) {
return this.toDoList;
}
//.

45. //.
execute(operation, args) {
if (operation //= "ADD") {
this.toDoList.push(args);
}
return this.toDoList;
}
//.
Statement Mutations
//.
execute(operation, args) {
if (operation //= "ADD") {
this.toDoList.push(args);
this.toDoList.push(args);
}
return this.toDoList;
}
//.

46. …
for (i = 0; i < toDoList.length; i/+) {
if (this.toDoList[i] //= item) {
return this.toDoList;
}
this.toDoList.push(item);
}
//.
Value Mutations
…
for (i = 0; i /= toDoList.length; i/+) {
if (this.toDoList[i] //= item) {
return this.toDoList;
}
this.toDoList.push(item);
}
//.

47. Photo by Andrea Piacquadio from Pexels

48. Photo by Simon Berger on Unsplash

49. Photo by Federica Galli on Unsplash

50. Photo by Marvin Meyer on Unsplash

52. Photo by David Levêque on Unsplash
Welcome
To
The
Robo-Bar!
http://qrco.de/bbZArc

53. $ git clone https://github.com/stryker-mutator/robobar-example
$ cd robobar-example
$ npm install
$ npm test
$ npm run stryker
Stryker: Getting Started

54. Coverage Report

55. Photo by Jamie Street on Unsplash

56. Mutation Report

58. Photo by Element5 Digital from Pexels

59. Residual Defect
Density
(RDD)
Photo by William Iven on Unsplash

60. Defect Count
Total # Lines of Code

61. 13
2000
0.065/LoC
6.5/KLoC

62. Using Mutation Testing
to Estimate RDD
Learned RDD
X
Mutation Score

63. Code Coverage is a Lie
Mutation Scores Highlight Blindspots
Residual Defect Density is a Thing
Mutation Scores Can Predict RDD

64. Photo by Ian Espinosa on Unsplash

65. Photo by Ian Espinosa on Unsplash

66. Photo by bruno neurath-wilson on Unsplash

67. Photo by Lama Roscu on Unsplash

69. Photo by taylor hernandez on UnsplashPhoto by taylor hernandez on Unsplash

71. Photo by Brian McGowan on Unsplash

72. Chris Shepherd
Ta
© 2020 ThoughtWorks

73. References
● Metamorphic Testing: A Review of Challenges and Opportunities - Tsong Yueh Chen et al
● Investigations of the Software Testing Coupling Effect - A. Jefferson Offutt
● Mutant census: An empirical examination of the competent programmer hypothesis - Gopinath, Rahul et al
● Mutation Testing - Stuart Anderson
● Fault-Based Testing - Mauro Pezzè & Michal Young
● Fault-Based Testing Without the Need of Oracles - TY Chen
● Theoretical and Empirical Studies of Program Testing - William Howden
● https://stryker-mutator.io/ - This is really good, you should check it out!
@the_sheps
me@thesheps.dev
thesheps.dev
© 2020 ThoughtWorks

74. Continue the
conversation on Slack
© 2020 ThoughtWorks
XConfEurope2020
xconfeurope2020.slack.com
#talk6-your-test-coverage-is-a-lie
#XConfOnline