The document discusses effective testing strategies, emphasizing the importance of focusing on what truly matters to users rather than just the technical aspects of testing. It outlines various types of tests, including unit, integration, and end-to-end tests, and contrasts good vs. bad testing practices. Additionally, the concept of mutation testing is introduced as a technique to assess the effectiveness of test cases.

1. ​Your Tests Are
Lying To You:
​Improving Your Tests by
Testing What Really Matters
​
​Open Source 101, April 2019

2. Confidential
Brian Childress
I build things @ Capital One
Cyber, Tech College

3. ​3
● Intro
● What?
● Why?
● How?
Agenda

4. ​4
● Intro
● What?
● Why?
● How?
Agenda
○ Is testing?
○ Is good vs. bad testing?

5. ​5
● Intro
● What?
● Why?
● How?
Agenda
○ Do we test?
○ Should I care?

6. ​6
● Intro
● What?
● Why?
● How?
Agenda
○ Can I get started?
○ Can I test better?

7. ​7
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
A discussion about how
we can achieve the best
testing results given
limited resources, time,
budget, and skill sets.
This is:
Debate over: which type
of test, framework,
library, language,
operating system,
keyboard layout, or text
editor is best.
This is not:

8. ​8
Poll
Photo by Chris Liverani on Unsplash

9. ​9Photo by Kate Stone Matheson on Unsplash

10. ​10
“how afraid is everyone of putting in
something new or changing something?”
- Anatoliy Zaslavskiy

11. ​11

12. Confidential ​12
Tests are
12ConfidentialPhoto by Stephane YAICH on UnsplashPhoto by Maranda Vandergriff on Unsplash

13. ​13
Happy Path
Photo by Paul Rysz on Unsplash

14. ​14
Reality
Photo by Ashim D’Silva on Unsplash

15. ​15

16. ​16
Unit Tests

17. ​17

18. ​18
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
● Contracts
● Isolation
● Bugs
● Input / Output
Cons
● Isolated
● Repeatable
● Fast
Pros
Unit Tests

19. ​19
Integration Tests

20. ​20

21. ​21
● Component Interaction
● Contracts
● Repeatable
● Fast
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
● Complex
● Fragile
● Input / Output
ConsPros
Integration Tests

22. ​22
End to End Tests

23. ​23

24. ​24
● Full Application Flow
● Improved tooling
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
● Complex!
● Slow
● Fragile
ConsPros
End to End Tests

25. ​25
Good vs. Bad?
describe("calc", function(){
it("add", function() {
expect(1 + 1).toEqual(2);
});
});

26. ​26
• Add unnecessary abstraction
• Require deep understanding of the application
• Don’t think about the future developer
Bad Tests:

27. ​27
Good vs. Bad?
https://howtodoinjava.com/scripting/javascript/jasmine-unit-testing-tutorial/
describe("when calc is used to perform basic math operations", function(){
// Spec for sum operation
it("should be able to calculate sum of 3 and 5", function() {
expect(calc.sum(3,5)).toEqual(8);
});
...
});

28. ​28
• Go against the rules
• Clear definition
• Concise
• Approachable
Good Tests:

29. ​29
What matters?

30. ​30
Boss

31. ​31
Users

32. ​32
What matters?
,
if they aren’t using our app it doesn’t
matter how good our tests are.
Users are what matters

33. ​33
Resources

34. ​34
Test Your Tests

35. ​35
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
Use this box for citations, sources, statements, notes, and legal disclaimers that are required.
“Mutation Testing is a type of software testing where
we mutate (change) certain statements in the source
code and check if the test cases are able to find the
errors.”
guru99.com

36. ​36
Mutation Testing
function isUserOldEnough(user) {
return user.age >= 18;
}
return user.age > 18;
return user.age < 18;
return false;
return true;

37. Confidential ​37
Support for:
C#
JavaScript
Scala
Stryker Mutator
37Confidential

38. ​38
Demo

39. ​39
Questions?

40. ​40
Thank You!
Brian Childress
brian.childress@capitalone.com