The document discusses moving beyond traditional agile testing approaches to lean testing. It argues that most unit tests are unnecessary and test scenarios that will never occur. It promotes exploratory and experience-based testing over unit testing alone. The document also advocates for shipping tests with code to catch bugs in the field, using assertions to make code more readable, and taking a lean approach where fixing testing processes is prioritized over fixing individual bugs.

1. Beyond Agile
Testing to Lean
Development
James O. Coplien
Gertrud & Cope

3. Processes and
Tools over
Individuals and
Interactions

4. Test
Testing
System
Testing
Exploratory
Testing
Usability Testing

7. Why doesn’t unit
testing work?
We can’t test things of business
value
Not knowing what to test, we test
many things that don’t need testing
Automated crap is still crap

8. Unit tests can’t test
things of business value
Can you demonstrate that Method X of
Object Y will ever be invoked in practice?
If a unit test fails, how does it affect ROI?
What’s amazing is that we continue to test…
Paths that will never be exercised
Methods that are never used
Naive method invocation sequences

9. Most unit tests are
unnecessary
We test scenarios that will never be
exercised in the real world
We test methods that will never be
invoked
We supply data that a real application
would never have to process

10. Automated crap is
still crap
Design requires intelligence:
intelligence cannot be automated
Autonomation instead of automation
Testing is about exploratory inference
Get manual procedures working first
and then automate the mundane parts

11. Automated testing is not
testing

12. Nokia Test Score 2:
Testing
No dedicated QA - 0
Unit tested - 1
Feature tested - 5
Feature tested as soon as completed - 7
Software passes acceptance testing - 8
Software is deployed - 10

13. Weinberg’s Law of
Decomposition

14. Weinberg’s Law of
Decomposition

15. Weinberg’s Law of
Decomposition

16. Weinberg’s Law of
Decomposition
Used Interface
Used Interface

17. Weinberg’s Law of
Decomposition
Used Interface
Used Interface
Mu s t unit-te s t!
Must unit-test!
Must unit-test!

18. Weinberg’s Law of
Decomposition

19. Weinberg’s Law of
Decomposition
Wasted testing effort

20. Weinberg’s Law of
Decomposition
Wasted testing effort
Test
Size

21. Weinberg’s Law of
Decomposition
Wasted testing effort
Use
Size
Test
Size

22. Part III: You Will Have Bugs!
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field

23. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field

24. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 1
Muda
Type 1: Defects

25. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 2
Muda
Type 1
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for

26. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 2
Muda
Type 1
Muda
Type 4
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for
Type 4:
Overprocessing

27. Testing waste
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 1
Muda
Type 4
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for
Type 4:
Type Overprocessing
1 Muda
(field)

28. Testing waste
Removing tests
leaves field faults
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
undetected!
Type 1
Muda
Type 4
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for
Type 4:
Type Overprocessing
1 Muda
(field)

29. Why do we unit
test?
Because we can
It doesn’t require teamwork
We don’t have to talk with the business
We get to interact with tools instead of
people
Our decisions are emotional, not rational

30. Reduction of Waste
Type 1: Better testing (e.g., going beyond
contextual testing to exploratory & experience-based
testing) (and better yet, better analysis
and design, where the real payoffs are)
Type 2: Don’t test dead code
Type 4: Remove redundant and tautalogical
tests, and tests that give no information
Type 1 in the field: Ship your tests!

31. Don’t take the tests
out when you ship!
Remember, you will have bugs in the
field!
Deployment is your largest test driver
Ship your tests as part of the code
IN case of a failure, it saves you
from Type 9 muda (unsafe execution)

32. Be Customer-
Focused
Assertions and Design by Contract!

33. How Adding ASSERTS reduces work
2500000
2000000
1500000
1000000
500000
0
15 bugs p=0.01 15 bugs p=0.02
10 bugs p=0.01 10 bugs p=0.02
5 bugs p=0.01 5 bugs p=0.02
0 1 2 5 10 20 50 100
number of ASSERTS
Work, Tsinglechecks
Overload
Magazine,
October 2014

34. Defensive
Programming
… is largely the incorporation of
tests in the code
Long known for high-reliability
systems
The value of asserting the obvious
Design by Contract

35. Assertions make code
readable!
- (void) prepareForSegue: (UIStoryboardSegue*) segue
sender: (id) sender
{
assert(segue != nil);
if ([[segue identifier] isEqualToString:
@"SegueAvatarPickerToPickOneLoginPlayer"]) {
assert((UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPhone));
// The field parentViewController_ (instance variable) is
// maybe null if we are in iPhone mode and doing a real
// segue, instead of a popover as we do on the iPad. So
// get the right object from the segue record.
KnowsyPlayerPickerCommonViewController *parentViewController =
[segue destinationViewController];
[parentViewController setKnowsyData: knowsyData_];
} else {
// iPad - no special processing - should be no popover
assert(parentViewController_ == nil);
}
}

36. Part IV: Beyond agile
testing, Lean testing
When a test finds a bug, it’s about
fixing the process — not the bug
The long-term payoff is higher!
No bug-tracking systems: we fix
problems NOW.
The tester should be the team’s
conscience

37. Other Lean-isms
Deming: Quality is inversely
proportional to the number of
testers
GM kept around its cars for 1 - 6
weeks for test and repair: Toyota
drove them off the end of the
assembly line onto the car carriers.

38. Conclusion
Be humble about what tests can accomplish
It’s not about testing, but value
Design tests to assess properties of business value
Go beyond unit testing to defensive
programming and design-by-contract
As agile testers, focus on process first and
product second