1. Test-Driven Development
Effectiveness
beyond anecdotal evidence
Krzysztof Jelski

2. About me
Training
Software

3. TDD

4. TDD benefits
Dev safety Better designLess defects
Less debugging Less „will it work?”

5. Why bother?
Proofs
Insights
Metrics
Tools

6. Internal quality External quality
Test qualities
Productivity Cost

7. Internal quality External quality
Test qualities
Cost
- no effect observed
Productivity

8. Internal quality External quality
Test qualities
Cost
- no effect observed
- significant result
Productivity

9. Common experiment design
themes
• TDD vs Test-Last
• Between-subjects or repeated measures design
• TDD training at the beginning
• Measuring conformance to TDD

10. Internal quality metrics
• CBO - Coupling Between Object Classes
• WMC - Weighted Methods per Class
• RFC - Response For a Class

11. Internal quality
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010

12. Internal quality
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100

13. Internal quality
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Coupling Between Objects

14. Internal quality
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Coupling Between Objects
Weighted Methods per Class
Response For a Class

15. 3 experiments
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Coupling Between Objects
Accounting Submissions Smells&Library

16. 5-7 weeks preparation
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Coupling Between Objects

17. 8-9 weeks duration
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Coupling Between Objects
23-27 User Stories

18. Custom tools
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Coupling Between Objects
Activity tracking
OO metrics
Mutation testing

19. External quality metrics
• Number of defects found
• Defect density (defects/KLOC)
• Percentage of acceptance tests passed
• Quality mark given by client

20. External quality - studies
Y. Rafique et al., „The Effects of Test-Driven Development
on External Quality and Productivity: A Meta-Analysis”,
2013
External quality (industrial studies)
27

21. External quality - studies
Y. Rafique et al., „The Effects of Test-Driven Development
on External Quality and Productivity: A Meta-Analysis”,
2013
External quality (industrial studies)
27
O. Slyngstad et al., „The Impact of Test Driven Development
on the Evolution of a Reusable Framework of Components: An
Industrial Case Study”, 2008. 100
Defect density -33%

22. External quality - studies
Y. Rafique et al., „The Effects of Test-Driven Development
on External Quality and Productivity: A Meta-Analysis”,
2013
External quality (industrial studies)
27
O. Slyngstad et al., „The Impact of Test Driven Development
on the Evolution of a Reusable Framework of Components: An
Industrial Case Study”, 2008. 100
Defect density -33%
N. Nagappan et al., “Realizing Quality Improvement
through Test Driven Development: Results and Experiences
of Four Industrial Teams,” 2008. 40
Defect density -62% .. -90%

23. Productivity metrics
• Development time per task
• Number of acceptance tests passed per hour
• (sadly LOC was used in several studies)

24. Productivity - studies
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Number of Acceptance Tests / hour
Y. Rafique et al., „The Effects of Test-Driven Development
on External Quality and Productivity: A Meta-Analysis”,
2013 27
Productivity (industrial) -22%

25. Test qualities
• Code coverage
• Mutation Score Indicator
• Number of negative test cases

26. Test qualities - studies
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
Code coverage
Mutation Score Indicator
No effects observed

27. Test qualities - studies
L. Madeyski, „Test-Driven Development.
An Empirical Evaluation of Agile Practice”, 2010 >100
A. Čaušević et al., „Industrial Study on Test Driven
Development: Challenges and Experience”, 2013 60
Number of negative test cases
Code coverage
Mutation Score Indicator
No effects observed

28. Cost - metrics
• Fault cost
• Total project cost
• Maintenance cost

29. Cost - studies
Damm, L.O., Lundberg, L.: „Results from introducing
component-level test automation and Test-Driven
Development.”, 2006

30. Do I do TDD?
conformance =
TDD_changes + refactorings
all _changes
Roberto Latorre, „Effects of developer experience on
learning and applying Unit Test-Driven Development”, 2013

31. Industrial Logic, e-learning

32. Industrial Logic, TDD Dashboard

33. Micheal Hilton, Danny Dig, http://cope.eecs.oregonstate.edu/visualization.html

34. In your team
Internal quality
External quality
TDD conformance*
Measure:
* - and tell me when you found out how

35. Internal quality

36. External quality

37. Productivity

38. Icons made by Freepik from www.flaticon.com CC 3.0 BY
krzysztof.jelski@pragmatists.pl
@krzysztofjelski
Thank you!