Context: One limitation of the empirical studies about test- driven development (TDD) is knowing whether the developers followed the advocated test-code-refactor cycle. Re- search dealt with the issue of process conformance only in terms of internal validity, while investigating the role of other confounding variables that might explain the contro- versial e↵ects of TDD. None of the research included process conformance as a fundamental part of the analysis. Goal: We aim to examine the impact of process conformance on the claimed effects of TDD on external quality, developers’ productivity and test quality. Method: We used data collected during a previous study to create regression models in which the level of process con- formance was used to predict external quality, productivity, and tests thoroughness. Result: Based on our analysis of the available data (n = 22), we observe that neither quality (p value = 0.21), produc- tivity (p value = 0.80), number of tests (p value = 0.39) nor coverage (pvalue = 0.09) was correlated with the level of TDD process conformance. Conclusion: While based on a small sample, we raise concerns about how TDD is interpreted. We also question whether the cost of strictly following TDD will pay-o↵ in terms of external quality, productivity, and tests thorough- ness.

1. Impact of Process Conformance on the Effects of
Test-driven Development
Davide Fucci, Burak Turhan, Markku Oivo
Department of Information Processing Science
Faculty of Information Technology and Electrical Engineering
University of Oulu

2. OUTLINE
• Test-driven development (TDD) cycle
• Research Questions
• Study Context
• Results
• Limitations
• Conclusion
2

3. THE TDD CYCLE
3

4. TDD CLAIMS
4

5. RESEARCH QUESTION
What is the relationship between the
claimed effects of Test-driven
Development and the conformance
to the Red-Green-Refactor process?
5

6. PROCESS CONFORMANCE
• Usually considered to mitigate the threat to construct validity
• We consider it as the main factor of interest (explanatory variable)
6
”The extent to which a developer follows the TDD cycle”

7. PROCESS CONFORMANCE
1. Create TestCase BowlingScorekeeperTest.java
2. Edit BowlingScorekeeperTest.java ADD testScore() METHOD
3. TestRun BowlingScorekeeperTest FAIL
4. Create JavaClass BowlingScorekeeper.java
5. Edit BowlingScorekeeper.java ADD int getScore() METHOD
6. TestRun BowlingScorekeeperTest OK
7. Edit BowlingScorekeeper.java MODIFY int getScore() METHOD
8. TestRun BowlingScorekeeperTest OK
7
Example of a TDD-compliant sequence

8. CONSTRUCTS
DVs
• External quality
• Productivity
• Testing effort
• Code coverage
8
IV
• Process
conformance

9. METRICS
• QLTY = sum of external quality of each user story
• User-story quality = % of passing acceptance tests
• PROD = delivered user stories
• TEST = number of unit-tests
• TCOV = block coverage
• CONF = % of TDD development events
9

10. THE STUDY
Data collected during a previous controlled experiment*
10
Subjects:
• 27 units
– 7 pairs
– 20 individual
• Experience: grad students
• Training: 20 hours
* Fucci, D.; Turhan, B., "A Replicated Experiment on the Effectiveness of Test-First Development," Empirical Software Engineering and Measurement, 2013

11. THE STUDY
11
Object:
• Uncle Bob’s Bowling Scorekeeper
• 13 fine-grained user stories
• Incremental difficulty

12. THE STUDY
12
Settings:
• Java and Junit
• Eclipse IDE
• 3 hours

13. RESULTS 1/2
13
1. Is there a linear relationship between external
quality and process conformance to TDD?
✗
p-value = .21
r2 = .076
2. Is there a linear relationship between
productivity and process conformance to TDD?
✗
p-value = .8
r2 = 1.8e-7

14. RESULTS 2/2
14
3. Is there a linear relationship between testing
effort and process conformance to TDD?
✗
p-value = .39
r2 = .036
4. Is there a linear relationship between code
coverage and process conformance to TDD?
✗
p-value = .09
r2 = .12

15. DISTRIBUTION OF CONFORMANCE
15
mean = 78.09, sd = 21.65

16. Do the results hold for different levels of process conformance?
cutoff = 85 (median)
High Conformance
• External Quality
• Productivity
• Testing effort
• Coverage
ADDITIONAL RQ
Low Conformance
• External Quality
• Productivity
• Testing effort
• Coverage
16
✗
✗
✗
✗
✗
✗
✗
✗

17. EXTERNAL QUALITY
17
• Higher baseline for High conformance subjects
• Reach a cap level after which quality doesn’t improve

18. PRODUCTIVITY
18
• Lower baseline for High conformance subjects
• Deviate from the process in order to get things done

19. TESTING EFFORT
19
• Higher baseline for Low conformance subjects
• Improvement for High conformance subjects

20. COVERAGE
20
• Decrease for High conformance subjects
• Production code is not so minimal

21. LIMITATIONS
• Representativeness of the subjects
• Other effects associated with skills?
• Interaction with pair-programming
• Small sample size
• Simple models
21

22. CONCLUSION
22
Summary
The effect of process conformance on:
• External Quality
✗
• Productivity
✗
• Testing effort
✗
• Coverage
✗
• TDD == write a lot of unit-tests
• Strictly following TDD is not viable in the short term

23. FUTURE WORK
• Replications:
• Consider the weight of refactoring during the process
• Use professional software developers
• Include developers’ skill in the model
23
Feel like replicating? Please, contact me

24. Q&A
davide.fucci@oulu.fi
24

25. SANITY CHECK
25

26. DETAILED RESULTS
26
ID Regression coeff. p-value r-squared
QLTY 6.1e-4 .21 .076
QLTY(LC) 3.2e-4 .66 .240
QLTY(HC) 3.4e-6 .99 1.8e-7
PROD -7.2e-04 .80 1.8e-7
PROD(LC) .002 .58 .030
PROD(HC) .007 .61 .030
TEST -.040 .39 .036
TEST(LC) -.010 .86 .003
TEST(HC) .080 .74 .010
TCOV -.310 .09 .120
TCOV(LC) -1.210 .70 .010
TCOV(HC) -0.070 .24 .013