The document analyzes code exclusion from test coverage, revealing that code is frequently omitted for reasons including complexity, debugging, or being non-runnable. A study of 68 popular Python projects found that 36% involved coverage exclusion, mainly in conditional and exception handling code. The findings suggest a need for better practices and guidelines regarding coverage exclusion to ensure accurate test assessments.

1. What Code Is Deliberately Excluded
from Test Coverage and Why?
MSR 2021
Andre Hora

2. Test Coverage
measures the
percentage of code
that is (un)covered
by tests
2

3. Identify untested areas of the code
Ensure that frequently changing
code is covered
Facilitate code review
Make sure that tests are not
getting worse over time
3

4. Codecov
4

5. But, not all code is equally important
for coverage analysis (eg, debug)
5

6. But, not all code is equally important
for coverage analysis (eg, debug)
6
• Coverage tools provide native
support to exclude code from
coverage analysis
• Developers can
fl
ag the code to
be ignored from coverage

7. But, not all code is equally important
for coverage analysis (eg, debug)
7
• Coverage tools provide native
support to exclude code from
coverage analysis
• Developers can
fl
ag the code to
be ignored from coverage

8. But, not all code is equally important
for coverage analysis (eg, debug)
8
• Coverage tools provide native
support to exclude code from
coverage analysis
• Developers can
fl
ag the code to
be ignored from coverage

9. But, not all code is equally important
for coverage analysis (eg, debug)
9
• Coverage tools provide native
support to exclude code from
coverage analysis
• Developers can
fl
ag the code to
be ignored from coverage
ignored code

10. 10
To do “good”
To do “evil”
😇
👿
Code exlusion from test
coverage can be used …

11. 11
To do “good”
To do “evil”
😇
👿
Code exlusion from test
coverage can be used …
• Fine-con
fi
gurate coverage reports
• Exclude debug code
• etc.

12. 12
To do “good”
To do “evil”
😇
👿
Code exlusion from test
coverage can be used …
• “Cheat” to increase coverage
• Avoid testing complex code
• etc.
• Fine-con
fi
gurate coverage reports
• Exclude debug code
• etc.

13. Better understanding coverage exclusion practices can
reveal novel usage cases that should be fomented by
developers and harmful cases that should be avoided
13

14. Study Design

15. Coverage Assessment
15
1

16. Coverage Assessment
16
1
Case Study Selection
2
• Starting from 68 popular Python projects
• 55/68 (80%) rely on Coverage.py
• 20/55 (36%) use coverage exclusion

17. Coverage Assessment
17
1
Case Study Selection
2
• Starting from 68 popular Python projects
• 55/68 (80%) rely on Coverage.py
• 20/55 (36%) use coverage exclusion
Research Questions
3
• RQ1 (Frequency): How frequent is code excluded from test coverage?
• RQ2 (Time): When is code excluded from test coverage?
• RQ3 (Code): What code is excluded from test coverage?
• RQ4 (Reasons): Why is code excluded from test coverage?

18. Results

19. RQ1
Frequency

20. RQ1 How frequent is code
excluded from test coverage?
20
Code is frequently excluded from test coverage analysis:
we
fi
nd 534 individual occurrences in 179 source
fi
les of the
20 selected projects.

21. RQ2
Time

22. RQ2 When is code excluded
from test coverage?
22
Most code is excluded from coverage analysis since its
creation (75%), meaning they are already created using the
exclusion feature. In 25% of the cases, the exclusion
feature is added over time, on the median, 24 days later.

23. RQ3
Code

24. RQ3 What code is excluded
from test coverage?
24
Most of the excluded code from test coverage happens in
conditional statements (42%) and exception handling statements
(29%), which are code snippets known to be harder to test.

25. RQ3 What code is excluded
from test coverage?
25
What speci
fi
cally?
Non-runnable, debugging-only, defensive code, platform-
speci
fi
c, and conditional importing.

26. RQ4
Reasons

27. RQ4 Why is code excluded
from test coverage?
27

28. RQ4 Why is code excluded
from test coverage?
28
Untested code: “Bring coverage up to
100%: Just adds a couple ‘# pragma no-
cover’ comments to skip coverage on
lines that already weren’t covered.”
Complexity: “Adds pragma: no
cover to the recursive functions”.
Deprecated code: “Add pragma no
cover to deprecated flags check”
Parallelism: “very rare multi-thread
only event [...] Disable test cover”
Trivial/Safe Code:“Add some no-cover
pragmas on functions that don’t need tests”

29. RQ4 Why is code excluded
from test coverage?
29
Mostly because it is already untested (22%), low-level (20%), or
complex (15%). Other rationales are related to deprecated/legacy
code, parallelism, trivial/safe code, and non-determinism.
Untested code: “Bring coverage up to
100%: Just adds a couple ‘# pragma no-
cover’ comments to skip coverage on
lines that already weren’t covered.”
Complexity: “Adds pragma: no
cover to the recursive functions”.
Deprecated code: “Add pragma no
cover to deprecated flags check”
Parallelism: “very rare multi-thread
only event [...] Disable test cover”
Trivial/Safe Code:“Add some no-cover
pragmas on functions that don’t need tests”

30. 1. Developers are using coverage exclusion
to avoid testing and yet increasing
coverage, which is not a best practice
2. We need novel techniques to spot biased
coverage reports
3. It is important to propose project
guidelines to enforce explanations when
using the exclusion feature
4. Detect and
fl
ag trivial/safe candidates for
coverage exclusion to produce more
accurate test coverage reports
Takeaways

31. What Code Is Deliberately Excluded
from Test Coverage and Why?
MSR 2021
Andre Hora