The document discusses strategies for increasing test coverage over time through automation, addressed by experts in the field. It highlights the concept of test coverage, its various types, and the importance of defining criteria and goals for effective testing. Additionally, the document emphasizes the role of automation in optimizing test coverage while managing the challenges of time and device fragmentation.

1. How to Increase Test Coverage Over
Time with Automation
Alexandra McPeakFederico Toledo PhD

2. About Federico
● Co-founder & CPIO, Abstracta
● 10+ years experience in testing
● PhD from UCLM, Spain
● Co-Organizer of TestingUY
● Founder of Abstracta Academy
● Wrote one of the first Spanish testing books

3. How to Increase Test Coverage Over Time with Automation
1. Introduction to test coverage
2. Strategy to increase test coverage over time
3. Automation
4. Q&A
Agenda

4. What is Test
Coverage?

5. What is Test Coverage?
How many of the total units did we examine?

6. What is Test Coverage?
Test coverage helps us know quantitatively the extent of our tests.
It does not talk about testing quality nor system’s quality neither!
● Does “100% coverage” mean it’s bug free?
● Is “20% coverage” so bad?

7. Test Coverage is Like Sweeping
● If you only included sweeping bedrooms for sweep
coverage criteria, would the whole house be clean if
you swept 100% of the bedrooms?
● How clean are the covered bedrooms?
● Different criteria over time:
○ The kitchen and my bedroom everyday.
○ All the bedrooms, bathroom and dining room
once a week.

8. Test Coverage is Like Sweeping
Test coverage is useful for seeing
what else should I test.
Also, to see when I should stop
testing (according to the criteria).

9. The criteria is very important
Different criteria
• Different perspectives (logic, data, code, requirements, etc.).
• Different depth (subsumption).
Example: code coverage
• Cover all branches is deeper than cover all lines.
• Branch coverage subsumes Line coverage.

10. Test Coverage: summary
Gives visibility
• Make us define a model, a criteria, a goal and a plan.
• Useful to see progress.
Helps to
• See what else we should test and when to stop testing.
• Share criteria and discuss it.
• Compare criteria.

11. Types of Test
Coverage

12. Code Coverage
Cover the SUT taking into account:
● All paths.
● Branch coverage.
● Line coverage.
White box approach.
Mainly useful for unit testing.
http://community.sharpdevelop.net

13. Data-oriented coverage
Input and output parameters with their own domain.
White box or black box approach.
https://en.wikipedia.org/wiki/Cartesian_product
Examples:
● Equivalence Partition and Boundaries.
● Combinatorial testing:
○ Each choice.
○ Pair-wise.
○ Cartesian product.

14. Others
● Requirements.
● Decision tables or decision trees.
● State-machines:
○ All states, all transitions, etc.
https://www.nuskool.com/learn/lesson/the-friendship-algorithm-how-to-scientifically-choose-your-
friends/
There is always an error theory behind every
test coverage criterion.

15. Don’t forget platform-related

16. Cross browser testing example
Variables:
● Test cases: {TC_01, TC_02, TC_03, TC_04}
● Browsers: { , , }
● Operating Systems: { , , }
Test sets:
● Cartesian Product: 36
combinations.
● Each choice: 4 combinations.
● Pairwise: 12 combinations.
https://pairwise.teremokgames.com/

17. How can we achieve
the desired level of
coverage?

18. It is impossible to test
everything
It is impossible to cover
everything

19. Laying Out a Plan to
Optimize Test Coverage
in the Long Term

20. How to plan good test coverage over many test cycles
When time is scarce, we have to
use it wisely and do our best to
reduce risk.

21. Overcome device fragmentation

22. Overcome device fragmentation

23. Automation can help

24. Increasing Test Coverage with
CrossBrowserTesting Record &
Replay

25. Wrapping Up

26. Q&A

27. Thank you!
Have anymore questions?
Contact federico@abstracta.us
or
Ask for a demo with CrossBrowserTesting!