Are Your Tests Well-Traveled? Thoughts on Test Coverage

W10
Test Coverage
10/18/2017 1:30:00 PM
Are Your Tests Well-Traveled? Thoughts
on Test Coverage
Presented by:
Dorothy Graham
Software Test Consultant
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-‐268-‐8770 ·∙ 904-‐278-‐0524 - info@techwell.com - https://www.techwell.com/

Dorothy Graham
Software Test Consultant
In software testing for more than forty years, Dorothy Graham is coauthor of four
books – Software Inspection, Software Test Automation, Foundations of Software
Testing, and Experiences of Test Automation – and is currently working with
Seretta Gamba on a test automation patterns wiki. A popular and entertaining
speaker at conferences and seminars worldwide, Dot has attended STAR
conferences since the first one in 1992. She was a founding member of the ISEB
Software Testing Board and a member of the working party that developed the
ISTQB Foundation Syllabus. Dot was awarded the European Excellence Award in
Software Testing in 1999 and the first ISTQB Excellence Award in 2012. Learn
more about Dot at DorothyGraham.co.uk.

info@dorothygraham.co.uk
© Dorothy Graham 2017
www.DorothyGraham.co.uk
www.TestAutomationPatterns.org
1
Are your tests well-
travelled? Thoughts
about coverage
- and strawberry jam
Prepared and presented by
Dorothy Graham
email: info@dorothygraham.co.uk
EuroStar Webinar
2
“We need to
increase our coverage”
“Make sure you cover 100%!”
“What coverage
are we getting?”
“If we automate,
we will get better coverage”
Have you heard:
“We need as much
coverage as possible”

3
Contents
Analogy with travelling
What is coverage?
Should testing be thorough?
What coverage is not (often mistaken for)
The four caveats of coverage
The questions you should ask
Twitter: @DorothyGraham
4
Where on earth have you been?
• have you seen a lot of the world?
– are you “well-travelled”?
• what does it mean if you say “yes”?
– what does it mean if you say “85%”?
– or “100%”?

5
country
coverage
6
Some travel metrics
• Have you been to
– every continent?
– every country in the
world?
– every state/province?
– every city/town?
– every street in the
place where you live?
• where haven’t you
been?

7
Contents
What is coverage?
8
Travelling
• have you seen a lot of the world?
– are you “well-travelled”?
• what does it mean if you say “yes”?
– what does it mean if you say “85%”?
– or “100%”?
Test coverage
your tests seen a lot of the system?
- do you have “good coverage”?
they

9
Some travel metrics
• Have you been to
– every continent?
– every country in the
world?
– every state/ province?
– every city/town?
– every street in the
place where you live?
• where haven’t you
been?
• Have your tests been to
– every program / function?
– every user story option?
– every menu option?
– every method / API?
– every data combination?
– every error message?
– every statement / decision
/ branch?
• where haven’t your tests
been?
coverage
10
What is coverage?
the
tests
this part of the
software has
been covered
by these tests
the rest has
not been
covered by
these tests

11
Tested everything?
system
system
system
system
software
more
tests
100%? – of what?
modules, statements, branches,
states, data, menu options, functions
methods, APIs, business rules, user stories
12
Statement vs decision coverage
read(a)
IF a > 6 THEN
b = a * 2
ENDIF
print b
1
2
3
4
5
Statement
numbers
B 3 3
5
1
3
4
True
Test
Case Input
Expected
Output
A 7 14
Test Path Decision Decision Statement
Case Taken Outcome Coverage Coverage
A 1, 2, 3, 4, 5 True 50% 100%
B 1, 2, 4, 5 False 50% 80%
True
2
Both 100% 100%

13
Caner’s 101 Coverage concepts
14
ISTQB definitions
• coverage
– the degree, expressed as a percentage, to which
a specified coverage item has been exercised by
a test suite
• coverage item
– an entity or property used as a basis for test
coverage, e.g. equivalence partitions or code
statements
ISTQB Glossary v 1.3

15
What is coverage?
• coverage is a relationship
– between a set of tests
– and some countable part of the system
• 100% coverage is not 100% tested
– 100% of some countable things – among several
types of countable things
• an objective measurement of some aspect of
thoroughness
– if thoroughness is what you want!
16
Benefits of (increasing) coverage
• you are testing more of the coverage items /
system
– may find bugs in areas you haven’t tested before
– story: increase from 60% to 80% in 2 weeks
• fewer areas not tested (at all)
– gives reassurance, confidence
• objective evidence about the thoroughness of
the testing in the particular dimension
– rather than “gut feel” or the “it” metric

17
Advice for measuring coverage
– start by designing good tests and just measure
(observe) coverage
• do not start by setting coverage targets!
– increase coverage only if you think it will help
• e.g. too many bugs not being found by current tests, in
parts that haven’t been tested
• may need extra work to simulate error conditions
– consider measuring coverage of only the most
important things
• e.g. 100% coverage of the 25% most used items
– better than 25% of all items
18
Contents
What is coverage?

19
Thoroughness of testing
• should testing be thorough? why?
• is testing like butter or like strawberry jam?
20
Testing is like butter?
• spread it evenly over
the bread
• same thickness
throughout
• no part not covered
• every part of the
software is tested to the
same extent

21
Testing is like strawberry jam?
• thicker in some
places than others
• big lumps
• some parts not
covered
• should some parts of
the software be
tested more than
others?
22
Breadth or depth? the coverage illusion
major bug minor bug
Breadth / width is coverage
Depth / lumpy testing is selective
“I’ve covered /
tested everything -
haven’t missed
anything!
An illusion!
What is better testing?

23
What’s the goal for testing?
• Width
– every part has been
tested once
– may be required by
regulatory bodies
– wide view, no area
untouched
– may miss something
• Selected depth
– not all parts of the
system are equally
important or equally risky
– focus on where testing
brings greatest value
– deep view, concentrate
on critical parts
– may miss something
24
Contents
What is coverage?

25
Coverage is NOT
the system
the
tests
the
tests
this is test completion!
don’t call it “coverage”!
“we’ve run all of the tests”
[that we have thought of]
26
Contents
What is coverage?

27
1) A single measure is only one level of
coverage out of many
• what is the “right” level of coverage?
– e.g. cities, states, countries?
– statements, APIs, user stories?
– 100%? 80%? See what you have tested / missed?
system
system
system
system
software
28
2) Only need one test to cover
• to cover Australia or Brazil
– only visit one city per country
• to cover a “coverage element”
(statement, decision outcome,
menu option)
– only one test per element
• might be hundreds of ways that
element is used

29
3) Not related to quality of the software
read(a)
IF a > 6 THEN
b = a * 2
ENDIF
print b
1
2
3
4
5
Statement
numbers
B 3 3
5
1
3
4
True
Test
Case Input
Expected
Output
A 7 14
Test Path Decision Decision Statement
Case Taken Outcome Coverage Coverage
A 1, 2, 3, 4, 5 True 50% 100%
B 1, 2, 4, 5 False 50% 80%
True
2
Test B: what
gets printed?
30
3) Not related to quality of the software
• we still have decision coverage, even though
the test fails (a bug)
• statement coverage only wouldn’t find the bug
• client story
– poor quality software from 3rd party
– required decision coverage (80%?)
– 3rd party got a tool, demonstrated coverage
– the software was still rubbish
• tests didn’t pass, lots of bugs
• but the tests exercised the required decisions

31
4) Coverage of what exists, not what
should exist
the system
as built
the system
as needed
coverage
(White Box)
not tested,
even with
100%
coverage
32
4) Coverage of what exists, not what
should exist
• map didn’t show some cities I have visited
– are they important?
• coverage does not show missing functions or
features
– are they important?
• what about requirements coverage?
– good idea if all requirements are important
– but still only covers the requirements you listed

33
Spotify (Kristian Karl) - 1
• Benefits and impact
– beneficial – gave us confidence that the
automation was actually testing the right things
– easy to implement, all open source tools familiar
to developers
– keeping the lights on & maintaining the automation
is much simpler – Test says what requirement it is
testing, giving clear transparent reason to exist
– a requirement that fails is more important that just
some tests that failed – easier to see business
consequences
34
Spotify (Kristian Karl) - 2
– requirements
• write requirements in text docs using Markdown format
• Internal pre-defined forma, (text) unique ID for each req
• stored in code version control system, same reviews
– test automation code
• log msgs in TA saying what req is being verified in test
• logs incl req ID, P/F, date/time, machine, test name
• log msgs read by Logstash, sent to ElasticSearch DB
– evaluate results: compare two lists
• text processing to extract all unique req IDs from reqs
• query ElasticSearch for test result requirement IDs

35
Coverage traps
• the four coverage caveats
– only one level / aspect of thoroughness
– only needs one test to “tick the box”
– not related to how good the tests or software are
– only what is there, not what’s missing
• the coverage illusion
– 100% coverage is NOT 100% tested!
– “coverage” feels comfy, re-assuring
• insurance – you’re covered
• you think you haven’t missed anything
36
Contents
What is coverage?

37
Next time you hear “we need coverage”
• ask: “of what?”
– exactly what countable things need to be
covered by tests?
– why is it important to test them [all]?
• ask: “why?”
– how “deeply” should we cover things?
• would testing be more effective if lumpy, not smooth?
• what can we safely not test (this time)?
• we always miss something
• better to miss it on purpose than fool yourself into
thinking you haven’t missed anything
38
Summary
• coverage is a relationship
– between tests and what-is-tested
• coverage is not:
– test completion (my tests tested what they tested)
– 100% tested – only in one dimension
• beware the coverage traps
• when you hear “coverage”
– ask “of what” and “why"

39
Thank you!
• More information:
• downloads www.DorothyGraham.co.uk
– articles and papers
• email info@DorothyGraham.co.uk
• blog http://dorothygraham.blogspot.com
– including automation, certification
• twitter
– @DorothyGraham
• TestAutomationPatterns.org
– free wiki of automation advice,
with Seretta Gamba
eBook available at
informit.com/swtest
ing
Save 35% with
discount code
SWTESTING

Are Your Tests Well-Traveled? Thoughts on Test Coverage

Recommended

Recommended

More Related Content

Similar to Are Your Tests Well-Traveled? Thoughts on Test Coverage

Similar to Are Your Tests Well-Traveled? Thoughts on Test Coverage (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Are Your Tests Well-Traveled? Thoughts on Test Coverage