1.
Intelligent Testing 2015
Bristol, 18th June 2015
Dr Guillem Bernat
bernat@rapitasystems.com
How to get to 100% code
coverage

2.
Introduction
 Code coverage is important
 Sometimes 100% is difficult
 What do we do if we can't get 100%

3.
Code coverage defined

4.
What does it mean to get 100% code coverage?

5.
“Testing shows the presence of bugs, not their absence”
Edsger Dijkstra (1969)
100% code coverage does not mean:
your code is correct

6.
100% code coverage does not mean:
your software requirements are correct

7.
100% code coverage does not mean:
the compiler translated your code correctly
const volatile int x;
volatile int y;
void foo(void){
for(y=0; y>10; y++)
{
int z = x;
}
}
foo:
mov1 $0, y
mov1 x, %eax
jmp .L2
.L3:
mov1 y, %eax
incl %eax
mov1 %eax, y
.L2:
mov1 y, %eax
cmp1 $10, %eax
jg .L3
ret
At the -0s optimization level, GCC 4.3.0 for IA32
compiles the C code on the left to the assembly on
the right.
The compiler output is wrong: the access to volatile-
qualified variable x should not be hoisted out of the
loop.

8.
100% code coverage does not mean:
you have covered 100% of your object code
Source code Object codeCompiled to

9.
Does it matter which code coverage criteria I'm
using?
Coverage type DO-178B/C
ISO 26262
(Software architecture)
ISO 26262
(unit test)
Function Used with MC/DC
ASIL C, D Highly
Recommended
ASIL A, B Recommended
-
Call Coverage Not required
ASIL C, D Highly
Recommended
ASIL A, B Recommended
-
Statement
Level A, B, C
Required
-
ASIL A, B Highly
Recommended
ASIL C, D Recommended
Decision
Level A, B
Required
- -
Branch Not required -
ASIL B, C, D Highly
Recommended
ASIL A Recommended
MC/DC Level A Required -
ASIL D Highly Recommended
ASIL A, B, C Recommended

10.
What are the barriers to 100% code coverage?
 Missing requirements.
 Missing or incorrect tests.
 Dead code.
 Deactivated code.
 Defensive programming.
 Impossible combinations of events.
 Compiler-introduced errors.

11.
Barrier #1: Missing Requirements
 Add new requirements
 Develop tests
 Run the tests
What to do about it

12.
Barrier #2: Missing/incorrect tests
What to do about it
 Implement additional tests
 Ensure that they trace to
the correct requirement(s)
 Run the new tests
 Implement additional tests
 Ensure that they trace to
the correct requirement(s)
 Run the new tests
What to do about it

13.
Barrier #3: Dead code
foo(a,c){
int b;
mode = MOVING;
if(mode == STATIONARY){
b = 1 + a;
}
c = a;
return c;
}
 Remove the dead code if
possible or
 Justify why the code cannot
be run
What to do about it
DO-178C recommends "The [dead] code should be removed and
an analysis performed to assess the effect and the need for
reverification." [6.4.4.3c]
For ISO 26262, the general guidance for "insufficient" coverage
applies, namely to provide a rationale for why the dead code is
acceptable [6:9.4.5].
What to do about it…

14.
Barrier #4: Deactivated code
settings = read_input(CONFIG_PINS);
if ((settings & CONFIG_MASK_A) == CONFIG_A) {
/* code to execute in configuration A */
...
} else if ((settings & CONFIG_MASK_B) == CONFIG_B) {
/* code to execute in configuration B */
...
}
 Justify why code won’t be
executed
 Never executed
during normal
operation
 Will not affect the
execution of the
system
What to do about it

15.
Barrier #5: Defensive Programming
switch(a % 4){
case 0:
..........
break;
case 1:
..........
break;
case 2:
..........
break;
case 3:
..........
break;
default:
..........
break;
}
 Justify why defensive
programming cannot be
executed
 Provide evidence that if
triggered, the defensive
programming works
correctly
What to do about it

16.
Barrier #6: Impossible Combinations of Events
-- “enabled” might already be set at this point
if speed < 1700 then
enabled := true;
end if;
if speed < 1500 and enabled then
...
end if;
 Simplify the condition?
or
 Justify why 100% could not
be achieved
What to do about it

17.
Barrier #7: Compiler Introduced Errors
int a,x;
if a < 10{ /*Compiler generates “a <=10”*/
result = x / 5;
} else {
result = x / 10;
}
a x
Test case 1 3 4
Test case 2 10 4
 Review existing code base
 Add to coding guidelines /
code review guidelines
What to do about it

18.
Justifying code: in-line justifications
void fatal_error (void)
{
#pragma RVS justification( "COV_FUNCTIONS", "The fatal_error
function cannot be executed during normal execution")
/* some code */
}
...
void check_status (int status)
{
if ( (check_enabled) && (FATAL_ERROR == status) ) {
#pragma RVS justification( "COV_DECISIONS", "Variable 'status'
cannot take value FATAL_ERROR during normal execution")
fatal_error();
}
}
...

19.
Justifying code: justifications in a separate file
#pragma RVS [selftest.c:34] justification("COV_MCDC", "2: (RAM_ERROR ==
status)", "RAM_ERROR cannot be triggered during normal test");
#pragma RVS [selftest.c:439] justification("COV_FUNCTIONS", "Cannot trigger
emergency_shutdown() during normal test");

20.
Justifications editor

21.
Review – establish
missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage

22.
Review – establish
missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
 Missing requirements.
 Missing or incorrect tests.
 Dead code.
 Deactivated code.
 Defensive programming.
 Impossible combinations of events.
 Compiler-introduced errors.

23.
Review – establish
missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
Remove code
Add (or fix)
requirements
Justify why code
won't be executed
Justify why it isn't
possible to test

24.
 reduce testing timescales by running fewer on-target tests
 reduce risk through greater tool flexibility
 reduced effort for certification activities
Structural coverage analysis to DO-178B/C Level A and ISO 26262

25.
Logos and images can be positioned within this area.
Change layout to blank (without position box) before use

26.
Q A