The document discusses the concept of pseudo-tested methods in software testing, highlighting their prevalence in Java open source projects and the potential to identify flaws in test suites. By using custom tooling, the study illustrates how pseudo-tested methods can provide concrete hints for improving test cases and assesses their relevance based on usage in the code. The findings underscore the importance of automated solutions for resolving these testing flaws, which could enhance software reliability.

1. Finding the worst tested methods
in a test suite
Oscar L. Vera
DiverSE – Inria Rennes
Mutation Testing Workshop
April 12th, 2019

2. Unit test case example
class VList {
private List elements;
private int version;
public void add(Object item) {
elements.add(item);
incrementVersion();
}
public int size () {
return elements.size();
}
private void incrementVersion () {
version ++;
}
}
class VListTest
@Test
public void testAdd (){
VList l = new VList();
l.add(1);
assertEquals(l.size(), 1);
}
}
2

3. Developers
• Do our tests assess the behavior of the program?
• Concrete improvements
3

4. Introduced in 2016
R. Niedermayr, E. Juergens, and S. Wagner
Will my tests tell me if I break this code?
Proceedings of the International Workshop on Continuous Software Evolution and Delivery, 2016, pp. 23–29.
Pseudo-tested methods
4

5. • Executed by the test suite
• The code can be removed
• No test case notice the change
public void setValue(int x) {
// pass
}
public void setValue(int x) {
if(a + x < 10)
a += x;
}
public int fact(int x) {
int result = 0;
for(int i=1; i <= x; i++) {
result *= i;
}
return result;
}
public int fact(int x) {
return 42;
}
Pseudo-tested methods
5

6. A pseudo-tested method
class VList {
private List elements;
private int version;
public void add(Object item) {
elements.add(item);
incrementVersion();
}
public int size () {
return elements.size();
}
private void incrementVersion () {
version ++;
}
}
class VListTest
@Test
public void testAdd (){
VList l = new VList();
l.add(1);
assertEquals(l.size(), 1);
}
}
Pseudo-tested
6

7. Can pseudo-tested methods provide concrete hints to
improve a test suite?
7

8. Conceptual replication
• Additional set of study subjects
• 21 Java open source projects
• Custom and dedicated tooling
• Descartes https://github.com/STAMP-project/pitest-descartes
8

9. Data: P /*Program*/, TS /*Test suite*/
Result: pseudo {pseudo-tested methods in P}
foreach m in P | covered(m, TS) and ofInterest(m):
variants = {extreme variants of m}
if returnsValue(m):
stripBody(m)
checkReturnType(m)
variants ! fixReturnValues(m)
else:
variants ! stripBody(m)
failure = false
foreach v in variants:
P’ ! replace(m, v, P)
failure ! failure or fails(TS, P’)
if no failure:
pseudo add m
Create method variants
Verify if the test suite fails
9

10. Excluded methods
• Not covered by any test
• Compiler generated methods
• Returning a literal constant
• Simple getters
• Simple setters
• Deprecated
• hashCode
• Empty
10
public int m () { return 3; }
public int getI () { return this.i; }
public void setI (int i) { this.i = i; }
public void m () { }
@Deprecated public void m () {...}
public int hashCode () {...}

11. Pseudo-tested methods
0
125
250
375
500
AuthZForce PDP Core Apache Commons Codec Apache Commons Lang Jaxen XPath Engine Joda-Time SAT4J Core Spoon
239
28
213
72
473
143
28
2
82
296
476
1110
100
47
29
40
12
2
224
13
11

12. Ratio of pseudo-tested methods
12
0 %
25 %
50 %
75 %
100 %
AuthZForce PDP Core Apache Commons Codec Apache Commons Lang Jaxen XPath Engine Joda-Time SAT4J Core Spoon
91 %
99 %
93 %
69 %
83 %80 %
96 %99 %97 %
90 %88 %
98 %98 %95 %98 %96 %97 %97 %99 %
89 %
96 %
9 %1 %7 %
31 %
17 %20 %
4 %1 %3 %
10 %12 %
2 %2 %5 %2 %4 %3 %3 %1 %
11 %
4 %
Pseudo-tested Tested

13. Real-life examples

14. Apache Commons Codec
public void testIsEncodeEquals() {
final String[][] data = {
{"Meyer", "Mu00fcller"},
{"Meyer", "Mayr"},
...
{"Miyagi", "Miyako"}
};
for (final String[] element : data) {
final boolean encodeEqual =
this.getStringEncoder().isEncodeEqual(element[1], element[0]);
}
}
No assertion
Pseudo-tested
14

15. Apache Commons IO
Weak oracle
Result is the same if nothing is written
Pseudo-tested
public void testTee() {
ByteArrayOutputStream baos1 = new ByteArrayOutputStream();
ByteArrayOutputStream baos2 = new ByteArrayOutputStream();
TeeOutputStream tos = new TeeOutputStream(baos1, baos2);
...
tos.write(array);
assertByteArrayEquals(baos1.toByteArray(), baos2.toByteArray());
}
15

16. Apache Commons Collection
abstract class AbstractHashedMap<K, V> {
protected void ensureCapacity(final int newCapacity) {
final int oldCapacity = data.length;
if (newCapacity <= oldCapacity) {
return;
}
if (size == 0) {
threshold = calculateThreshold(newCapacity, loadFactor);
data = new HashEntry[newCapacity];
}
else {
//Rehash code
...
threshold = calculateThreshold(newCapacity, loadFactor);
data = newEntries;
}
}
...
}
Covered by 38 test cases
Executed 11593 times
16
Pseudo-tested

17. Apache Commons Collections
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new UnsupportedOperationException();
}
...
}
class SingletonListIteratorTest {
...
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
...
}
No exception is thrown
A fail is needed here
Pseudo-tested
17

18. Amazon Web Services SDK
class SdkTLSSocketFactory {
protected void prepareSocket(SSLSocket s){
...
s.setEnabledProtocols(protocols);
...
}
}
@Test
void typical() {
SdkTLSSocketFactory f = ...;
f.prepareSocket(new TestSSLSocket() {
@Override
public void setEnabledProtocols
(String[] protocols) {
assertTrue(
Arrays.equals(protocols, expected));
}
...
});
}
Pseudo-tested
No assertion is verified if
the method is emptied
18

19. Relevant
• Widely used in the code
• Relevant for external clients
• Supports a core functionality
Not relevant
• Part of debug functionalities
• Not widely used
• Deprecated
• Simple or trivial
• Placeholders
30% of pseudo-tested methods
considered as relevant by developers
Decision is subjective in many cases
19

20. Eight selected issues
• 4 communicated directly
• 1 issue created in the tracker but no action
• 2 direct commits with fixes
• 4 pull requests fixing the issues
• All accepted
20

21. Conclusions
• Pseudo-tested methods may uncover real testing flaws
• The testing issues are easy to understand
• These flaws can be relevant
• Room for automation
21

22. Current work
Can these issues be automatically solved?
• State-of-the-art test generation tools
• Test improvement tools
22