This document analyzes the relationship between software bug type and the number of factors involved in failures. It finds that most bugs are caused by one or two factors, while progressively fewer bugs involve more factors. It also finds that complex "Mandelbug" software bugs fall within the range that can be tested using conventional combinatorial testing techniques. This suggests that combinatorial testing can achieve confidence levels similar to exhaustive testing for more complicated bugs.

1. The Relationship Between Software Bug Type
and Number of Factors Involved in Failures
Zachary Ratliff
Computer Security Division
Security Components & Mechanisms
6 August 2015

2. Background
Why do we want to determine this relationship?
• Help us better understand detection differences between
different types of software bugs
• Determine if complex software bugs fall within range for
using conventional covering arrays as seen in previous
studies
This is the first study looking at this relationship
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 1

3. Trend of Software Interaction Strength
• The majority of faults are
induced by a single factor or
the joint combinatorial
effect of two factors.
• Progressively fewer faults
are induced by 3 or more
factors.
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 2
N-wise covering array

4. Counting Factors
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 3
• Two factors that must be present to trigger fault
• Could be detected with a 2-way covering array
Format String bug when:
n = %x%x%x%x & privs = true

5. Covering Arrays
• Each row is a single test
• • 2³ = 960 possible
3-way combinations
• All 3-way combinations
generated with 13 tests
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 4
3-Way Covering Array
10
3

6. Applications of Combinatorial Testing
Lockheed Martin joint work with NIST applied basic
combinatorial testing to 8 different pilot projects
• Could save up to 20% of testing costs if used early
enough on a program
• Could increase test coverage by 20% to 50%
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 5

7. Classifications of Software Bugs
• Bohrbug (BOH)
• Non Age-Related Mandelbug (NAM)
• Age-related bug (ARB)
• Unknown (UNK)
Research at Duke University looking at different aspects of
failures
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 6

8. Bohrbug
•Only factors
•Less complicated to
reproduce
Mandelbug
•At least one
factor
•May have direct factors
•More complex error
propagation
•Include age-related bugs
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 7
direct indirect

9. Direct versus Indirect Factors
Direct
• User input causes buffer
overflow
• Wrong output when a negative
number is passed to a function
Indirect
• The internal clock is out of sync
with real time causing a
calculation to be incorrect
• Program crashes only when ran
on a x86 Windows 7 operating
system
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 8

10. MySQL Data
51.65%
27.69%
6.20%
14.46%
BOH NAM ARB UNK
242 total bugs
125 BOH
67 NAM
15 ARB
35 UNK
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 9
We could not determine the
factors present in approximately
25% of the bugs analyzed

11. Coverage Analysis
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 10
1 2 3 4 5
Bohrbugs 41% 84% 97% 100% -
Mandelbugs 31% 69% 90% 97% 100%
• MySQL software bugs followed the same trend as software in
previous studies
• Determined that Mandelbugs had a significantly higher
average interaction strength

12. MySQL Interaction Strength
0
20
40
60
80
100
1 2 3 4 5
SQL-Bohrbugs SQL-Mandelbugs
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 11
Coverage(%)

13. Discovery of MySQL Software Bugs
• Are bugs with higher interaction strengths found later
in the software life-cycle?
Considerations:
-Increase of users
-Better testing practices
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 12

14. Bug Discovery
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 13

15. What it means for Software Testers
•Finding Mandelbugs during software testing can be
very expensive and time consuming
•In the past, testers failed to locate complex bugs
early in the software lifecycle
The interaction strength of a software bug
correlates with the time period of the bug’s
discovery
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 14

16. •Complex software bugs i.e. Mandelbugs, fall
within range for conventional covering arrays
used in combinatorial testing
•A confidence level similar to that of exhaustive
testing can be achieved when using combinatorial
testing for complex software bugs
This is contingent on indirect factors being present at the time of testing
-More studies needed to further firm up our conclusions
-Currently working on analyzing NASA software bugs
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015 15

17. Combinatorial Testing Tools
•ACTS – Automated Combinatorial Testing for
Software
http://csrc.nist.gov/groups/SNS/acts/index.html
•ComTest – Combinatorial Testing Plugin for the
Eclipse IDE
https://github.com/comtest/comtestnist
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015

18. • Rick Kuhn, National Institute of Standards and Technology
• Raghu Kacker, National Institute of Standards and Technology
• Kishor Trivedi, Duke University
• Michael Grottke, University of Erlangen-Nuremberg
• Zheng Zheng, Beijing University of Aeronautics and
Astronautics
• Allen Nikora, NASA / Jet Propulsion Laboratory
• SURF Program, National Institute of Standards and
Technology
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015
Special Thanks

19. References
1. Covering arrays: D.R. Kuhn, R.N. Kacker, Y. Lei, J. Hunter, “Combinatorial
Software Testing”, IEEE Computer Society, August 2009
2. Distribution of faults: D.R. Kuhn, D.R. Wallace, A.M. Gallo, Jr., “Software
Fault Interactions and Implications for Software Testing”, IEEE
Transactions on Software Engineering, June 2004
3. Lockheed Martin / NIST study: J. Hagar, T. Wissink, D.R. Kuhn, R.N. Kacker,
“Introducing Combinatorial Testing in a Large Organization”, IEEE
Computer, April 2015
4. Software Bug Classifications / Indirect vs. Direct Factors: M. Grottke, A.P.
Nikora, K.S. Trivedi, “An Empirical Investigation of Fault Types in Space
Mission System Software”, IEEE/IFIP International Conference on
Dependable Networks and Systems(DSN), June/July 2010
Zachary Ratliff
(Computer Security / Security Components & Mechanisms)
The Relationship Between Bug Type and Number of Factors
Involved in Failures
6 August 2015