These are slides from a talk I gave at the British Computer Society's SIGIST Conference in June 2013. The talk attempts to provoke the audience into think beyond the current standard approaches for testing in the industry.
14. The Problem
Static analysis tools can identify thousands of alerts
This leads to cognitive overload…
… which in turn leads to…
… the technique being abandoned
Concept: Actionable Alert Identification Techniques (AAIT)
15.
16. Some discoveries
0
1
2
3
4
5
Priority of
alert
Size/Length of
the method
Size/Length of
the class
Age of alert Date of last
modification
Code churn Complexity of
a method
Nesting of a
line of code
Code (line)
coverage
NumberofInterviewees
Very useful Somewhat useful Slightly useful Irrelevant Missing
18. Idea: We should be aware of the effectiveness of our test suites
19.
20. Statement coverage can be misleading
public int multiply(int x, int y){
return (x y);
}
@Test
public void testMultiply(){
assertEquals(5, multiply(5, 1));
}
*/
Passes
x
23. Problems with Mutation Testing
Generating mutants is expensive
Executing tests is expensive
Susceptible to equivalent mutants
Manual investigation of unkilled mutants
25. The results
Code Churn Affected
LOC
Unchanged
code
Decrease
in mutants
Decrease in
execution time
Low 12 99.8% 91% 91%
Medium 60 98.8% 62% 88%
High 720 85.6% 46% 89%
30. Problems with Runtime Testing
Defining mathematical properties may
not come naturally to people
Performance Overheads
How do you fit this into your development
process?
31. Applying Context
Given I am a bank teller processing a transaction
And account 67 has $100 and account 113 has $50
When I transfer $50 from account 67 to account 113
Then account 67 will have $50
And account 113 will have $100
And this should happen within 3 seconds
And the transfer should be logged
And a receipt should be printed
And the respective clients should be notified
Relevant to feature
Cross-Cutting Criteria
34. You can…
Periodically scan academic journals for new ideas
Try Google Scholar, set up alerts for your favourite topics
Sell the benefits of research to yourself and colleagues
Actively seek out collaborations with research groups,
local or otherwise
Consider pursuing research for academic credit yourself
or sponsor and employee to do so
35. Some collaboration models
Provide a case study
Sponsor an intern
Hand off a problem to a research group
Contribute to a research trust