The document discusses bug prediction in software development using code history, highlighting the limitations of traditional practices like code reviews and automated testing. It describes a process of collecting data from commits to improve predictions of bug occurrences, utilizing machine learning tools such as Weka and implementing various classifiers. It concludes by noting the importance of analyzing commit history to reduce bugs and improve team practices.

1. Bug prediction
based on your code history
Alexey Tokar
VP of Engineering @ WorldAPP

2. SDLC
2
Software
Development
Life Cycle
PHASE 2
Design
PHASE 1
Requirements
analysis
PHASE 3
Development
PHASE 5
Maintenance
PHASE 4
Testing

3. Zoom in to quality control
● static code analyzers find non conceptual issues
● automated tests cover predefined scenarios
● code review are aimed on sharing and controlling best practices and less than
10% of the discussions discover logical issues.
● and, finally, QA has no idea which parts of a system could be affected by a
code change, neither do a programmer
3

4. 20
bugs in a production environment
per week
4

5. Let's try to guess common patterns
● a tired engineer makes more mistakes
● the more an engineer knows about certain module the fewer bugs (s)he will
produce
● small changes have fewer bugs than long listings
● some parts of the system are more complicated than another, so the risk of get
a bug increases
● huge changes in a short period of time contains more bugs (done in a hurry)
5

6. What tools do we have across SDLC?
6
● ticket types
● action history
● exact code changes
● author of modifications
● class complexity
● code metrics

7. Hypothesis
If we know that certain commit has fixed a bug, than we know that a commit, when
the changed lines were introduced, did contain a bug.
7
Author: John
public int sum( int a, int b )
{
return a + b;
}
C
Author: Bob
public int sum( int a, int b )
{
return a * b;
}
BA
public int sum( int a, int b )
{
return a + b;
}

8. Algorithm of metrics collection
● Export all tasks from Jira to inmemory dictionary
● For each commit run a backtrace to mark it as buggy, fixing or regular
● Collect all meaningful data about the commit:
○ Month of year, Day of week, Hour of day, Who, How many lines and files, Which classes and
packages, Class complexity and amount of notices, How long a task is in progress
● Put a line with the data to Attribute-Relation File Format (ARFF) file
8

9. Getting educated. WEKA
Waikato Environment for Knowledge Analysis - is a suite of machine learning
software written in Java, developed at the University of Waikato, New Zealand.
● Parsers
● Classifiers
● Training/test splits
9

10. WEKA challenges
● Convert your data to corresponding vectors
● Choose proper data transformers
● Select and tweak desired Classifiers
● Run experiments and adjust your settings
Good materials about WEKA for beginners:
● How to Run Your First Classifier in Weka
● Data mining with WEKA, Part 2. Classification and clustering
● Document Classification using WEKA
10

11. Decision Tree
Ease of results interpretation
Any data can be fed to the method
Can work with scalars and intervals
11

12. Decision Tree
12
Changed less
than 300 lines?Changed more
than 50 lines?
Author is Bob?
Author is John?
Has no bugs :)
Has no bugs :)
Is it Friday?
Has no bugs :)
Has a bug :(
Has no bugs :)
Has a bug :(
● John never has bugs!
● Everybody except John and Bob has bugs on Friday.
● Bob has bugs only if he changed more than 300 lines of code.

13. Decision Tree
13
The simplest method for building a tree is ID3 (Iterative Dichotomiser 3*).
Build steps:
● Find an attribute with lowest entropy (or largest information gain)
● Split the data set by the found attribute
● Recursively build a tree for each of the subsets
* fates of ID2 and ID1 are lost in history

14. Naive Bayes
classifier
≈80% accuracy*
Simple implementation
Easy to understand
14

15. Naive Bayes classifier
15

16. Naive Bayes classifier
16
30% of all commits with bugs were done by Bob P(Bob|bug)
10% of all commits without bugs were done by Bob P(Bob|~bug)
40% of all commits have bugs P(bug)
60% of all commits have no bugs P(~bug)
What probability that next commit from Bob will have a bug?
P(bug|Bob)

17. Support Vector
Machine
Better quality of results
The model is based on relations in data
Sounds fancy :)
17

18. Support Vector Machine
18

19. Support Vector Machine
19
Kernel trick
http://mechanoid.kiev.ua/ml-svm.html

20. WEKA output
results 20

21. Output results example (Bayes)
Correctly Classified Instances 14381 77.4755 %
Incorrectly Classified Instances 4181 22.5245 %
Kappa statistic 0.3085
Mean absolute error 0.2637
Root mean squared error 0.3963
=== Detailed Accuracy By Class ===
TP Rate FP Rate Precision Recall F-Measure ROC Area Class
0.856 0.544 0.861 0.856 0.858 0.761 false
0.456 0.144 0.444 0.456 0.45 0.761 true
Weighted Avg. 0.775 0.463 0.777 0.775 0.776 0.761
=== Confusion Matrix ===
a b <-- classified as
12670 2140 | a = false
2041 1711 | b = true
21

22. Output results example (RandomTree)
form < 1
| Registration < 1
| | alexey.tokar@worldapp.com < 1
| | | tpl < 1
| | | | filters < 1
| | | | | frontend@worldapp.com < 1
| | | | | | middlejava@worldapp.com < 1 : false
| | | | | | middlejava@worldapp.com >= 1 : true
| | | | | frontend@worldapp.com >= 1
| | | | | | ObjectDesign < 1 : true
| | | | | | ObjectDesign >= 1 : false
| | | | filters >= 1 : false
| | | tpl >= 1 : true
| | alexey.tokar@worldapp.com >= 1
| | | bundle < 1
| | | | xmail < 1
| | | | | general < 1
| | | | | | dataimport < 1
| | | | | | | oracle < 1 : false
| | | | | | | oracle >= 1 : true
| | | | | | dataimport >= 1 : false
| | | | | general >= 1
| | | | | | filesedited < 2 : false
| | | | | | filesedited >= 2 : true
| | | | xmail >= 1 : false
| | | bundle >= 1 : true
| Registration >= 1 : true
22

23. Integration example
TL;DR: we use GitLab web-hooks and HipChat.
* a long story about SDLC improvements with help of IM bot and a set of
integrations will be available in a week at XPDays conference.
23

24. Summary
● we found that certain classes are too complex as almost every change will end
up with a bug
● some of engineers shouldn't open some packages at all (or at least we should
properly educate them)
● there are still many rooms for improvements (overlapping hiding commits,
another meaningful features, more accurate code history, etc)
● It does not show you where the error exists. But you will be able to analyze a
commit more carefully.
● It was fun! :)
24

25. Questions?
Alexey@Tokar.net.ua
VP of Engineering @ WorldAPP
25