The document discusses automating bug prediction based on a developer's code history. It details various metrics that can be collected from code commits, issue tracking systems, and code reviews to train machine learning models to predict whether a commit is likely to contain a bug. Decision trees and naive Bayes classifiers are proposed to classify commits based on attributes like author, lines of code changed, complexity of modified classes, and other code metrics. The models can help prioritize code reviews and testing but do not pinpoint the location of errors. Overall, the process showed promise while also identifying opportunities for improving the models through additional data and features.

1. Bug prediction
based on your code history

2. 2
3y 4y 5y 2y
Developer Developer & Founder Team Lead & Architect VP of Engineering
@ Various companies @ Startup @ Yandex @ WorldAPP
13 years in practical engineering
80 people at the department
10+ projects from initial commit to production

3. backlogDEV + QA
Automation engineers

4. Automate bug
searching

5. What already have been automated
5
MaintenanceDevelop Test
● Unit and Integration tests run on every commit to MR branches
● Static code analysis on each push
● Cross references between GitLab and Jira
● HipChat notifications about created Merge Requests

6. What already have been automated
6
MaintenanceDevelop Test
● Deploy a successful build to the test environment
● UI and Performance tests run on every commit to a develop branch
● Check against different types of supported DBMS

7. What already have been automated
7
MaintenanceDevelop Test
● Deploy a successful build to the production environment
● Grafana alerting to HipChat

8. Issues with these opportunities
● static code analyzers find only non conceptual issues
● automated tests cover only predefined scenarios
● code reviews are aimed on sharing and controlling best practices and less
than 10% of all discussions discover logical issues.
● and, finally, QA has no idea which parts of the system could be affected by a
code change… neither do a programmer
8

9. 20
bugs in a production environment
per week
9

10. A guess. Let's examine human factor
● a tired engineer makes more mistakes
● the more an engineer knows about certain module the fewer bugs (s)he will
produce
● small changes have fewer bugs than long listings
● some parts of the system are more complicated than another, so the risk of get
a bug increases
● huge changes in a short period of time contains more bugs (done in a hurry)
10

11. Hypothesis
If we know that certain commit has fixed a bug, than we know that a commit, when
the changed lines were introduced, did contain the bug.
11
Author: John
public int sum( int a, int b )
{
return a + b;
}
C
Author: Bob
public int sum( int a, int b )
{
return a * b;
}
BA
public int sum( int a, int b )
{
return a + b;
}

12. What tools can help us?
12
● ticket types
● action history
● exact code changes
● author of modifications
● class complexity
● code metrics

13. Our new team member. Overlord
13

14. WebHooks
ScheduledExecutorService
14
java.util.concurrent.

15. Improve cross references between tools
15
● Notifies about missed ticket key in MR title
● Fills MR with information from Jira
● Fixes common mistakes in MR creation

16. Propose the best reviewers based on MR changeset
16
● Who previously has edited the touched code lines
● Who has coded more than others in the files
● Who is team lead / owner of the service / package

17. Task updates according to the workflow
17
● Transitions task status
● Assigns proper person for the next step
● Marks if task has SQL changes
● Adds a label with branch merged into

18. Check that MR has 2 upvotes before merging
18
● Check that rules are followed
● Notify TeamLead / Dev manager about any
violation
● Push an author to ask colleagues to look at his
masterpiece

19. Another automated processes
● Notifies author about old MR without any reactions
● Notifies assignee that MR can be merged
● Notifies if you have lots of “In Progress” tickets or don’t have them at all
● Provides a list of merged tasks in the particular branch
19

20. Now we have got all the data

21. Algorithm of metrics collection
● Export all tasks from Jira to inmemory dictionary
● For each commit run a backtrace to mark it as buggy, fixing or regular
● Collect all meaningful data about commit:
○ Month of year, Day of week, Hour of day, Who, How many lines and files, Which classes and
packages, Class complexity and amount of notices, How long a task is in progress
● Put a line with the data to Attribute-Relation File Format (ARFF) file
21

22. Getting educated. WEKA
Waikato Environment for Knowledge Analysis - is a suite of machine learning
software written in Java, developed at the University of Waikato, New Zealand.
● Parsers
● Classifiers
● Training/test splits
22

23. WEKA challenges
● Convert your data to corresponding vectors
● Choose proper data transformers
● Select and tweak desired Classifiers
● Run experiments and adjust your settings
Good materials about WEKA for beginners:
● How to Run Your First Classifier in Weka
● Data mining with WEKA, Part 2. Classification and clustering
● Document Classification using WEKA
23

24. Decision Tree
Ease of results interpretation
Any data can be fed to the method
Can work with scalars and intervals
24

25. Decision Tree
25
Changed less
than 300 lines?Changed more
than 50 lines?
Author is Bob?
Author is John?
Has no bugs :)
Has no bugs :)
Is it Friday?
Has no bugs :)
Has a bug :(
Has no bugs :)
Has a bug :(
● John never has bugs!
● Everybody except John and Bob has bugs on Friday.
● Bob has bugs only if he changed more than 300 lines of code.

26. Decision Tree
26
The simplest method for building a tree is ID3 (Iterative Dichotomiser 3*).
Build steps:
● Find an attribute with lowest entropy (or largest information gain)
● Split the data set by the found attribute
● Recursively build a tree for each of the subsets
* fates of ID2 and ID1 are lost in history

27. Naive Bayes
classifier
≈80% accuracy*
Simple implementation
Easy to understand
27

28. Naive Bayes classifier
28

29. Naive Bayes classifier
29
30% of all commits with bugs were done by Bob P(Bob|bug)
10% of all commits without bugs were done by Bob P(Bob|~bug)
40% of all commits have bugs P(bug)
60% of all commits have no bugs P(~bug)
What probability that next commit from Bob will have a bug?
P(bug|Bob)

30. Output results example (Bayes)
Correctly Classified Instances 14381 77.4755 %
Incorrectly Classified Instances 4181 22.5245 %
Kappa statistic 0.3085
Mean absolute error 0.2637
Root mean squared error 0.3963
=== Detailed Accuracy By Class ===
TP Rate FP Rate Precision Recall F-Measure ROC Area Class
0.856 0.544 0.861 0.856 0.858 0.761 false
0.456 0.144 0.444 0.456 0.45 0.761 true
Weighted Avg. 0.775 0.463 0.777 0.775 0.776 0.761
=== Confusion Matrix ===
a b <-- classified as
12670 2140 | a = false
2041 1711 | b = true
30

31. Output results example (RandomTree)
form < 1
| Registration < 1
| | alexey.tokar@worldapp.com < 1
| | | tpl < 1
| | | | filters < 1
| | | | | frontend@worldapp.com < 1
| | | | | | middlejava@worldapp.com < 1 : false
| | | | | | middlejava@worldapp.com >= 1 : true
| | | | | frontend@worldapp.com >= 1
| | | | | | ObjectDesign < 1 : true
| | | | | | ObjectDesign >= 1 : false
| | | | filters >= 1 : false
| | | tpl >= 1 : true
| | alexey.tokar@worldapp.com >= 1
| | | bundle < 1
| | | | xmail < 1
| | | | | general < 1
| | | | | | dataimport < 1
| | | | | | | oracle < 1 : false
| | | | | | | oracle >= 1 : true
| | | | | | dataimport >= 1 : false
| | | | | general >= 1
| | | | | | filesedited < 2 : false
| | | | | | filesedited >= 2 : true
| | | | xmail >= 1 : false
| | | bundle >= 1 : true
| Registration >= 1 : true
31

32. Summary
● we found that certain classes are too complex as almost every change in them
will end up with a bug
● some of engineers shouldn't open some packages at all (or at least we should
properly educate them)
● there are still many rooms for improvements (overlapping hiding commits,
another meaningful features, more accurate code history, etc)
● It does not show you where an error exists. But you will be able to analyze a
commit more carefully.
● It was fun! :)
32

33. Questions?
Alexey@Tokar.net.ua
VP of Engineering @ WorldAPP
33