MSR2017-RevHelper
•Download as PPTX, PDF•
0 likes•100 views
Predicting Usefulness of Code Review Comments using Textual Features and Developer Experience
Recommended
More Related Content
What's hot
What's hot (17)
Similar to MSR2017-RevHelper
Similar to MSR2017-RevHelper (20)
More from Masud Rahman
More from Masud Rahman (20)
Recently uploaded
Recently uploaded (20)
MSR2017-RevHelper
- 1. PREDICTING USEFULNESS OF CODE REVIEW COMMENTS USING TEXTUAL FEATURES AND DEVELOPER EXPERIENCE Mohammad Masudur Rahman, Chanchal K. Roy, and Raula G. Kula* University of Saskatchewan, Canada, Osaka University, Japan* 14th International Conference on Mining Software Repositories (MSR 2017), Buenos Aires, Argentina
- 2. RESEARCH PROBLEM: USEFULNESS OF CODE REVIEW COMMENTS 2 What makes a review comment useful or non-useful? 34.5% of review comments are non- useful at Microsoft (Bosu et al., MSR 2015) No automated support to detect or improve such comments so far
- 3. STUDY METHODOLOGY 3 1,482 Review comments (4 systems) Manual tagging with Bosu et al., MSR 2015 Non-useful comments (602) Useful comments (880) (1) Comparative study (2) Prediction model
- 4. COMPARATIVE STUDY: VARIABLES Independent Variables (8) Response Variable (1) Reading Ease Textual Comment Usefulness (Yes / No) Stop word Ratio Textual Question Ratio Textual Code Element Ratio Textual Conceptual Similarity Textual Code Authorship Experience Code Reviewership Experience External Lib. Experience Experience 4 Contrast between useful and non-useful comments. Two paradigms– comment texts, and commenter’s/developer’s experience Answers two RQs related to two paradigms.
- 5. ANSWERING RQ1: READING EASE Flesch-Kincaid Reading Ease applied. No significant difference between useful and non- useful review comments. 5
- 6. ANSWERING RQ1: STOP WORD RATIO Used Google stop word list and Python keywords. Stop word ratio = #stop or keywords/#all words from a review comment Non-useful comments contain more stop words than useful comments, i.e., statistically significant. 6
- 7. ANSWERING RQ1: QUESTION RATIO Developers treat clarification questions as non-useful review comments. Question ratio = #questions/#sentences of a comment. No significant difference between useful and non-useful comments in question ratio. 7
- 8. ANSWERING RQ1: CODE ELEMENT RATIO Important code elements (e.g., identifiers) in the comments texts, possibly trigger the code change. Code element ratio = #source tokens/#all tokens Useful comments > non-useful comments for code element ratio, i.e., statistically significant. 8
- 9. ANSWERING RQ1: CONCEPTUAL SIMILARITY BETWEEN COMMENTS & CHANGED CODE How relevant the comment is with the changed code? Do comments & changed code share vocabularies? Yes, useful comments do more sharing than non-useful ones, i.e., statistically significant. 9
- 10. ANSWERING RQ2: CODE AUTHORSHIP File level authorship did not make much difference, a bit counter-intuitive. Project level authorship differs between useful and non-useful comments, mostly for Q2 and Q3 10
- 11. ANSWERING RQ2: CODE REVIEWERSHIP Does reviewing experience matter in providing useful comments? Yes, it does. File level reviewing experience matters. Especially true for Q2 and Q3. Experienced reviewers provide more useful comments than non- useful comments. 11
- 12. ANSWERING RQ2: EXT. LIB. EXPERIENCE Familiarity with the library used in the changed code for which comment is posted. Significantly higher for the authors of useful comments for Q3 only. 12
- 13. SUMMARY OF COMPARATIVE STUDY 13 RQ Independent Variables Useful vs. Non-useful Difference RQ1 Reading Ease Not significant Stop word Ratio Significant Question Ratio Not significant Code Element Ratio Significant Conceptual Similarity Significant RQ2 Code Authorship Somewhat significant Code Reviewership Significant External Lib. Experience Somewhat significant
- 14. EXPERIMENTAL DATASET & SETUP 14 1,482 code review comments Evaluation set (1,116) Validation set (366) Model training & cross-validation Validation with unseen comments
- 15. REVHELPER: USEFULNESS PREDICTION MODEL 15 Review comments Manual classification using Bosu et al. Useful & non-useful comments Model training Prediction model New review comment Prediction of usefulness for a new review comment to be submitted. Applied three ML algorithms– NB, LR, and RF Evaluation & validation with different data sets Answered 3 RQs– RQ3, RQ4 and RQ5
- 16. ANSWERING RQ3: MODEL PERFORMANCE Learning Algorithm Useful Comments Non-useful Comments Precision Recall Precision Recall Naïve Bayes 61.30% 66.00% 53.30% 48.20% Logistic Regression 60.70% 71.40% 54.60% 42.80% Random Forest 67.93% 75.04% 63.06% 54.54% 16 Random Forest based model performs the best. Both F1-score and accuracy 66%. Comment usefulness and features are not linearly correlated. As a primer, this prediction could be useful.
- 17. ANSWERING RQ4: ROLE OF PARADIGMS 17
- 18. ANSWERING RQ4: ROLE OF PARADIGMS 18
- 19. ANSWERING RQ5: COMPARISON WITH BASELINE (VALIDATION) 19
- 20. ANSWERING RQ5: COMPARISON WITH BASELINE (ROC) 20
- 21. TAKE-HOME MESSAGES Usefulness of review comments is complex but a much needed piece of information. No automated support available so far to predict usefulness of review comments instantly. Non-useful comments are significantly different from useful comments in several textual features (e.g., conceptual similarity) Reviewing experience matters in providing useful review comments. Our prediction model can predict the usefulness of a new review comment. RevHelper performs better than random guessing and available alternatives. 21
- 22. THANK YOU!! QUESTIONS? 22 Email: chanchal.roy@usask.ca or masud.rahman@usask.ca
Editor's Notes
- Introduce yourself +introductory statements. Code review is a widely used practice for modern software development, and review comments are the building blocks for code review. However, it is often difficult to find out whether a review comment is useful or not. In my talk, I am going to discuss which factors might be related to the usefulness of code review comments.
- For example, these are two review comments. This one triggers a new code change whereas the other one does not. Now, professional developers @Microsoft considered the change-triggering comment as the useful one. Now, in this research, we try to answer what makes a review comment useful? If not useful, can we identify that before sending them to the patch submitters? About 35% of the review comments are found non-useful at Microsoft, which is a significant amount. Unfortunately, there exist no automated supports for detecting or improving such comments so far. So, here comes our study.
- So, in this research, we try to understand what makes a review comment useful. For that, we need real code review data. So, we collaborated with Vendasta, a local software company with 150+ employees, and collect about 1,500 recent review comments from four of their systems. Then we apply the change-triggering heuristic of an earlier study, and manually annotate the comments. That is, if a review comment triggers new code change in its vicinity, it is a useful comment and vice versa. Now, we have to distinct sets of comments. Now we do comparative analysis between them, which is our first contribution. Then, we also develop a prediction model so that the non-useful comments can be predicted before submitting to the developers. The goal is to improve the non-useful review comments through identification and possible suggestions.
- In the comparative study, we contrast between useful and non-useful comments. We have independent and response variables. We consider two paradigms– comment texts, and the corresponding developer’s experience. In the textual aspect we consider five features of the comments such as reading ease, code element ratio or conceptual similarity between comment texts and the changed code. In the experience paradigm, we consider the authorship, review experience, and library experience of the developers. We have one response variable, that is the comment is useful or not– yes or no.
- We applied Flesch-Kincaid reading ease to both review comments which returns a ease score between 0 to 100. We did not find any significant difference in these scores for useful and non-useful comments.
- The second variable is stop word ratio. We used Google’s stop word list and Python keyword list since our code base is of Python. We found that non-useful comments contain significantly higher ratio of stop words than useful comments. The findings is also confirmed from quartile level analysis.
- Developers often treat clarification question as a sign of non-useful comments. So, we determine question ratio = # of questions/#sentences for each of the comments. We did not find any such evidence of developers claim. There is no significant difference between useful and non-useful comments in the question ratio
- From our manual observations, we note that review comments often contain relevant code elements such as method names, or identifiers. We also contrast on code element ratio between useful and non-useful comments. We found that useful comments contain more code elements than non-useful comments.
- How relevant the comment is against the changed code for which the comment is posted? This can be determined between lexical similarity between comments and the changed code. Similar concepts are applied with Stack Overflow questions and answers. We found that useful comments are conceptually more similar to their target code, than the non-useful comments. That is, you have to use familiar vocabulary to patch submitters to make your comment useful.
- As opposed to earlier findings, file level authorship did not make too much difference. That is, 46% of the useful comment authors changed file at least once. The same statistic for non-useful comments is 49% However, in the case with project level authorship of the developers, we found some difference between useful and non-useful comments. Mostly for Q2 and Q3. For more details, please read the paper.
- We also investigate if the reviewing experience is different between the authors of useful and non-useful comments. We found that file level reviewing experience matters here. Reviewing experience reduces the non-useful comments from a developer. Also experienced developers provide more useful comments than less experienced developers.
- We repeat the same for the external library experience, and found that developer’s experience for useful comments is slightly higher than that of non-useful comments.
- So, these are the summaries from our comparative study. We found three textual features are significantly different between useful and non-useful comments which are interesting. No studies explored these aspects earlier. We also confirmed some of the earlier findings on the developer experience paradigm as well.
- We divide our dataset into two groups– evaluation set and validation set. With the evaluation set, we evaluate our technique with 10-fold cross validation. Using the validation set, we validate the performance of our prediction model.
- These are the steps of our model development. We first annotate the comments using change-triggering heuristic. Then we extract the textual and experiment features of the comments, train our model using three learning algorithms. Then we use our model to predict whether a new review comment is useful or not. If a review comment is not useful, we can explain that with our identified features as well.
- Our evaluation shows that RF-based model performs the best, especially it can identify the non-usefulness comments more than the other variants. Since RF is a tree-based model, it also confirms that comment usefulness and features are NOT linearly correlated. The model provides 66% accuracy with 10-fold cross validation. As a state-of-the-art, this could be useful. Something is always better than nothing we believe.
- We also investigate how each of the features perform or contribute during comment usefulness prediction. We see that Conceptual Similarity, a textual feature has the strongest role in the prediction. For example, if we discard that feature, our prediction accuracy drops by 25% The next important features are reviewing experience stats of the developers.
- We also contrast between two group of features– textual features and experience features. While experience is a strong metric overall, it has limited practical use/applicability during usefulness prediction. That is why, we introduce the textual features. It also shows some interesting results. ROC and PR-curve show that our prediction is much better than random guessing.
- We compare we five variants of a baseline model – Bosu et al., MSR 2015 That model considers a set of keywords and sentiment analysis to separate useful and non-useful comments. However, according to our experiments, these metrics are not sufficient enough, especially for usefulness prediction. We see that our identified features are more effective, and prediction much more accurate than theirs.
- This is ROC for our model and the competitive models. While the baseline models are close to random guessing, we are doing much better. Obviously, these can be further improved with more accurate features and possibly more training dataset.
- Possibly you can read them out I guess.
- Thanks for your time. Questions!!