CodeInsight-SCAM2015
•Download as PPTX, PDF•
0 likes•52 views
The document describes a technique called CodeInsight that mines insightful code comments from crowdsourced knowledge on Stack Overflow. An exploratory study of Stack Overflow discussions found that around 22% of comments discuss tips, bugs, or warnings related to code examples. CodeInsight uses heuristics like popularity, relevance, comment rank, sentiment, and word count to retrieve these insightful comments for a given code segment. An empirical evaluation showed the technique could recall over 80% of relevant comments on average. A user study with professional developers found that 80% of the comments recommended by CodeInsight were accurate and useful.
Recommended
More Related Content
Similar to CodeInsight-SCAM2015
Similar to CodeInsight-SCAM2015 (20)
More from Masud Rahman
More from Masud Rahman (20)
Recently uploaded
Recently uploaded (20)
CodeInsight-SCAM2015
- 1. RECOMMENDING INSIGHTFUL COMMENTS FOR SOURCE CODE USING CROWDSOURCED KNOWLEDGE Mohammad Masudur Rahman, Chanchal K. Roy and +Iman Keivanloo Department of Computer Science University of Saskatchewan, +Queen’s University, Canada 15th International Working Conference on Source Code Analysis and Manipulation (SCAM 2015) Presented by: Jeffrey Svajlenko
- 2. CODE COMMENTS 2 Programmer-readable annotations in the code Explains what the code does and how Make the code easier to understand Part of a good coding practice Does not discuss issues, concerns of the code Does not contain expert’s observation Such insights required for reuse or maintenance
- 3. 3 Android: show soft keyboard automatically when focus is on an EditText
- 4. OUTLINE OF THIS TALK 4 Exploratory study CodeInsight Empirical evaluation (for comment ranking) User study (for comment quality) Conclusion
- 5. EXPLORATORY STUDY Exp-RQ1: Do the follow-up discussions from SO contain any useful information that is likely to aid software maintenance activities (e.g., bug fixation, code quality improvement)? Exp-RQ2: Which API classes and methods are used in SO code examples that might encourage those insightful discussions? 5
- 6. EXPLORATORY DATASET 6 Items Java Android C# Total Questions 98 81 103 282 Accepted answers 98 81 103 282 Code segments 101 83 108 292 Discussion comments 276 161 269 706 The question should be widely viewed (i.e., 500 times), and the answer should contain one or more code segments. The answer should contain at least 10 discussion comments The comments should be up-voted at least by 5 users
- 7. STUDY DESIGN Two separate analyses for two RQs. Careful manual analysis on comment texts Discover the intent behind each comment. Identified 7 types of comments in the discussions Topic modeling and cross-domain analysis for RQ2. 7 Exp-RQ1 Exp-RQ2
- 8. STUDY FINDINGS Seven types of discussion comments found. Particularly interested on two types- Tips and Bugs & warnings. On average, about 22% comments from each domain fall in those types, which is significant. 8
- 9. CODEINSIGHT: PROPOSED TECHNIQUE’S OVERVIEW Insightful comments for a given code segment by exploiting crowd knowledge Extracts discussion comments based on five heuristics. Recommends tips, bugs or concerns in the code identified by the crowd. 9
- 10. HEURISTICS: CAPTURING INSIGHT Popularity (P) Insight makes the comment popular Important observations are paid up by votes Relevance (R) References to relevant API methods or classes in code. Based on cosine similarity measure Comment Rank (CR) User references (e.g., @Thomas) among the comments make them important PageRank for relative importance of any comment Sentiment (S) Polarity based sentiment analysis from comment texts Issues, concerns are mostly associated with negativity Word Count (WC) Too small comments do not contain insight Too big comments are noisy 10
- 11. CODEINSIGHT: MINING INSIGHTFUL CODE COMMENTS FOR SOURCE CODE 11
- 12. EVALUATION OF CODEINSIGHT Two-fold evaluation– (1) comment ranking technique and (2) comment quality RQ1: How effective the technique is in retrieving the comments that discuss bugs, concerns and tips for improvement in the code? RQ2: Are the recommended comments accurate, precise and concise in describing the potential issues or troubleshooting tips? RQ3: Are the recommended comments useful for static analysis involving maintenance of the target code? 12
- 13. EMPIRICAL DATASET Exploratory dataset reused for experiments Manually labeled/classified comments as gold comments Recommended comments matched against gold comments Top 3 comments recommended from the ranking Two performance metrics– Recall (R) and Mean Reciprocal Rank (MRR) 13
- 14. EMPIRICAL EVALUATION: COMMENT RANKING TECHNIQUE Heuristics Metrics Java Android C# Average C3 C4 C3 C4 C3 C4 C3 C4 { P } Recall 60.00% 66.67% 95.65% 87.50% 89.47% 69.44% 81.71% 74.53% MRR 0.44 0.50 0.59 0.57 0.47 0.56 0.50 0.54 {P, R,CR} Recall 66.67% 70.83% 95.65% 84.38% 84.21% 77.78% 82.18% 77.66% MRR 0.60 0.29 0.73 0.56 0.44 0.50 0.59 0.45 { P, R,CR, WC, S } Recall 60.00% 79.16% 95.65% 96.88% 94.74% 86.11% 83.46% 87.38% MRR 0.44 0.32 0.55 0.52 0.33 0.45 0.44 0.43 14 C3 = Tips, C4 = Bugs & warnings P = Popularity, R = Relevance, CR = Comment Rank WC = Word Count, S = Sentiment
- 15. USER STUDY: COMMENT QUALITY Dataset for Study Case study with 82 OS projects from GitHub 85 code segments similar to SO segments Using GitHub code search Study Participants Four professional developers from two companies Specialized in mobile and web technologies Professional experience of 1-2.5 yr. in Java, 1.5-5 yr. in Android and 1.5-3.5 yr. in C# 15
- 16. USER STUDY DESIGN 16 Accurate? Precise? Concise? Useful? OS code Segments (85) Random selector CodeInsight Participant pool Segments (20) Insightful comments Code segments
- 17. USER STUDY: COMMENT QUALITY EVALUATION Responses Average Accurate Precise Concise Useful Strongly agree 82.50% 80.83% 78.33% 79.17% Agree Neutral 6.67% 7.50% 12.50% 10.00% Disagree 10.83% 10.00% 9.17% 10.83% Strongly disagree 17
- 18. THREAT TO VALIDITY Limited dataset: The dataset is quite limited for empirical evaluation. Limited participant pool: Only four professional developers are involved. Closed source code: SO generally hosts code segments from open source code. Thus, the technique might not be effective for closed source code. 18
- 19. TAKE-HOME MESSAGE Traditional code comments do not provide insight on quality or issues in the code. About 22% of discussion comments for Java, Android and C# code segments from SO discuss issues, concerns or tips. Heuristics such as Popularity, Relevance, Comment Rank or Sentiment are found quite successful in capturing such insight in comments. Professional developers confirmed that 80% of the recommended comments are accurate and useful indeed. 19
- 20. THANK YOU!! 20
- 21. PROVOCATIVE STATEMENT Descriptive identifiers vs. inline code comments. Which one will you choose? Why? 21
Editor's Notes
- Introduce yourself +introductory statements. Today, I am going to talk about how can we mine insightful comments for source code from Stack Overflow.
- Code commenting is a very old practice among the developers. Its basically a programmer-readable annotation of the code that explains what the code does and how it does. Such comments are helpful to understand the code in the later time. Meaningful commenting is often considered as a part of good coding style. However, the traditional comment does not provide all information we need. It does not discuss the issues, concerns or the strength of the code. One might think of it’s outside the scope of the code comments. However, imagine if such information is attached to the source code, wouldn’t it be helpful for a developer to change or maintain the code? We argue that the answer is yes!
- Now lets consider this discussion thread from Stack Overflow. The code segment shows how to make the android keyboard visible when a textbox is focused. The code is quite self-documented, easy to understand, and looks good. However, when such code is posted in front of a large technical crowd, the crowd actually does a really good job in analyzing the code. Too many eyeballs, all bugs are shallow. For example, they post concerns, observations or troubleshooting tips which might be non-trivial, and the developer probably never thought of those. Such insights are really important for reuse or further maintenance of that piece of code. In this paper, we exploit these types of information for insightful comment recommendation.
- We first present an exploratory study that investigates/motivates whether the SO discussions contain enough meaningful information or not. Then comes our proposed technique– CodeInsight for comment recommendation. We then perform two types of evaluation– an empirical evaluation of the technique’s comment ranking and a user study for evaluating the quality of the recommended comments. Then we conclude the talk by summarizing the findings.
- In the exploratory study, we attempt to answer these two research questions. We particularly wanted to know if SO discussions contain enough information that is meaningful and also can be mined easily.
- We used a dataset of 706 discussion comments collected from Stack Overflow questions of three domains– Java, Android and C#. The domains are determined based on question tags. We first choose 282 questions from those three domains based on certain restrictions such their view count and code segments in the answers. Then from their accepted answers which are followed by discussion comments, we collect 706 comments. Also we discard the non-voted comments. Thus, 706 comments are chosen for manual analysis out of 5K comments based on their vote counts.
- We performed two different analyses for two research questions of the exploratory study. In case of first RQ1, we perform careful manual analysis on the discussions comments, and attempt to determine their intent. Although the process might be a bit subjective, clearly there exist different intents for different comments. We discover 7 such intents from our observation on the 706 chosen comments. More interestingly, some comments clearly specify the bugs, concerns and limitations of the code whereas some comments provide important tips for improving the code. For RQ2, we apply topic modeling, and the details can be found on the paper.
- As mentioned, we discovered 7 types of comments such as Clarification questions, Documentations, Tips, Bugs or warnings, Strength statements, Miscellaneous and finally non-informative comments. Among those 7 types, two types– Tips and Bugs/warnings draw our attention since they are non-trivial comments and they are mostly from expert people Or people having some working experience on the code. They are not merely clarification questions or thank you! Statements. We then plot the comment statistics based on the types for each domain. Interestingly, we note that on average about 22% of the comments discuss about tips, bugs or concerns, which is significant. This finding motivates us to mine the SO discussions automatically.
- We now discuss our proposed technique for insightful comment recommendation. It recommends comments by exploiting the crowd knowledge. It is also a heuristic-based technique, that means extracts the comments based on certain heuristics. By word “insightful”, we mean those comments that identify bugs, concerns or tips. That means, those comments provide some kind of insight into the quality of the code.
- We apply five heuristics for capturing the insight in the comment texts. Popularity is a very important heuristic—insightful comments are generally popular since the non-trivial observations are paid up by votes in SO community. Relevance between comment text and code is also important. As it uses cosine similarity, it is made sure that the comment is discussing relevant APIs from the code. Comment Rank is basically modeled after the PageRank from IR domain. Discussion comments generally refer to one another for supporting or disagreeing. We capture such user references (@Thomas) to develop a comment interaction graph for each discussion thread. We then apply PageRank algorithm to determine the relative importance of each comment in the graph. Sentiment is another important paradigm for capturing insight. We run Stanford Sentiment Analyzer, which is a polarity-based analyzer. We note that insightful comments discussing issues or concerns generally have a negative polarity. Word Count is another heuristic that we used to discard too short or too long comments. All heuristic measures are normalized before feeding to ranking algorithm.
- The slide shows the different steps of our proposed technique. (1) We first take Stack Overflow code segments and their corresponding discussion comments as input. Then the non-voted or flagged comments are discarded. We don’t get confidence on those comments since the crowd didn’t like them. (2) We calculate heuristics for each of the remaining comments, and rank them based on each individual heuristic. We then combine all the heuristic rankings by applying Borda count. Thus, the comments that topped in most of the rankings are finally selected. The top-ranked comments are then refined so that they get rid or personal or possessive pronouns. (3) Now, we have a code-to-comment collection which can be searched for insightful comments. Now, if one provides a code segment similar to one or more segments from the collection, out technique recommends insightful comments for the given code. We apply OSS code segments for testing.
- Our evaluation is two-fold. First, we evaluate the proposed technique for comment ranking using an empirical evaluation. Then we evaluate the recommended comments using a user study involving professional software developers. We basically attempt to answer these three research questions through our evaluation.
- We reused the exploratory data for our empirical evaluation. In particular, the manually labeled comments are used as gold comments in this experiment. We compare the recommended comments by our technique against the gold comments, and apply two performance metrics – Recall and Mean Reciprocal Rank (MRR) for evaluation.
- In the empirical evaluation, we evaluate our comment ranking technique, and we particularly retrieve insightful comments, that means comments of C3 and C4 categories. We use the manually classified comments from the exploratory study as the gold set for this evaluation. We note that popularity dominates over other heuristics since it alone can retrieve a major portion of the gold comments. However, the performance is only overwhelming for Android domain. We then gradually add more heuristics based on controlled iterations, and finally found that the combination of the five heuristics actually provides the maximum performance for all domains in terms of recall. Thus, finally we get an average recall of 85.42%, which is promising. The findings also suggest that the proposed heuristics are quite successful in capturing the insightful comments. The MRR remains comparable all the time.
- In the second experiment, we evaluate the quality of the recommended comments by our technique using a user study. We first collect 85 code segments from 82 open source projects of GitHub using GitHub code search feature. They are the candidate segments for comment recommendation. We involve four professional developers specialized in mobile and web technologies. They have substantial amount of professional coding experience, and troubleshooting as reported. The goal is to determine whether they find the recommended comments accurate, precise, concise or useful.
- This slides shows the user study design. We use a random selector to choose 20 code segments (5 for Java, 10 for Android and 5 for C#) from the code segment pool for each participant. Then for each segment, our tool CodeInsight recommends three insightful comments. Then both comments and code segments are sent to the participant for review. The participant reviews the comments, and sends his/her evaluation for four different paradigm– accuracy, precision, conciseness and usefulness of the comments. We use a scale from Strongly disagree to Strongly agree for each of those comment quality aspects. Each participant provides a total of 80 data points, and thus we collect 320 data points in total.
- By analyzing the 320 responses from the participants, we get these findings. We report the summary here, the details can be found in the paper. We note that about 80% of the times the recommended comments found accurate, precise, concise or useful on average. We also note some neutral and negative responses which need further analysis, and our algorithm fails to discard those popular but meaningless comments. However, the result also suggests that the proposed technique has high potential for suggesting meaningful insights in the code.
- We identified three threats to the validity of our findings. Limited dataset is used for empirical evaluation. However, manual labeling of comments is costly, and we sample those comments from a large collection based on certain restrictions. Limited no. of participants are involved. However, involving professional developers is always costly, plus we collected a good number of data points for drawing the conclusion The technique might not be effective for closed source code since SO does not host closed source code segments.
- OK these are take-home messages. Traditional comments are not insightful. Roughly about 22% of the discussion comments from SO code examples discuss about issues, concerns or tips for the code. Heuristics like Popularity, Relevance, Comment Rank or Sentiment are found quite effective in capturing such insight in the comments. Professional developers also confirmed the usefulness or quality of 80% of the recommended comments.
- Thanks for your time. Questions!!