Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Summarization Techniques for Code, Change, Testing and User Feedback

181 views

Published on

Summarization Techniques for Code, Change, Testing and User Feedback - Validation, Analysis and Evolution of Software Tests (2018)

  • Be the first to comment

  • Be the first to like this

Summarization Techniques for Code, Change, Testing and User Feedback

  1. 1. Sebastiano Panichella Institut für Informatik Universität Zürich panichella@ifi.uzh.ch Validation, Analysis and Evolution of Software Tests (2018) Summarization Techniques for Code, Change, Testing and User Feedback
  2. 2. My Background University of Sannio PhD Student June 2014 University of Salerno Master Student December 2010 University of Zurich Research Associate Since October 2014 2 University of Molise Bachelor Student
  3. 3. My Background University of Sannio PhD Student June 2014 University of Salerno Master Student December 2010 University of Zurich Research Associate Since October 2014 3 University of Molise Bachelor Student VST 2018
  4. 4. My Research Interests 4 software learning developers maintenance testingchanges machine program comprehension test code study method reviews prediction number defects classes precision set source information projects dataapp user cost project empirical summaries bugs sentences models model defect analysis engineering modep methods work case cases emails api surf users setup evolution releases class bug decision participants recall upgraded documents descriptions specific automated timefeedback chat generated review defectprone example communication training terms paragraphs techniques release regression step issues reports sentence predictors logistic metrics coverage multiobjective research issue apps found problemlanguage singleobjective features apache java categories inspection useful summary tree http dependencies page sources topics lucene automatically mining systems algorithm process words extracted classification size performed analyzed mobile category values system topic saved natural level change crossproject mailing context experiment tasks commandsfolders commandtxt printed analyzing developer heuristics nlp tests generation development traceability kloc ecosystem total feature validity lists exception evaluation links text documentation description value eclipse identify average quality reported instead similar perform approaches effectiveness client consider identified comments summarization order traditional possible analyze manually recovery constraints compared defined performance working authors large document important extract future finally contain classifier open relevant aim name several help content impact list loc algorithms percentage upgrade Word-cloud from my research papers
  5. 5. Why Software Maintenance and Testing are Relevant in our Society? 5
  6. 6. The Cost of Software Bugs National Institute of Standards and Technology $ 59.5 cost to US economy in 2013 40%-50% could be saved with proper Software Maintenance and Testing billion $ 312 billion worldwide cost in 2013 6
  7. 7. Current Release Cycle Release Planning 1 Coding Activity Testing Tasks Deployment Phase Source Code Release App Reviews User (or costumer) Feedback 2 3 5 6 CD/CI 4 Test Code 7
  8. 8. Current Release Cycle Manual testingAutomated testing CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … …… … Software documentation Maintenance activities Coding activities 8
  9. 9. “In modern software companies it is nowadays, crucial to enact a software development process able to dynamically react to market requirements (i.e., users requests), delivering at same time high quality and reliable software”. Current Release Cycle 9
  10. 10. What are the Most Expensive Activities in Software Maintenance and Testing? 10
  11. 11. What are the Most Expensive Activities in Software Maintenance and Testing? What are the Open Challenges? 11 Empirical Studies to Understand Developers’ Behaviour Tools for Automating Software Maintenance and Testing How to Identify such Activities? How to Reduce the Cost of such Activities?
  12. 12. Outline What are the Open Challenges? 12 Tools for Automating Software Maintenance and Testing How to Identify such Activities? How to Reduce the Cost Activities? PART I PART II PART III Empirical Studies to Understand Developers’ Behaviour How to Reduce the Cost of such Activities?
  13. 13. Empirical Software Engineering Empirical Studies To Understand Developers Behaviour 13 PART I
  14. 14. What is Empirical Software Engineering? Isaac Newton 14
  15. 15. What is Empirical Software Engineering? Isaac Newton Empirical Studies To Understand Developers Behaviour Join a Project Continuous Delivery 15
  16. 16. What is Empirical Software Engineering? Isaac Newton Empirical Studies To Understand Developers Behaviour Join a Project Program Comprehension Tasks Maintenance tasks Continuous Delivery 16
  17. 17. What is Empirical Software Engineering? Isaac Newton Empirical Studies To Understand Developers Behaviour Development Team Join a Project Program Comprehension Tasks Maintenance tasks Team Work Testing tasks Continuous Delivery 17
  18. 18. What is Empirical Software Engineering? Isaac Newton Empirical Studies To Understand Developers Behaviour Development Team Join a Project Program Comprehension Tasks Maintenance tasks Team Work Testing tasks Continuous Delivery Data Extraction Recommenders • Versioning Systems • Mailing Lists • Issue trackers • Q&A site (e.g. StackOverflow) Empirical Studies
  19. 19. Empirical Studies To Understand Developers Behaviour During Program Comprehension and Maintenance Tasks Study A How documentation and code are browsed by developers to perform maintenance activities? Study B: How does a software ecosystem evolve? An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks. The 29th International Conference on Software Maintenance (ICSM) - 2013 The Evolution of Project Inter-Dependencies in a Software Ecosystem: the Case of Apache. NOMINATED AS BEST PAPER at the 29th IEEE International Conference on Software Maintenance (ICSM) - 2013 19
  20. 20. Study A: Context • Object: software artifacts from SMOS, a school automation system developed by professional developers of an IT Company in Salerno (Italy). • Subjects: 121 121 667 72 33 Developers An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks. The 29th International Conference on Software Maintenance (ICSM) - 2013 20 ……………. S DUJ
  21. 21. Maintenance Tasks Bug Fixing: Add a new feature: Improve existing features: 21
  22. 22. How Much Time did Participants Spend on Different Kinds of Artifacts? 72% 13%10% 3%2% 22 S DUJ
  23. 23. Navigation Patterns Followed By Developers Before Reaching Source Code 23 S = Sequence Diagram D = Class Diagram U = Use Case J = Javadoc Simple Navigation Patterns (SD)+ = Sequence Diagram before Class Diagram (US)+ = Use Case before Sequence Diagram (DS)+ = Class Diagram before Sequence Diagram U(SD)+ = Use Case before (SD)+ Complex Navigation Patterns
  24. 24. S D (SD)+ (US)+ U(SD)+ (DS)+ J U S(US)+ SU(SD)+ Other 0.0545 0.0121 0.0121 0.0242 0.006 0.0364 0.0364 0.097 0.1212 0.1636 0.1151 0.042 0 0.011 0 0.032 0.011 0.011 0.021 0.021 0.084 0.179 Junior Developers Professional Developers S= Sequence Diagram D= Class Diagram U= Use Case J= Javadoc Most Frequent Navigation Patterns 
 Before Reaching Source Code 24
  25. 25. S D (SD)+ (US)+ U(SD)+ (DS)+ J U S(US)+ SU(SD)+ Other 0.0545 0.0121 0.0121 0.0242 0.006 0.0364 0.0364 0.097 0.1212 0.1636 0.1151 0.042 0 0.011 0 0.032 0.011 0.011 0.021 0.021 0.084 0.179 Junior Developers Professional Developers S= Sequence Diagram D= Class Diagram U= Use Case J= Javadoc Most Frequent Navigation Patterns 
 Before Reaching Source Code 25
  26. 26. 26 Lessons Learnt from Study A Lesson 2 Lesson 1
  27. 27. 27 Lessons Learnt from Study A Lesson 2 Lesson 1 During the Maintenance Tasks Most of the Time Was Spent by Developers on Source Code.
  28. 28. 28 Lessons Learnt from Study A Lesson 2 Help (Junior) Developers during Program Comprehension Tasks Lesson 1 To Reduce the Maintenance Cost we Need to: During the Maintenance Tasks Most of the Time Was Spent by Developers on Source Code.
  29. 29. Effort in Program Comprehension A. Von Mayrhauser and A. Vans. Program comprehension during software maintenance and evolution. IEEE Computer, 28(8):44–55, Aug, 1995. “Developers spend more time reading than writing code” 29
  30. 30. Activities in Software Maintenance Change documentation 5% Change implementation 10% Change planning 10% Change Testing 25%Source code comprehension 50% Source: Principles of Software Engineering and Design, Zelkovits, Shaw, Gannon 1979 Activities of Software Maintenance Source: Principle of Software Engineering and Design Zelkovits, Shan, Gannon 1979 30
  31. 31. Activities and Cost of Software Maintenance Relative Maintenance COST: between 50% and 75% of global cost is spent on maintenance! Source: Principle of Software Engineering and Design Zelkovits, Shan, Gannon 1979 Activities in Software Maintenance Change documentation 5% Change implementation 10% Change planning 10% Change Testing 25%Source code comprehension 50% Source: Principles of Software Engineering and Design, Zelkovits, Shaw, Gannon 1979 Cost of MaintenanceActivities of Maintenance 31
  32. 32. 32 Lessons from Study A Lesson 2 Help (Junior) Developers during Program Comprehension Tasks Lesson 1 To Reduce the Maintenance Cost we Need to: During the Maintenance Tasks Most of the Time Was Spent by Developers on Source Code. 32
  33. 33. The Evolution of Project Inter-Dependencies in a Software Ecosystem: the Case of Apache Gabriele Bavota, Gerardo Canfora, Massimiliano Di Penta, Rocco Oliveto, Sebastiano Panichella Study B: Context How the Apache Community Upgrades Dependencies: An Evolutionary Study. Empirical Software Engineering (EMSE 2014). Evolution of Project Inter-Dependencies in a Software Ecosystem: the Case of Apache. NOMINATED AS BEST PAPER at the 29th IEEE International Conference on Software Maintenance (ICSM) - 2013 https://www.apache.org 33
  34. 34. How the Software Projects Evolve over the Time 34
  35. 35. How the Software Projects Evolve over the Time 35
  36. 36. View of Projects Dependencies in APACHE 36
  37. 37. View of Projects Dependencies in APACHE 37
  38. 38. View of Projects Dependencies in APACHE 38
  39. 39. View of Projects Dependencies in APACHE 39
  40. 40. View of Projects Dependencies in APACHE 40
  41. 41. View of Projects Dependencies in APACHE 41
  42. 42. 42 Lessons Learnt from Study B Lesson 2 Lesson 1 To Reduce the Maintenance Cost we Need: Mentors or Expert to Train Junior Developers
  43. 43. Motivation https://community.apache.org/mentoringprogramme.html https://community.apache.org/mentoringprogramme.html 43 Mentors in Software Projects
  44. 44. 44 Lessons from Study B Lesson 2 Lesson 1 To Reduce the Maintenance Cost we Need: Mentors or Expert to Train Junior Developers
  45. 45. 45 Lessons from Study B Lesson 2 Need of Tools for Collaborative Maintenance and Testing Tasks Lesson 1 To Reduce the Maintenance Cost we Need: Mentors or Expert to Train Junior Developers
  46. 46. 46 Lessons from Experiment B Lesson 2 Need of Mentors or Expert to Train Junior Developers Need of Tools for Collaborative Maintenance and Testing Tasks Lesson 1 Who is Going to Mentor Newcomers in Open Source Projects?   International Symposium on the Foundations of Software Engineering (SIGSOFT FSE 2012)
  47. 47. Suggest Appropriate Mentors to Help Newcomers in Open Source Projects Development Team Join a Project Program Comprehension Tasks Maintenance tasks Team Work Testing tasks
  48. 48. Mentoring of Project Newcomers is Highly Desirable…. Previous Work Dagenais et al. ICSE 2010 MENTOR 48
  49. 49. • Small Projects: find Mentors is a trivial problem • Large Projects: find Mentors is not a trivial problem When a Newcomer Joins a Project ......... 49
  50. 50. Motivation https://community.apache.org/mentoringprogramme.html https://community.apache.org/mentoringprogramme.html 50 Identifying Mentors in Software Projects
  51. 51. Characteristics of a Good Mentor 51 Enough ability to help other people. Enough expertise about the topic of interest for the newcomer.
  52. 52. 1) Find Past Successful Mentors 2) Suggest Mentors Having Specific Skills YODA (Young and newcOmer Developer Assistant) 52 Approach for Mentors Identification in Open Source Projects Who is Going to Mentor Newcomers in Open Source Projects?   International Symposium on the Foundations of Software Engineering (SIGSOFT FSE 2012)
  53. 53. Score Computed Aggregating the Factors in a Weighted Sum Identify Past Successful Mentors ∑= 5 1i ii fw F1 F2 > F3 F4 - 1st F5
  54. 54. When Alice joins the project Jim Alice Is the mentor of IF F1 F2 > F3 F4 - 1st F5 F5 Time F5: Commits YODA
  55. 55. Score Computed Aggregating the Factors in a Weighted Sum Identify Past Successful Mentors ∑= 5 1i ii fw F1 F2 > F3 F4 - 1st F5
  56. 56. Recommending Mentors 56 Time Project Developers
  57. 57. 57 Time Project Developers Newcomer t Recommending Mentors
  58. 58. 58 Time Project Developers Newcomer t ? Mentor with Adequate Skills Recommending Mentors
  59. 59. 59 Time Past Mentors Newcomer t Inspired to the Work on Bug Triaging by J. Anvik et al., TOSEM 2011 Recommending Mentors
  60. 60. 60 Time t Inspired to the Work on Bug Triaging by J. Anvik et al., TOSEM 2011 Recommending Mentors
  61. 61. 61 Time t Inspired to the Work on Bug Triaging by J. Anvik et al., TOSEM 2011 Past Project Developers Recommending Mentors
  62. 62. 62 Time Past Mentors Newcomer t Inspired to the Work on Bug Triaging by J. Anvik et al., TOSEM 2011 Recommending Mentors
  63. 63. 63 Time Past Mentors Newcomer t Inspired to the Work on Bug Triaging by J. Anvik et al., TOSEM 2011 Recommending Mentors
  64. 64. 64 Mentor RecommendaTons: Precision on Top 1 and Top 2 Precision 0% 25% 50% 75% 100% Apache FreeBSD PostgreSQL Python Samba 82% 77% 100% 24% 81% 94% 64% 100% 30% 85% Top 1 Top 2 Is it Possible to Recommend Mentors 
 to Project Newcomers?
  65. 65. 65 Mentor RecommendaTons: Precision on Top 1 and Top 2 Precision 0% 25% 50% 75% 100% Apache FreeBSD PostgreSQL Python Samba 76% 84% 89% 45% 72% 92%91%90% 67% 80% Top 1 Top 2 Results when Using both Mails and Issues (Oracle Derived from Historical Data)
  66. 66. 66 Mentor RecommendaTons: Precision on Top 1 and Top 2 Precision 0% 25% 50% 75% 100% Apache FreeBSD PostgreSQL Python Samba 76% 84% 89% 45% 72% 92%91%90% 67% 80% Top 1 Top 2 With YODA it Possible to Recommend Mentors It is Possible to Recommend Mentors 
 to Project Newcomers?
  67. 67. 67 YODA Tool YODA: Young and newcOmer Developer Assistant. In Proceedings of the 35th International Conference on Software Engineering (ICSE 2013)
  68. 68. Summary of PART I Development Team Join a Project Program Comprehension Tasks Maintenance tasks Team Work Testing tasks 2) Suggesting Mentors or Expert to Junior Developers Continuous Delivery 3) Need of Tools for Collaborative Software Maintenance and Testing 1) Help (Junior) Developers during Program Comprehension Task 68
  69. 69. Development Team Join a Project Program Comprehension Tasks Maintenance tasks Team Work Testing tasks 1) Help (Junior) Developers during Program Comprehension Task 2) Suggesting Mentors or Expert to Junior Developers Continuous Delivery Who is Going to Mentor Newcomers in Open Source Projects? (SIGSOFT FSE 2012) 3) Need of Tools for Collaborative Software Maintenance and Testing 69 Summary of PART I
  70. 70. Development Team Join a Project Program Comprehension Tasks Maintenance tasks Team Work Testing tasks 2) Suggesting Mentors or Expert to Junior Developers Continuous Delivery Who is Going to Mentor Newcomers in Open Source Projects? (SIGSOFT FSE 2012) 3) Need of Tools for Collaborative Software Maintenance and Testing 1) Help (Junior) Developers during Program Comprehension Task 70 Summary of PART I
  71. 71. PART 2 How to Reduce the Cost of Maintenance and Testing Activities? 71
  72. 72. Collaborate with Managers and Developers of Various IT Companies ITALYSWITZERLAND NETHERLAND JAPAN (Credit Suisse, Google, etc.) (ING Netherland, etc.) (Telecom, Flosslab, etc.) (Sony Mobile Communication, etc.) Development Team Join a Project Program Comprehension Tasks Maintenance tasks Team Work Testing tasks Continuous Delivery 72
  73. 73. Many Emerging Maintenance and Testing Problems ITALYSWITZERLAND NETHERLAND JAPAN (Credit Suisse, Google, etc.) (ING Netherland, etc.) (Telecom, Flosslab, etc.) (Sony Mobile Communication, etc.) Traceability Recovery Inconsistencies between Software Documents and Code Need to Integrate User Feedback in the Maintenance Process Need of better Tools for Software Testing 73
  74. 74. Many Emerging Maintenance and Testing Problems ITALYSWITZERLAND NETHERLAND JAPAN (Credit Suisse, Google, etc.) (ING Netherland, etc.) (Telecom, Flosslab, etc.) (Sony Mobile Communication, etc.) Inconsistencies between Software Documents and Code Need of better Tools for Software Testing Need to Integrate User Feedback in the Maintenance Process Traceability Recovery 74
  75. 75. Many Emerging Maintenance and Testing Problems ITALYSWITZERLAND NETHERLAND JAPAN (Credit Suisse, Google, etc.) (ING Netherland, etc.) (Telecom, Flosslab, etc.) (Sony Mobile Communication, etc.) Need to Integrate User Feedback in the Maintenance Process Traceability Recovery Inconsistencies between Software Documents and Code Need of better Tools for Software Testing 75
  76. 76. Summarization Approaches
  77. 77. Summarization Approaches “…have the general capability of automatically extracting or abstracting key content from one or more sources of information thus, determining the relevant information in the source being summarized and reducing its content…” 1) indicative summary: it provides a direct link to the required content relevant sources to users, so that they can read the provided information more depth. 2) informative summary it has the goal to substitute the origisource of information, by mainly assembling the relevant content, presenting it in a new, more concise and structured form. 3) critical summary (or review) it eports or selects the main opinions or statements related to a specific discussed topic, thus, it brings the most relevant feedback, both positives and negatives, about a given subject discussed in the source document. 77
  78. 78. Example of informative summary MILAN, Italy, April 18. A small airplane crashed into a government building in heart of Milan, setting the top floors on fire, Italian police reported. There were no immediate reports on casualties as rescue workers attempted to clear the area in the city's financial district. Few details of the crash were available, but news reports about it immediately set off fears that it might be a terrorist act akin to the Sept. 11 attacks in the United States. Those fears sent U.S. stocks tumbling to session lows in late morning trading. Witnesses reported hearing a loud explosion from the 30-story office building, which houses the administrative offices of the local Lombardy region and sits next to the city's central train station. Italian state television said the crash put a hole in the 25th floor of the Pirelli building. News reports said smoke poured from the opening. Police and ambulances rushed to the building in downtown Milan. No further details were immediately available.
  79. 79. MILAN, Italy, April 18. A small airplane crashed into a government building in heart of Milan, setting the top floors on fire, Italian police reported. There were no immediate reports on casualties as rescue workers attempted to clear the area in the city's financial district. Few details of the crash were available, but news reports about it immediately set off fears that it might be a terrorist act akin to the Sept. 11 attacks in the United States. Those fears sent U.S. stocks tumbling to session lows in late morning trading. Witnesses reported hearing a loud explosion from the 30-story office building, which houses the administrative offices of the local Lombardy region and sits next to the city's central train station. Italian state television said the crash put a hole in the 25th floor of the Pirelli building. News reports said smoke poured from the opening. Police and ambulances rushed to the building in downtown Milan. No further details were immediately available. How many victims? Was it a terrorist act? What was the target? What happened? Says who? When, where? Example of informative summary
  80. 80. 35 http://textcompactor.com/ Example of informative summary
  81. 81. 35 http://textcompactor.com/ Example of informative summary
  82. 82. Release Cycle Maintenance activities Coding activities Automated testing CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … … … … ? … Software documentation Testing
  83. 83. Release Cycle Maintenance activities Coding activities Automated testing CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … … …Software documentation … … Testing ?
  84. 84. Activities in Software Maintenance Change documentation 5% Change implementation 10% Change planning 10% Change Testing 25%Source code comprehension 50% Source: Principles of Software Engineering and Design, Zelkovits, Shaw, Gannon 1979 Source Code Summaries: Why? 84
  85. 85. Source Code Summaries: Why? Prevent Maintenance Cost…. 85
  86. 86. Understanding Code… Not So Happy Developers Happy Developers Absence of Comments in the Code again !! Comments in the Code again !! 86
  87. 87. Understanding Code… Not So Happy Developers Happy Developers Absence of Comments in the Code again !! Comments in the Code again !! 87
  88. 88. Source Code Summaries: How?
  89. 89. Questions when Generating Summaries of Java Classes ■ 1) What information to include in the summaries? ■ 2) How to generate and present the summaries? 89
  90. 90. Software Words Usage Model: deriving <actions>, <themes>, and <secondary arguments> from class, methods, attributes and variable identifiers E. Hill et al. Automatically capturing source code context of NL-queries for software maintenance and reuse. ICSE 2009 Summary Generator
  91. 91. When Navigating Java Classes… https://github.com/larsb/atunesplus/blob/master/aTunes/src/main/java/net/sourceforge/atunes/kernel/modules/repository/audio/AudioFile.java we look at - Name of the Class - Attributes - Methods - Dependencies between Classes Source Code Summaries: How?
  92. 92. When Navigating Java Classes… Source Code Summaries: How? ■ Generic responsibilities (domain independent) ■ Class stereotypes [Dragan et al., ICSM’10] ■ E.g., data class, entity, controller, boundary, etc.
  93. 93. When Navigating Java Classes… Source Code Summaries: How? ■ Generic responsibilities (domain independent) ■ Class stereotypes [Dragan et al., ICSM’10] ■ E.g., data class, entity, controller, boundary, etc.
  94. 94. An approach for Summarizing a Java Class (JSummarizer) [ Moreno at al. - ICPC 2013 ] 94
  95. 95. How to present and generate the summaries? Other Code Artefacts can be Summarised as well: - Packages - Classes - Methods - etc.http://www.cs.wayne.edu/~severe/jsummarizer/
  96. 96. [ Moreno at al. - ICPC 2013 ] Evaluation of the Summaries
  97. 97. Potential Useful Code Descriptions can be Found in Developers’ Discussions… source'code'descrip+ons'in'external' ar+facts.! .................................................. When call the method IndexSplitter.split(File destDir, String[] segs) from the Lucene cotrib directory(contrib/misc/src/java/org/apache/ lucene/index) it creates an index with segments descriptor file with wrong data. Namely wrong is the number representing the name of segment that would be created next in this index. .................................................. CLASS:'IndexSplitter! METHOD:'split! ■ Others researchers proposed to detect source code descriptions from sources external to the source code: ■ Mailing list and Issue tracker ■ StackOverflow Discussions [ Vassallo at al. - ICPC 2014 ][ Panichella at al. - ICPC 2012 ]
  98. 98. CODES: mining source code descriptions from developers discussions.  BEST TOOL AWARD at the 22nd International Conference on Program Comprehension (IEEE ICPC 2014) 98 CODES was able to automatically re-document 25% OF 36% OF WITH a precision >80% Mining source code descriptions from developer communications. International Conference on Program Comprehension (IEEE ICPC 2012) CODES: Approach for Mining Method Descriptions
  99. 99. Software Changes over the Time… “…as consequence the original documentation tend to be incomplete and inconsistent with the source code…” Insufficient Information Source Code Difficult to Understand APIs Documents Inconsistent withComing back to the reality... Inconsistent/ Incomplete 99
  100. 100. Source Code APIs Documents Inconsistent API Document Defects are Frequent API Document JDK-1.8 Class: InputEvent Method: getMaskForButton(int button) ————————————-—————————————- ————————————-—————————————- https://docs.oracle.com/javase/8/docs/api/java/awt/event/InputEvent.html “…and tend to be discovered and fixed after long time…” http://stackoverflow.com/questions/2967303/inconsistency-in-java-util-concurrent-future 100
  101. 101. DRONE DetectoR of dOcumentatioN dEfects Code API Document Software Artifacts AST Parsing Pre-Process and POS Tagging Defect Reports Control Flow- Based Constraint Analysis SMT Solver Dependency Parsing and Pattern Analysis Code Constraint FOL Generating Doc Constraint FOL Generating HeuristicsHeuristics 101
  102. 102. DRONE DetectoR of dOcumentatioN dEfects Code API Document Software Artifacts AST Parsing Pre-Process and POS Tagging Defect Reports Control Flow- Based Constraint Analysis SMT Solver Dependency Parsing and Pattern Analysis Code Constraint FOL Generating Doc Constraint FOL Generating HeuristicsHeuristics 102
  103. 103. DRONE DetectoR of dOcumentatioN dEfects Code API Document Software Artifacts AST Parsing Pre-Process and POS Tagging Defect Reports Control Flow- Based Constraint Analysis SMT Solver Dependency Parsing and Pattern Analysis Code Constraint FOL Generating Doc Constraint FOL Generating HeuristicsHeuristics 103
  104. 104. DRONE DetectoR of dOcumentatioN dEfects Code API Document Software Artifacts AST Parsing Pre-Process and POS Tagging Defect Reports Control Flow- Based Constraint Analysis SMT Solver Dependency Parsing and Pattern Analysis Code Constraint FOL Generating Doc Constraint FOL Generating HeuristicsHeuristics 104
  105. 105. Code API Document Software Artifacts AST Parsing Pre-Process and POS Tagging Defect Reports Control Flow- Based Constraint Analysis SMT Solver Dependency Parsing and Pattern Analysis Code Constraint FOL Generating Doc Constraint FOL Generating HeuristicsHeuristics DRONE Documentation Code With DRONE we analyzed over 1 million of LOC and more than 30,000 Javadoc documents belonging to 8 java libraries detecting around 2000 of API documentation defects. with high precision (values between 0.58 - 0.83) and an high recall (values > 0.81) results. “Analyzing APIs Documentation and Code to Detect Directive Defects”. ICSE 2017 Evaluation of the Summaries 105
  106. 106. II. Code Change Summarization
  107. 107. 107 1) Generating Commit Messages via Summarization of Source Code Changes 2) Automatic Generation of Release Notes [ Binkley at al. - ICSM 2013 ] To Improve Commits Quality To Improve Releases Note Quality Task-Driven Software Summarization
  108. 108. 108 The what: changes implemented during the incremental change The why: motivation and context behind the changes Commit Message Should Describe
  109. 109. 109 10% of the messages were removed: - they were empty - had very short strings or lacked any semantical sense Maalej and Happel - MSR 10 Commit Message Should Describe
  110. 110. 110 Generating Commit Messages via Summarization of Source Code Changes
  111. 111. Original Message This is a large modifier commit: this is a commit with many methods and combines multiple roles. This commit includes changes to internationalization, properties or configuration files (pom.xml). This change set is mainly composed of: 1. Changes to package retrofit.converter: 1.1. Add a Converter implementation for simple XML converter. It allows to: Instantiate simple XML converter with serializer; Process simple XML converter simple XML converter from body; Convert simple XML converter to body Referenced by: SimpleXMLConverterTest class Message Automatically Generated 52% prefer for small commits 64% prefer for large commits 111
  112. 112. Release Cycle Testing Maintenance Activities Coding Activities CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … … … … … Software documentation Automated testing
  113. 113. Release Cycle Testing Maintenance activities Coding activities CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … … … … … Software documentation Automated testing
  114. 114. V.S.! Integration of User Feedback in the Maintenance Process
  115. 115. 115 Maintenance of Mobile Applications “About one third of app reviews contain useful information for developers” Pagano et. al. RE2013
  116. 116. 116 “About one third of app reviews contain useful information for developers” Pagano et. al. RE2013 Manual analysis of User Reviews
  117. 117. 117 Chen et al – ICSE 2014! Panichella et al – ICSME 2015! PAST WORK
  118. 118. 118 Feature Requests Bug Reports The Problem
  119. 119. 119 SURF (Summarizer of User Reviews Feedback) Summaries of User Reviews What Would Users Change in My App? Summarizing App Reviews for Recommending Software Changes. 24th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE) - 2016. SURF: Summarizer of User Reviews Feedback. Proceedings of the 39th IEEE International Conference on Software Engineering (ICSE) - 2017
  120. 120. 120 User Review Model
  121. 121. 121 User Review Model I love this app but it crashes my whole iPad and it has to restart itself • User intention: Problem Discovery! ! • Review topics: App, Model! “…The User Reviews Model proposed by the authors is impressive in how it analyzes a review sentence by sentence and is able to characterize a sentence with multiple labels…” – one of FSE reviewers! 11
  122. 122. 122 SUMMARIZER OF USER REVIEW FEEDBACK
  123. 123. 123 SUMMARIZER OF USER REVIEW FEEDBACK
  124. 124. 124 SUMMARIZER OF USER REVIEW FEEDBACK
  125. 125. 125 SUMMARIZER OF USER REVIEW FEEDBACK
  126. 126. 126 5. Summary Generation5! http://www.ifi.uzh.ch/en/seal/people/panichella/tools/SURFTool.html SURF: Summarizer of User Reviews Feedback. Proceedings of the 39th IEEE International Conference on Software Engineering (ICSE) - 2017
  127. 127. 127 Case Study Involving 23 Developers 2622 Reviews Of 12 Apps
  128. 128. 128 Case Study Involving 23 Developers 2622 Reviews Of 12 Apps JAPAN Sony Mobile Communication
  129. 129. 129 Results V.S.! 2) SURF helps to prevent more than half of the time required for analyzing users feedback and planning software changes.
  130. 130. 130 Results V.S.! 2) SURF helps to prevent more than half of the time required for analyzing users feedback and planning software changes. 3) 92% of manually extracted feedback appears also in the automatic generated summaries.
  131. 131. 131 Results V.S.! 2) SURF helps to prevent more than half of the time required for analyzing users feedback and planning software changes. 3) 92% of manually extracted feedback appears also in the automatic generated summaries. 4) Summaries generated by SURF are reasonably correct, adequate, concise, and expressive.
  132. 132. Many Emerging Maintenance and Testing Problems ITALYSWITZERLAND NETHERLAND JAPAN (Credit Suisse, Google, etc.) (ING Netherland, etc.) (Telecom, Flosslab, etc.) (Sony Mobile Communication, etc.) Need to Integrate User Feedback in the Maintenance Process Traceability Recovery Inconsistencies Between Software Documents and Code Need of better Tools For Software Testing
  133. 133. Many Emerging Maintenance and Testing Problems ITALYSWITZERLAND NETHERLAND JAPAN (Credit Suisse, Google, etc.) (ING Netherland, etc.) (Telecom, Flosslab, etc.) (Sony Mobile Communication, etc.) Need to Integrate User Feedback in the Maintenance Process Traceability Recovery Inconsistencies Between Software Documents and Code Need of better Tools For Software Testing 133
  134. 134. Release Cycle Maintenance activities Coding activities Automated testing CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … …Software documentation… Testing
  135. 135. Release Cycle Maintenance activities Coding activities CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … …… Software documentation Automated testing Testing
  136. 136. Release Cycle Testing CD/CI A/B testing App reviews Crowdsourcing Microservices Team coordination Code Review Survey Structured e-mails Security Performance Testing … …… Software documentation Automated testing Maintenance activities Coding activities
  137. 137. Test Cases Summarization
  138. 138. @Test public void test0() throws Throwable { Option option0 = new Option("aaabbb", true, "aaabbb"); Option option1 = new Option("aaabbb", true, "aaabbb"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } @Test public void test1() throws Throwable { Option option0 = new Option("aaabbb", true, "aaabbb"); Option option1 = new Option("aaabbb", true, "aaabbb"); option0.setLongOpt("adafv"); option1.setLongOpt("adafv"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } Test Code Comprehension Generated Tests Production Code public class Options implements Serializable { private static final long serialVersionUID = 1L; /** a map of the options with the character key */ private Map shortOpts = new HashMap(); /** a map of the options with the long key */ private Map longOpts = new HashMap(); /** a map of the required options */ private List requiredOpts = new ArrayList(); Earl T. Barr, et al. “The Oracle Problem in Software Testing: A Survey”. IEEE Tran. on Software Engineering, 2015. 138
  139. 139. Are Generated Tests Easy to Understand? G. Fraser et al., Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study, TOSEM 2015. Do not lead to detection of many critical faults. 0% TestingComprehension Testing time 75% 100% Automatically Generated Test cases allow to achieve high structural coverage. 139
  140. 140. @Test public void test0() throws Throwable { Option option0 = new Option("aaa", true, "aaa"); Option option1 = new Option("aaa", true, "aaa"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } Tool Why? Target Class Class Name: Option.java Library: Apache Commons-Cli 140
  141. 141. @Test public void test0() throws Throwable { Option option0 = new Option("aaa", true, "aaa"); Option option1 = new Option("aaa", true, "aaa"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } Tool Why? Random string S.Afshan et al. "Evolving Readable String Test Inputs Using a Natural Language Model to Reduce Human Oracle Cost". ICST 2013 Target Class Class Name: Option.java Library: Apache Commons-Cli 141
  142. 142. @Test public void test0() throws Throwable { Option option0 = new Option("aaa", true, "aaa"); Option option1 = new Option("aaa", true, "aaa"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } Tool Why? Target Class Class Name: Option.java Library: Apache Commons-Cli Meaningless method name Random string 142
  143. 143. @Test public void test0() throws Throwable { Option option0 = new Option("aaa", true, "aaa"); Option option1 = new Option("aaa", true, "aaa"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } Tool Why? Target Class Class Name: Option.java Library: Apache Commons-Cli Meaningless Identifier Meaningless method name Random string 143
  144. 144. @Test public void test0() throws Throwable { Option option0 = new Option("aaa", true, "aaa"); Option option1 = new Option("aaa", true, "aaa"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } Tool Why? Target Class Class Name: Option.java Library: Apache Commons-Cli Meaningless Identifier Meaningless method name Random string @Test public void test1() throws Throwable { Option option0 = new Option("aaa", true, "aaa"); Option option1 = new Option("aaa", true, "aaa"); option0.setLongOpt("adafv"); option1.setLongOpt("adafv"); boolean boolean0 = option1.equals((Object) option0); assertEquals("arg", option1.getArgName()); assertTrue(option0.hasArg()); assertTrue(boolean0); } What are the main differences? 144
  145. 145. Combining Test Case Generation with NLP “The impact of test case Summaries on Bug Fixing Performance: And Empirical Investigation”. ICSE 2016 Generated Unit Test … with Descriptions 145
  146. 146. Covered Code Summary Generator Main Steps in TestDescriber: public class Option { public Option(String opt, String longOpt, boolean hasArg, String descr) throws IllegalArgumentException { OptionValidator.validateOption(opt); this.opt = opt; this.longOpt = longOpt; if (hasArg) { this.numberOfArgs = 1; } this.description = descr; } ... } 146
  147. 147. Summary Generator public class Option { public Option(String opt, String longOpt, boolean hasArg, String descr) throws IllegalArgumentException { OptionValidator.validateOption(opt); this.opt = opt; this.longOpt = longOpt; if (hasArg) { this.numberOfArgs = 1; } this.description = descr; } ... } Main Steps in TestDescriber: 1) Select the covered statements Covered Code 147
  148. 148. public class Option { public Option(String opt, String longOpt, boolean hasArg, String descr) throws IllegalArgumentException { OptionValidator.validateOption(opt); this opt = opt; this longOpt = longOpt; if (hasArg) {false } this description = descr; } ... } Summary Generator Main Steps in TestDescriber: 1) Select the covered statements 2) Filter out Java keywords, etc. Covered Code 148
  149. 149. public class Option { public Option(String opt, String long Opt, boolean has Arg, String descr) throws IllegalArgumentException { Option Validator.validate Option(opt); this opt = opt; this long Opt = long Opt; if (has Arg) {false ; } this description = descr; } ... } Summary Generator Covered Code Main Steps in TestDescriber: 1) Select the covered statements 2) Filter out Java keywords, etc. 3) Identifier Splitting (Camel case) 149
  150. 150. public class Option { public Option(String option, String long Option, boolean has Argument String description) throws IllegalArgumentException { Option Validator.validate Option(option); this option = option; this long Option = long Option; if (has Argument) {false } this description = description; } ... } Summary Generator Covered Code Main Steps in TestDescriber: 1) Select the covered statements 2) Filter out Java keywords, etc. 3) Identifier Splitting (Camel case) 4) Abbreviation Expansion (using external vocabularies) 150
  151. 151. Main Steps in TestDescriber: 1) Select the covered statements 2) Filter out Java keywords, etc. 3) Identifier Splitting (Camel case) 4) Abbreviation Expansion (using external vocabularies) 5) Part-of-Speech tagger Summary Generator <actions> = Verbs <themes> = Nouns/Subjects <secondary arguments> = Nouns / objectes, adjectives, etc public class Option { Option(String option, String long Option , boolean has Argument String description) throws IllegalArgumentException Option Validator.validate Option(option); this option = option ; this long Option = long Option; if (has Argument false } this description = description; } NOUN NOUN NOUNADJ NOUNNOUNVERB NOUN NOUN NOUN NOUN VERB NOUN NOUNADJ ADJ ADJ ADJ NOUN NOUN NOUN VERB ADJ NOUN CON NOUN ADJ Covered Code 151
  152. 152. Summary Generator NOUN NOUN NOUNADJ NOUNNOUNVERB NOUN NOUN NOUN NOUN VERB NOUN NOUNADJ ADJ ADJ ADJ NOUN NOUN NOUN VERB ADJ NOUN CON NOUN The test case instantiates an "Option" with: - option equal to “...” - long option equal to “...” - it has no argument - description equal to “…” An option validator validates it The test exercises the following condition: - "Option" has no argument public class Option { Option(String option, String long Option , boolean has Argument String description) throws IllegalArgumentException Option Validator.validate Option(option); this option = option ; this long Option = long Option; if (has Argument false } this description = description; } NOUN NOUN NOUNADJ NOUNNOUNVERB NOUN NOUN NOUN NOUN VERB NOUN NOUNADJ ADJ ADJ ADJ NOUN NOUN NOUN VERB ADJ NOUN CON NOUN ADJ Natural Language Sentences Parsed Code 152
  153. 153. RQ: How do test case summaries impact the number of bugs fixed by developers? 153
  154. 154. 30 Developers RQ: How do test case summaries impact the number of bugs fixed by developers? 154
  155. 155. Both the two groups had 45 minutes to fix each class 30 Developers RQ: How do test case summaries impact the number of bugs fixed by developers? 155
  156. 156. Participants WITHOUT TestDescriber summaries fixed 40% of bugs Both the two groups had 45 minutes to fix each class 30 Developers RQ: How do test case summaries impact the number of bugs fixed by developers? 156
  157. 157. 30 Developers Participants WITHOUT TestDescriber summaries fixed 40% of bugs Both the two groups had 45 minutes to fix each class Participants WITH TestDescriber summaries, fixed 60%-80% of bugs RQ: How do test case summaries impact the number of bugs fixed by developers? 157
  158. 158. Test Cases Summaries and Comprehension WITHOUT Summaries: (i) Only 15% of participants consider the test cases as “easy to understand”. (iii) 40% of participants considered the test cases as incomprehensible. WITH Summaries: (i) 46% of participants consider the test cases as “easy to understand”. (iii) Only 18% of participants considered the test cases as incomprehensible. Without With 4% 6% 14% 33% 14% 6% 32% 9% 36% 45% Medium High Very High Low Very Low Perceived test comprehensibility WITH and WITHOUT TestDescriber summaries 158
  159. 159. Empirical Studies to understand developers’ behaviour Tools for Automating Software Maintenance and Testing How to Identify such Activities? How to Reduce the Cost Activities? Summary L1: (Junior) Developers Need proper Tools for Program Comprehension tasks 159
  160. 160. Empirical Studies to understand developers’ behaviour Tools for Automating Software Maintenance and Testing How to Identify such Activities? How to Reduce the Cost Activities? L1: (Junior) Developers Need proper Tools for Program Comprehension tasks L2: Comprehensibility of Tests is Important Role on Bug fixing Tasks. 160 Summary
  161. 161. Empirical Studies to understand developers’ behaviour Tools for Automating Software Maintenance and Testing How to Identify such Activities? How to Reduce the Cost Activities? L1: (Junior) Developers Need proper Tools for Program Comprehension tasks 161 Summary L2: Comprehensibility of Tests is Important Role on Bug Fixing Tasks. L3: Need of better Tools for Software Testing
  162. 162. Empirical Studies to understand developers’ behaviour Tools for Automating Software Maintenance and Testing How to Identify such Activities? How to Reduce the Cost Activities? L3: Need of better Tools for Software Testing 162 Summary L4: Need to Integrate User Feedback in the Maintenance Process L1: (Junior) Developers Need proper Tools for Program Comprehension tasks L2: Comprehensibility of Tests is Important Role on Bug fixing Tasks.
  163. 163. TestingAutomated testing CD/CI A/B testing Crowdsourcing Microservices Team coordination Survey Structured e-mails Security Performance Testing … … … … … … Software documentation Code Review Coding Activities Maintenance Activities App reviews Summarization Techniques
  164. 164. Open Challenges Challenges C1: Automating the Process of Continuous Integration (Testing) C2: Integrating User Feedback in the Software Testing Process C3: Conceive Visualisation Tools for Software Testing C4: Conceive a Tools for Collaborative Software Maintenance and Testing C5: Deliver Tools and Prototypes for Support Developers During Code Inspection Tasks 164
  165. 165. Other Technical and Research Challenges Challenges C1: Summarization of Heterogeneous Data C3: Scalability and Integration in the other Development Contexts 165 C2: Visualization of Software Summaries
  166. 166. TestingAutomated testing CD/CI A/B testing Crowdsourcing Microservices Team coordination Survey Structured e-mails Security Performance Testing … … … … … Code Review Coding Activities Maintenance Activities App reviews Potential Application Contexts …Software documentation
  167. 167. TestingAutomated testing CD/CI A/B testing Crowdsourcing Microservices Team coordination Survey Structured e-mails Security Performance Testing … … … … … Software documentation Code Review Coding Activities Maintenance Activities App reviews Potential Application Contexts …
  168. 168. G. Grano, A. Ciurumelea, S. Panichella, F. Palomba, H. Gall SANER - 2018 Giovanni Grano
  169. 169. Summarization Techniques for Code, Change, Testing and User Feedback Sebastiano Panichella Research Associate University of Zurich Thanks for the Attention! 169

×