Downloaded 58 times
![1. Tokenization
2. Normalization
1.5 Identifier
Splitting
• Splitting algorithms based on naming conventions
• camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’
• NullHandle ==> Null | Handle
• displayBox ==> display | Box
draw, the, are,
box, r,
rectangl, g,
graphic,
color,
....
STATEOFTHEARTTOOLS
display, box
null, handl](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-64-320.jpg)
![1. Tokenization
2. Normalization
1.5 Identifier
Splitting
• Splitting algorithms based on naming conventions
• camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’
• NullHandle ==> Null | Handle
• displayBox ==> display | Box
draw, the, are,
box, r,
rectangl, g,
graphic,
color,
....
STATEOFTHEARTTOOLS
display, box
null, handl](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-65-320.jpg)
![1. Tokenization
2. Normalization
1.5 Identifier
Splitting
• Splitting algorithms based on naming conventions
• camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’
• NullHandle ==> Null | Handle
draw, the, are,
box, r,
rectangl, g,
graphic,
color,
....
STATEOFTHEARTTOOLS
display, box
null, handl](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-66-320.jpg)
![1. Tokenization
2. Normalization
1.5 Identifier
Splitting
• Splitting algorithms based on naming conventions
• camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’
• NullHandle ==> Null | Handle
• displayBox ==> display | Box
draw, the, are,
box, r,
rectangl, g,
graphic,
color,
....
STATEOFTHEARTTOOLS
null, handl
display, box](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-67-320.jpg)
![• camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’
IDENTIFIERSSPLITTING](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-68-320.jpg)
![• camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’
• drawXORRect ==> drawXOR | Rect
IDENTIFIERSSPLITTING](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-69-320.jpg)
![• camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’
• drawXORRect ==> drawXOR | Rect
• drawxorrect ==> NO SPLIT
Splitting algorithms based on naming conventions
are not robust enough
IDENTIFIERSSPLITTING](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-70-320.jpg)
![SKETCHOFTHEALGORITHM
• Every Arc is Labelled with the corresponding dictionary word
• Weights represent the “cost” of each matching
• Cost function [c(“word”)] favors longest words and domain-
related information
Model: Weighted Directed Graph
Example: drawXORRect identifier
0
11
1
9
4
“r”,C-MAX
“d”,C-MAX
5
7
8
2
“R”,C-MAX
6
“a”,C-MAX
3
“w”,C-MAX
“X”,C-MAX
“O”,C-MAX
“e”,C-MAX
10
“c”,C-MAX“t”,C-MAX
“raw”,c(“raw”)
“draw”,c(“draw”)
“or”,c(“or”)
“xor”,c(“xor”)
“rectangle”,c(“rectangle”)
“R”,C-MAX](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-81-320.jpg)
![while
block<
x y
KERNELS
FOR CODE
STRUCTURES:
AST
KERNELFEATURES
IC = Conditional-Expr
I = Less-operator
C = Loop
Ls= [x,y]
IC = Loop
I = while-loop
C = Function-Body
Ls= [x, y]
Instruction Class (IC)
i.e., LOOP, CALL,
CONDITIONAL_STATEMENT
Instruction (I)
i.e., FOR, IF, WHILE, RETURN
Context (C)
i.e., Instruction Class of
the closer statement node
Lexemes (Ls)
Lexical information gathered
(recursively) from leaves
IC = Block
I = while-body
C = Loop
Ls= [ x ]](https://image.slidesharecdn.com/ml4sedefense-130627044912-phpapp02/85/Improving-Software-Maintenance-using-Unsupervised-Machine-Learning-techniques-110-320.jpg)
Uploaded byValerio Maggio
Improving Software Maintenance using Unsupervised Machine Learning techniques
Il dottorato di Valerio Maggio si concentra sull'apprendimento automatico non supervisionato per la manutenzione del software, proponendo soluzioni per l'analisi automatica del codice sorgente e il supporto alle attività di manutenzione. La tesi affronta questioni cruciali come la rimodularizzazione del software, la normalizzazione del codice sorgente e la rilevazione dei clone, evidenziando l'utilizzo di tecniche machine learning e recupero informatico. La ricerca mira a migliorare l'efficienza e l'efficacia della manutenzione del software, che rappresenta una delle fasi più costose e complesse nel ciclo di vita del software.
More Related Content
Similar to Improving Software Maintenance using Unsupervised Machine Learning techniques
Improving Software Maintenance using Unsupervised Machine Learning techniques
- 1. Dottorato in ScienzeComputazionali e Informatiche, XXV Ciclo Ph.D. Candidate: Valerio Maggio Thesis Advisors: Dr. Sergio Di Martino Dr. Anna Corazza June 5th, 2013 UNSUPERVISED MACHINE LEARNING FOR SOFTWARE MAINTENANCE
- 2. THESIS G O AL UNSUPERVISED MACHINE LEARNING FOR SOFTWARE MAINTENANCE These solutions exploit (unsupervised) machine learning techniques to mine information from the source code Define and experimentally evaluate solutions (techniques and prototype tools) for automatic software analysis to support software maintenance activities.
- 3. There exist differenttypes of Software Maintenance. SOFTWARE MAINTENANCE “A software system must be continually adapted during its overall life cycle or it progressively becomes less satisfactory.” (cit. Lehman’s First Law of Software Evolution)
- 4. SOFTWARE MAINTENANCE “A softwaresystem must be continually adapted during its overall life cycle or it progressively becomes less satisfactory.” (cit. Lehman’s First Law of Software Evolution) Software Maintenance represents the most expensive, time consuming and challenging phase of the whole development process.
- 5. SOFTWARE MAINTENANCE “A softwaresystem must be continually adapted during its overall life cycle or it progressively becomes less satisfactory.” (cit. Lehman’s First Law of Software Evolution) Software Maintenance represents the most expensive, time consuming and challenging phase of the whole development process. Software Maintenance could account up to the 85-90% of the total software costs.
- 6. ISSUES SOFTWARE MAINTENANCE “Software Maintenance isabout change!” (cit. S. Jarzabek) Change Analysis
- 7. ISSUES SOFTWARE MAINTENANCE “Software Maintenance isabout change!” (cit. S. Jarzabek) Change Analysis Program Comprehension The documentation is usually scarce or not up to date!
- 8. ISSUES SOFTWARE MAINTENANCE The source code(usually) represents the most reliable source of information about the system “Software Maintenance is about change!” (cit. S. Jarzabek) Change Analysis Program Comprehension The documentation is usually scarce or not up to date!
- 9. ISSUES SOFTWARE MAINTENANCE The source code(usually) represents the most reliable source of information about the system “Software Maintenance is about change!” (cit. S. Jarzabek) Change Analysis Program Comprehension Reverse Engineering The documentation is usually scarce or not up to date!
- 10. REVERSE E N GI N E E R I N G • Definition of tools and techniques to support maintenance activities • Goal: Build higher-level software models in an automatic fashion gathering information from the source code or any other document • Goal: To aid the comprehension of the system
- 11. REVERSE E N GI N E E R I N G • Definition of tools and techniques to support maintenance activities • Goal: Build higher-level software models in an automatic fashion gathering information from the source code or any other document • Goal: To aid the comprehension of the system STATIC ANALYSIS DYNAMIC ANALYSIS
- 12. REVERSE E N GI N E E R I N G • Definition of tools and techniques to support maintenance activities • Goal: Build higher-level software models in an automatic fashion gathering information from the source code or any other document • Goal: To aid the comprehension of the system STATIC ANALYSIS DYNAMIC ANALYSIS
- 13. MACHINE L E AR N I N G • Provides computational effective solutions to analyze large data sets
- 14. MACHINE L E AR N I N G • Provides computational effective solutions to analyze large data sets • Provides solutions that can be tailored to different tasks/domains
- 15. MACHINE L E AR N I N G • Provides computational effective solutions to analyze large data sets • Provides solutions that can be tailored to different tasks/domains • Requires many efforts in: • the definition of the relevant information best suited for the specific task/domain
- 16. MACHINE L E AR N I N G • Provides computational effective solutions to analyze large data sets • Provides solutions that can be tailored to different tasks/domains • Requires many efforts in: • the definition of the relevant information best suited for the specific task/domain • the application of the learning algorithms to the considered data
- 17. UNSUPERVISEDLEARNING • Supervised Learning: •Learn from labelled samples • Unsupervised Learning: • Learn (directly) from the data Learn by examples
- 18. UNSUPERVISEDLEARNING • Supervised Learning: •Learn from labelled samples • Unsupervised Learning: • Learn (directly) from the data Learn by examples
- 19. UNSUPERVISEDLEARNING • Supervised Learning: •Learn from labelled samples • Unsupervised Learning: • Learn (directly) from the data Learn by examples (+) No cost of labeling samples (-) Trade-off imposed on the quality of the data
- 20. THESIS C O NT R I B U T I O N S &
- 21. THESIS C O NT R I B U T I O N S Contributions to three relevant and related open issues in Software Maintenance &
- 22. THESIS C O NT R I B U T I O N S Contributions to three relevant and related open issues in Software Maintenance 1. SOFTWARE RE-MODULARIZATION 3. CLONE DETECTION 2. SOURCE CODE NORMALIZATION &
- 23.
- 24. Software Classes UI Process Components UI Components DataAccess Components Data Helpers / Utilities Security Operational Management Communications Business Components Application Facade Buisiness Workflows Messages Interfaces Service Interfaces Re-modularization provides a way to support software maintainers by automatically grouping together (clustering) “related” software classes SOFTWARE RE-MODULARIZATION PROBL EM S T A T E M E N T
- 25. Re-modularization provides away to support software maintainers by automatically grouping together (clustering) “related” software classes SOFTWARE RE-MODULARIZATION External Systems Service Consumers Services Service Interfaces Messages Interfaces Cross Cutting Security OperationalManagement Communications Data Data Access Components Data Helpers / Utilities Presentation UI Components UI Process Components Business Application Facade Buisiness Workflows Business Components Clusters of Software Classes PROBL EM S T A T E M E N T
- 26.
- 27.
- 28. SOURCECODE LEXICALINFORMATION (IDEA): Exploitthe lexical information gathered from the source code to produce clusters of classes that are lexically related.
- 29. SOURCECODE LEXICALINFORMATION (IDEA): Exploitthe lexical information gathered from the source code to produce clusters of classes that are lexically related. State of the Art: Information Retrieval (IR) based approaches.
- 30.
- 31. Implicit assumption: The“same” words are used whenever a particular concept occurs IRINDEXING
- 32. 1. Tokenization 2. Normalization draw,the, are, null, handl, box, r, rectangl, g, graphic, box, display, box, ... Draws, the, are, NullHandle, box, r, Rectangle, g, Graphics, box, displayBox, ... Implicit assumption: The “same” words are used whenever a particular concept occurs IRINDEXING
- 33.
- 34.
- 35.
- 36. SOURCECODEZONES LEXICALINFORMATION Class Names Attribute Names MethodNames Method Names
- 37. SOURCECODEZONES LEXICALINFORMATION Class Names Attribute Names MethodNames Parameter Names Method Names Parameter Names
- 38. SOURCECODEZONES LEXICALINFORMATION Class Names Attribute Names MethodNames Parameter Names Comments Comments Method Names Parameter Names Comments
- 39. SOURCECODEZONES LEXICALINFORMATION Source Code Class Names AttributeNames Method Names Parameter Names Comments Comments Method Names Parameter Names Source Code Comments
- 40.
- 41. ZONEINDEXING RQ1: Do termsin different Zones provide different contributions?
- 42. 1. Tokenization 2. Normalization ZONEINDEXING Draws,the, are, NullHandle, box, r, draw, g, Graphics, color, displayBox, ... draw, the, are, null, handl, box, r, draw, g, graphic, color, display, box, ... RQ1: Do terms in different Zones provide different contributions?
- 43.
- 44. SOURCECODELEXICON JFreeChart • Goodlexicon in every Zone!
- 45. SOURCECODELEXICON JEdit • Verygood in Comments • Very poor in Method and Parameter names
- 46. SOURCECODELEXICON JUnit • Goodin Class and Attribute names • No Comments at all
- 47. SOURCECODELEXICON Xerces • Poorin Method, Parameter names and Comments • Good in Method Code and Variable names
- 48. SOURCECODELEXICON JFreeChart JEdit JUnitXerces RQ2: How to automatically weight the different contributions ?
- 49. CONTRI BUTION SOFTWARE RE- MODULARIZATION • Wepropose a source code Zone Indexing Corazza A., Di Martino, S., Maggio, V., Scanniello, G. Investigating the use of lexical information for software system clustering (2011) Proceedings of the European Conference on Software Maintenance and Reengineering, CSMR, art. no. 5741257, pp. 35-44. ISSN: 15345351 ISBN: 978-076954343-7 Corazza, A., Di Martino, S., Maggio, V., Scanniello, G. Combining machine learning and information retrieval techniques for software clustering (2012) Communications in Computer and Information Science, 255 CCIS, pp. 42-60. ISSN: 18650929 ISBN: 978-364228032-0
- 50. CONTRI BUTION SOFTWARE RE- MODULARIZATION • Wepropose a source code Zone Indexing • An approach to automatically weight the contribution of different terms in Zones • Maximum-Likelihood Estimation (MLE) Approach Corazza A., Di Martino, S., Maggio, V., Scanniello, G. Investigating the use of lexical information for software system clustering (2011) Proceedings of the European Conference on Software Maintenance and Reengineering, CSMR, art. no. 5741257, pp. 35-44. ISSN: 15345351 ISBN: 978-076954343-7 Corazza, A., Di Martino, S., Maggio, V., Scanniello, G. Combining machine learning and information retrieval techniques for software clustering (2012) Communications in Computer and Information Science, 255 CCIS, pp. 42-60. ISSN: 18650929 ISBN: 978-364228032-0
- 51. CONTRI BUTION SOFTWARE RE- MODULARIZATION • Wepropose a source code Zone Indexing • An approach to automatically weight the contribution of different terms in Zones • Maximum-Likelihood Estimation (MLE) Approach • A variant of the K-medoid clustering algorithm • Tailored to the software clustering task Corazza A., Di Martino, S., Maggio, V., Scanniello, G. Investigating the use of lexical information for software system clustering (2011) Proceedings of the European Conference on Software Maintenance and Reengineering, CSMR, art. no. 5741257, pp. 35-44. ISSN: 15345351 ISBN: 978-076954343-7 Corazza, A., Di Martino, S., Maggio, V., Scanniello, G. Combining machine learning and information retrieval techniques for software clustering (2012) Communications in Computer and Information Science, 255 CCIS, pp. 42-60. ISSN: 18650929 ISBN: 978-364228032-0
- 52.
- 53.
- 54.
- 55.
- 56.
- 57. AUTHORITATIVENESS RE SULTS Comparison of ClusteringResults with an Authoritative Target Partition Ant Lucene Tomcat Azureus Hibernate ITextdotNet jEdit JFreeChart JFTP JHotDraw JRefactory JUnit LiferayPortal Pmd Synapse TigerEnvelopes Velocity Xalan Xerces 0 0.2 0.4 0.6 0.8 1.0 19 target Open Source Java Systems FLAT INDEXING (NO ZONES) = STATE OF THE ART FLAT INDEXING (NO ZONES) ZONE INDEXING ZONE INDEXING + MLE WEIGHTS
- 58. AUTHORITATIVENESS RE SULTS Comparison of ClusteringResults with an Authoritative Target Partition Ant Lucene Tomcat Azureus Hibernate ITextdotNet jEdit JFreeChart JFTP JHotDraw JRefactory JUnit LiferayPortal Pmd Synapse TigerEnvelopes Velocity Xalan Xerces 0 0.2 0.4 0.6 0.8 1.0 19 target Open Source Java Systems FLAT INDEXING (NO ZONES) = STATE OF THE ART FLAT INDEXING (NO ZONES) ZONE INDEXING ZONE INDEXING + MLE WEIGHTS
- 59. AUTHORITATIVENESS RE SULTS Comparison of ClusteringResults with an Authoritative Target Partition Ant Lucene Tomcat Azureus Hibernate ITextdotNet jEdit JFreeChart JFTP JHotDraw JRefactory JUnit LiferayPortal Pmd Synapse TigerEnvelopes Velocity Xalan Xerces 0 0.2 0.4 0.6 0.8 1.0 19 target Open Source Java Systems FLAT INDEXING (NO ZONES) = STATE OF THE ART FLAT INDEXING (NO ZONES) ZONE INDEXING ZONE INDEXING + MLE WEIGHTS
- 60. AUTHORITATIVENESS RE SULTS Comparison of ClusteringResults with an Authoritative Target Partition 19 target Open Source Java Systems FLAT INDEXING (NO ZONES) = STATE OF THE ART 0 0.2 0.4 0.6 0.8 1.0 Ant Lucene Tomcat Azureus Hibernate ITextdotNet jEdit JFreeChart JFTP JHotDraw JRefactory JUnit LiferayPortal Pmd Synapse TigerEnvelopes Velocity Xalan Xerces FLAT INDEXING (NO ZONES) ZONE INDEXING ZONE INDEXING + MLE WEIGHTS
- 61.
- 62. SOURCECODE LEXICALINFORMATION Implicit assumption:The “same” words are used whenever a particular concept occurs
- 63. SOURCECODE LEXICALINFORMATION Implicit assumption:The “same” words are used whenever a particular concept occurs
- 64. 1. Tokenization 2. Normalization 1.5Identifier Splitting • Splitting algorithms based on naming conventions • camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’ • NullHandle ==> Null | Handle • displayBox ==> display | Box draw, the, are, box, r, rectangl, g, graphic, color, .... STATEOFTHEARTTOOLS display, box null, handl
- 65. 1. Tokenization 2. Normalization 1.5Identifier Splitting • Splitting algorithms based on naming conventions • camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’ • NullHandle ==> Null | Handle • displayBox ==> display | Box draw, the, are, box, r, rectangl, g, graphic, color, .... STATEOFTHEARTTOOLS display, box null, handl
- 66. 1. Tokenization 2. Normalization 1.5Identifier Splitting • Splitting algorithms based on naming conventions • camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’ • NullHandle ==> Null | Handle draw, the, are, box, r, rectangl, g, graphic, color, .... STATEOFTHEARTTOOLS display, box null, handl
- 67. 1. Tokenization 2. Normalization 1.5Identifier Splitting • Splitting algorithms based on naming conventions • camelCase Splitter: r’(?<=!^)([A-Z][a-z]+)’ • NullHandle ==> Null | Handle • displayBox ==> display | Box draw, the, are, box, r, rectangl, g, graphic, color, .... STATEOFTHEARTTOOLS null, handl display, box
- 68. • camelCase Splitter:r’(?<=!^)([A-Z][a-z]+)’ IDENTIFIERSSPLITTING
- 69. • camelCase Splitter:r’(?<=!^)([A-Z][a-z]+)’ • drawXORRect ==> drawXOR | Rect IDENTIFIERSSPLITTING
- 70. • camelCase Splitter:r’(?<=!^)([A-Z][a-z]+)’ • drawXORRect ==> drawXOR | Rect • drawxorrect ==> NO SPLIT Splitting algorithms based on naming conventions are not robust enough IDENTIFIERSSPLITTING
- 71. Splitting algorithms basedon naming conventions are not robust enough ABBREVIATIONSEXPANSION
- 72. Splitting algorithms basedon naming conventions are not robust enough • Heavy use of Abbreviations in the source code • r as for Rectangle • rect as for Rectangle ABBREVIATIONSEXPANSION
- 73. Splitting algorithms basedon naming conventions are not robust enough • Heavy use of Abbreviations in the source code • r as for Rectangle • rect as for Rectangle ABBREVIATIONSEXPANSION
- 74. 1. Tokenization CODENORMALIZATION 2. Normalization 1.5Code Normalization • AMAP (Hill and Pollock, 2008) • SAMURAI (Enslen, et.al , 2011) • TIDIER (Guerrouj, et.al , 2011) • GenTest+Normalize (Lawrie and Binkley, 2011) • LINSEN (Corazza, Di Martino and Maggio, 2012) draw, the, are, null, handl, box, rectangl, graphic, color, display, box, rectangl, draw, xor, rectangl
- 75. 1. Tokenization CODENORMALIZATION 2. Normalization 1.5Code Normalization • AMAP (Hill and Pollock, 2008) • SAMURAI (Enslen, et.al , 2011) • TIDIER (Guerrouj, et.al , 2011) • GenTest+Normalize (Lawrie and Binkley, 2011) • LINSEN (Corazza, Di Martino and Maggio, 2012) draw, the, are, null, handl, box, rectangl, graphic, color, display, box, rectangl, draw,xor, rectangl
- 76. CONTRI BUTION SOURCE CODE NORMALIZATION • LINSEN:novel technique for code normalization • Able to both Split Identifiers and Expand possible occurring abbreviations Corazza, A., Di Martino, S., Maggio, V. LINSEN: An efficient approach to split identifiers and expand abbreviations (2012) IEEE International Conference on Software Maintenance, ICSM, art. no. 6405277, pp. 233-242. ISBN: 978-146732312-3
- 77. CONTRI BUTION SOURCE CODE NORMALIZATION • LINSEN:novel technique for code normalization • Able to both Split Identifiers and Expand possible occurring abbreviations • Based on an efficient String Matching technique: Baeza-Yates&Perlberg Algorithm (BYP) • Exploit different Sources of Information to find the matching words Corazza, A., Di Martino, S., Maggio, V. LINSEN: An efficient approach to split identifiers and expand abbreviations (2012) IEEE International Conference on Software Maintenance, ICSM, art. no. 6405277, pp. 233-242. ISBN: 978-146732312-3
- 78. SKETCHOFTHEALGORITHM • NODES correspondto characters of the current identifier Model: Weighted Directed Graph Example: drawXORRect identifier 0 11 1 9 4 5 7 8 2 63 10
- 79. SKETCHOFTHEALGORITHM • ARCS correspondsto matchings between identifier substrings and dictionary words • NODES correspond to characters of the current identifier Model: Weighted Directed Graph Example: drawXORRect identifier 0 11 1 9 4 5 7 8 2 63 10
- 80. SKETCHOFTHEALGORITHM • ARCS correspondsto matchings between identifier substrings and dictionary words • Padding Arcs to ensure the Graph always connected • NODES correspond to characters of the current identifier Model: Weighted Directed Graph Example: drawXORRect identifier 0 11 1 9 4 “r”,C-MAX “d”,C-MAX 5 7 8 2 “R”,C-MAX 6 “a”,C-MAX 3 “w”,C-MAX “X”,C-MAX “O”,C-MAX “e”,C-MAX 10 “c”,C-MAX“t”,C-MAX “raw”,c(“raw”) “draw”,c(“draw”) “or”,c(“or”) “xor”,c(“xor”) “rectangle”,c(“rectangle”) “R”,C-MAX
- 81. SKETCHOFTHEALGORITHM • Every Arcis Labelled with the corresponding dictionary word • Weights represent the “cost” of each matching • Cost function [c(“word”)] favors longest words and domain- related information Model: Weighted Directed Graph Example: drawXORRect identifier 0 11 1 9 4 “r”,C-MAX “d”,C-MAX 5 7 8 2 “R”,C-MAX 6 “a”,C-MAX 3 “w”,C-MAX “X”,C-MAX “O”,C-MAX “e”,C-MAX 10 “c”,C-MAX“t”,C-MAX “raw”,c(“raw”) “draw”,c(“draw”) “or”,c(“or”) “xor”,c(“xor”) “rectangle”,c(“rectangle”) “R”,C-MAX
- 82. SKETCHOFTHEALGORITHM • The finalMapping Solution corresponds to the sequence of labels in the path with the minimum cost (Djikstra Algorithm) Model: Weighted Directed Graph Example: drawXORRect identifier 0 11 1 9 4 “r”,C-MAX “d”,C-MAX 5 7 8 2 “R”,C-MAX 6 “a”,C-MAX 3 “w”,C-MAX “X”,C-MAX “O”,C-MAX “e”,C-MAX 10 “c”,C-MAX“t”,C-MAX “raw”,c(“raw”) “draw”,c(“draw”) “or”,c(“or”) “xor”,c(“xor”) “rectangle”,c(“rectangle”) “R”,C-MAX
- 83. SPLITTING Accuracy Rates forthe comparison with DTW (Madani et. al 2010) 0 0.25 0.5 0.75 1 JhotDraw 5.1 Lynx 2.8.5 DTW LINSEN DTW LINSEN DTW LINSEN RE SULTS # 1 Accuracy Rates for the comparison with GenTest (Lawrie and Binkley, 2011) 0 0.25 0.5 0.75 1 which 2.20 a2ps 4.14 GenTest LINSEN DTW O(n3)
- 84. Accuracy Rates forthe comparison with AMAP (Hill and Pollock, 2008) 0 0.25 0.5 0.75 1 CW DL OO AC PR SL TOTAL (Avg) AMAP LINSEN EXPANSION RE SULTS # 2 CW: Combination Words DL: Dropped Letters OO: Others AC: Acronyms PR: Prefix SL: Single Letters
- 85.
- 86. PROBL EM S T AT E M E N T CLONE DETECTION
- 87. PROBL EM S T AT E M E N T CLONE DETECTION Clones Textual Similarity
- 88. PROBL EM S T AT E M E N T CLONE DETECTION Clones Functional Similarity
- 89. PROBL EM S T AT E M E N T CLONE DETECTION Clones affect the reliability of the system! Sneaky Bug!
- 90.
- 91. Duplix Scorpio PMD CCFinder Dup CPD Duplix Shinobi Clone Detective Gemini iClones KClone ConQAT Deckard Clone Digger JCCD CloneDr SimScan CLICS NiCAD Simian Duploc Dude SDD TextBased Tools: Text is compared line by line STATEOFTHEARTTOOLS
- 92. Duplix Scorpio PMD CCFinder Dup CPD Duplix Shinobi Clone Detective Gemini iClones KClone ConQAT Deckard Clone Digger JCCD CloneDr SimScan CLICS NiCAD Simian Duploc Dude SDD TokenBased Tools: Token sequences are compared to sequences STATEOFTHEARTTOOLS
- 93. Duplix Scorpio PMD CCFinder Dup CPD Duplix Shinobi Clone Detective Gemini iClones KClone ConQAT Deckard Clone Digger JCCD CloneDr SimScan CLICS NiCAD Simian Duploc Dude SDD SyntaxBased Tools: Syntax subtrees are compared to each other STATEOFTHEARTTOOLS
- 94. Duplix Scorpio PMD CCFinder Dup CPD Duplix Shinobi Clone Detective Gemini iClones KClone ConQAT Deckard Clone Digger JCCD CloneDr SimScan CLICS NiCAD Simian Duploc Dude SDD GraphBased Tools: (sub) graphs are compared to each other STATEOFTHEARTTOOLS
- 95. CONTRI BUTION CLONE DETECTION • Combiningdifferent sources of information to improve the effectiveness of the detection process Corazza, A., Di Martino, S., Maggio, V., Scanniello, G. A Tree Kernel based approach for clone detection (2010) IEEE International Conference on Software Maintenance, ICSM, art. no. 5609715. ISBN: 978-142448629-8 Corazza, A., Di Martino, S., Maggio, V., Moschitti, A., Passerini, A., Scanniello, G., Silvestri F. Using Machine Learning and Information Retrieval Techniques to Improve Software Maintainability (2013) Communications in Computer and Information Science, In Press
- 96. CONTRI BUTION CLONE DETECTION • Combiningdifferent sources of information to improve the effectiveness of the detection process • Structural and Lexical Information Corazza, A., Di Martino, S., Maggio, V., Scanniello, G. A Tree Kernel based approach for clone detection (2010) IEEE International Conference on Software Maintenance, ICSM, art. no. 5609715. ISBN: 978-142448629-8 Corazza, A., Di Martino, S., Maggio, V., Moschitti, A., Passerini, A., Scanniello, G., Silvestri F. Using Machine Learning and Information Retrieval Techniques to Improve Software Maintainability (2013) Communications in Computer and Information Science, In Press
- 97. CONTRI BUTION CLONE DETECTION • Combiningdifferent sources of information to improve the effectiveness of the detection process • Structural and Lexical Information • Application of Kernel Methods to Source Code Structures Corazza, A., Di Martino, S., Maggio, V., Scanniello, G. A Tree Kernel based approach for clone detection (2010) IEEE International Conference on Software Maintenance, ICSM, art. no. 5609715. ISBN: 978-142448629-8 Corazza, A., Di Martino, S., Maggio, V., Moschitti, A., Passerini, A., Scanniello, G., Silvestri F. Using Machine Learning and Information Retrieval Techniques to Improve Software Maintainability (2013) Communications in Computer and Information Science, In Press
- 98. CONTRI BUTION CLONE DETECTION • Combiningdifferent sources of information to improve the effectiveness of the detection process • Structural and Lexical Information • Application of Kernel Methods to Source Code Structures • Typically applied in other field such as NLP or Bioinformatics Corazza, A., Di Martino, S., Maggio, V., Scanniello, G. A Tree Kernel based approach for clone detection (2010) IEEE International Conference on Software Maintenance, ICSM, art. no. 5609715. ISBN: 978-142448629-8 Corazza, A., Di Martino, S., Maggio, V., Moschitti, A., Passerini, A., Scanniello, G., Silvestri F. Using Machine Learning and Information Retrieval Techniques to Improve Software Maintainability (2013) Communications in Computer and Information Science, In Press
- 99. CODE STRUCTURES KERNELSFORSTRUCTURES Computation of thedot product between (Graph) Structures K( ),
- 100. CODE STRUCTURES KERNELSFORSTRUCTURES Abstract Syntax Tree(AST) Tree structure representing the syntactic structure of the different instructions of a program (function) Program Dependencies Graph (PDG) (Directed) Graph structure representing the relationship among the different statement of a program Computation of the dot product between (Graph) Structures K( ),
- 101. CODE STRUCTURES KERNELSFORSTRUCTURES Abstract Syntax Tree(AST) Tree structure representing the syntactic structure of the different instructions of a program (function) Program Dependencies Graph (PDG) (Directed) Graph structure representing the relationship among the different statement of a program Computation of the dot product between (Graph) Structures K( ),
- 102.
- 103. < x y == x + x 1 y - y 1 while block while block block if > b a = = a + a 1 b - b 1 > b 0 = c 3 CODE AST KERNELFORCLONES
- 104. < x y == x + x 1 y - y 1 while block while block block if > b a = = a + a 1 b - b 1 > b 0 = c 3 CODE AST AST KERNEL KERNELFORCLONES < block while = = block = y - = x + + x 1 - y 1 < x y > b 0 = c 3 if block > b a - b 1 < block while + a 1 = b - = a +
- 105.
- 106. while block< x y KERNELS FOR CODE STRUCTURES: AST KERNELFEATURESInstruction Class (IC) i.e., LOOP, CALL, CONDITIONAL_STATEMENT
- 107. while block< x y KERNELS FOR CODE STRUCTURES: AST KERNELFEATURESInstruction Class (IC) i.e., LOOP, CALL, CONDITIONAL_STATEMENT Instruction (I) i.e., FOR, IF, WHILE, RETURN
- 108. while block< x y KERNELS FOR CODE STRUCTURES: AST KERNELFEATURESInstruction Class (IC) i.e., LOOP, CALL, CONDITIONAL_STATEMENT Instruction (I) i.e., FOR, IF, WHILE, RETURN Context (C) i.e., Instruction Class of the closer statement node
- 109. while block< x y KERNELS FOR CODE STRUCTURES: AST KERNELFEATURESInstruction Class (IC) i.e., LOOP, CALL, CONDITIONAL_STATEMENT Instruction (I) i.e., FOR, IF, WHILE, RETURN Context (C) i.e., Instruction Class of the closer statement node Lexemes (Ls) Lexical information gathered (recursively) from leaves
- 110. while block< x y KERNELS FOR CODE STRUCTURES: AST KERNELFEATURES IC= Conditional-Expr I = Less-operator C = Loop Ls= [x,y] IC = Loop I = while-loop C = Function-Body Ls= [x, y] Instruction Class (IC) i.e., LOOP, CALL, CONDITIONAL_STATEMENT Instruction (I) i.e., FOR, IF, WHILE, RETURN Context (C) i.e., Instruction Class of the closer statement node Lexemes (Ls) Lexical information gathered (recursively) from leaves IC = Block I = while-body C = Loop Ls= [ x ]
- 111. CLONE DETECTION • Comparisonwith another (pure) AST-based clone detector • Comparison on a system with randomly seeded clones 0 0.25 0.5 0.75 1 Precision Recall F-measure CloneDigger Tree Kernel Tool RE SULTS Results refer to clones where code fragments have been modified by adding/ removing or changing code statements
- 112.
- 113.
- 114.
- 115.
- 116.
- 117. CONCLUSIONS FUTURE RESEARCH DIRECTIONS •Re-modularization: • Integrate code normalization algorithm (LINSEN) to lexical analysis • Code Normalization: • Investigate the contributions of different sources of information used for disambiguations • Clone Detection: • Combine Static (and Kernel methods) with Dynamic Analysis
- 118. THANK YOU June 5th,2013 - Ph.D. Defense Valerio Maggio Ph.D. Student, University of Naples “Federico II” valerio.maggio@unina.it