Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. A Heuristic-based Approach to Identify Concepts in Execution Traces Fatemeh Asadi* Massimiliano Di Penta** Giuliano Antoniol* Yann-Gaël Guéhéneuc** * Ecole Polytechnique de Montréal, Canada ** Dept. Of Engineering -– Univ. of Sannio, Italy CSMR 2010 Madrid (Spain) 1
  2. 2. Motivations• Software systems lack adequate documentation• Developers try to understand systems through – Static analyses, visualizations built upon static data – Dynamic analyses, requiring the execution of the system• (Dynamic) concept identification – Identify sets of method calls in execution traces responsible for the implementation of domain concepts or user-observable features – Existing approaches based on static analysis [Anquetil and Lethbridge (1998)], dynamic analysis [Wilde and Scully (1995) Tonella and Ceccato (2004)], IR techniques [Poshyvanyk et al. (2007)], or hybrid ones [Eaddy et al. (2008)] CSMR 2010 - Madrid (Spain) 2
  3. 3. Proposed approach A novel approach that analyzes execution traces and groups together method calls that: (i) sequentially invoked together/in sequence (ii) cohesive and decoupled from a conceptual point of view Assumptions Let us consider a feature is being executed in a scenario – e.g., “Open a Web page from a browser” or “Save an image in a paint application” The set of methods related to the feature is likely to be: – (i) conceptually cohesive – (ii) decoupled from those of other features – (iii) sequentially invoked CSMR 2010 - Madrid (Spain) 3
  4. 4. Proposed approach Step I – System instrumentation Step II – Execution trace collection Step III – Trace pruning and compression Step IV – Textual analysis of methods’ source code Step V – Search-based concept identification CSMR 2010 - Madrid (Spain) 4
  5. 5. Step I and Step II – Getting Traces Step I - System instrumentation System instrumented using the MoDeC instrumentor – MoDeC tool to extract and model sequence diagrams for Java systems Java bytecode instrumentation tool – Inserts appropriate and dedicated method invocations in the system to method/constructor entry/exit, points – Allows for trace tagging Step II - Execution trace collection We exercise a system following operation sequences taken from user manuals or use case descriptions CSMR 2010 - Madrid (Spain) 5
  6. 6. Step III – Trace Pruning and Compression Removing methods not very useful for feature identification Methods occurring in many scenarios – Are often utility methods – We use the same idea of tf-idf in Information Retrieval Too frequent methods – Could be for example related to crosscutting concerns – We remove methods having a frequency Q3 + 2 × IQR (75% percentile + 2 × the interquartile range) Trace compression Aim: collapse repetitions in execution traces Purpose: reduce the search space for Step V Examples: – m1(); m1(); m1(); m1(); m1; m2(); – m1(); m2(); m1(); m2(); Performed using the Run Length Encoding (RLE) Applied for sub-sequences having an arbitrary length CSMR 2010 - Madrid (Spain) 6
  7. 7. Step IV Conceptual cohesion and coupling determined according to [Marcus et al., 2008] and [Poshyvanyk et al., 2006] Index identifiers, comments contained in methods Extraction of identifiers and comment words Camel-case splitting of composed identifiers Stop word removal (English + Java keywords) Stemming using the Porter stemmer Indexing using tf-idf Reduce the term-document space into a (smaller) concept- document space using Latent Semantic Indexing (LSI) – Helps to cope with synonymy and homonymy – Concept space=50 CSMR 2010 - Madrid (Spain) 7
  8. 8. Step V We use a search-based optimization technique based on Genetic Algorithms (GA) to split traces into segments Representation: a bit-vector where 1 indicates the end of a segment Trace splitting m1 m2 m1 m3 m4 m1 m4 m6 m1 Representation 0 1 0 0 1 0 0 0 1 Mutation: randomly flips a bit (i.e., splits or merge segments)0 1 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 Crossover: two-points0 1 0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 Selection: Roulette Wheel CSMR 2010 - Madrid (Spain) 8
  9. 9. Step V – Quality of the Solution Fitness Function: Segment Cohesion is the average (textual) similarity between any pair of methods in a segment Segment Coupling is the average (textual) similarity between a segment and all other segments in the trace Other GA parameters 200 individuals 2,000 generations for JHotDraw and 3,000 for ArgoUML 5% mutation probability, 70% crossover probability Distributed GA implementation (across 4 servers) CSMR 2010 - Madrid (Spain) 9
  10. 10. Empirical Study • Goal: analyze the novel concept location approach based • Purpose: of evaluating its capability of identifying meaningful concepts • Quality focus: accuracy and completeness of the identified concepts • Context: an implementation of our approach and execution traces extracted from two open source systems, JHotDraw and ArgoUML CSMR 2010 - Madrid (Spain) 10
  11. 11. Research Questions RQ1: How stable is the GA, through multiple runs, when identifying concepts into execution traces? RQ2: To what extent the identified concepts match the ones in the oracle? RQ3: How accurate is the identification of concepts in execution traces? CSMR 2010 - Madrid (Spain) 11
  12. 12. RQ1: GA stability We compute the overlap between segmentations obtained in multiple runs using the Jaccard overlap Score Two segments overlaps when they contain calls in the same position of the trace Because a segment of trace T1 overlaps with more segments of T2, the highest similarity is chosen Run 1 m1 m2 m1 m3 m4 m1 m4 m6 m1 Run 2 m1 m2 m1 m3 m4 m1 m4 m6 m1 2/3 2/4 3/4 CSMR 2010 - Madrid (Spain) 12
  13. 13. RQ1: Results Average overlap between 72% and 84% Slightly higher convergence for ArgoUML Ability of the algorithm to converge, despite the relatively large search space CSMR 2010 - Madrid (Spain) 13
  14. 14. RQ2: Matching with the Oracle We manually tag start-end of features while executing the system Using the MoDeC instrumentation tool While executing the instrumented system, the user triggers the introduction of <Start> and <Stop> tags in the trace Matching between identified traces and oracle computed as in RQ1 Run 1 m1 m2 m1 m3 m4 m1 m4 m6 m1 Oracle m1 m2 m1 m3 m4 m1 m4 m6 m1 2/3 2/4 3/4 CSMR 2010 - Madrid (Spain) 14
  15. 15. RQ2: Results High overlap for some features e.g., Draw rectangle or Draw circle Lower for features obtained adapting other ones e.g., Add text obtained adapting Draw rectangle In other cases, low overlap is due to large segments split into more smaller and cohesive ones CSMR 2010 - Madrid (Spain) 15
  16. 16. RQ3: Accuracy in trace identification Computed similarly to RQ2, however we use Precision instead of Jaccard overlap Score Run 1 m1 m2 m1 m3 m4 m1 m4 m6 m1 Oracle m1 m2 m1 m3 m4 m1 m4 m6 m1 2/2 2/3 3/4 CSMR 2010 - Madrid (Spain) 16
  17. 17. RQ3: Results Precision often very high In most cases above 85% and often equal to 100% Low precision (mean 32%) for Add text Relatively low (mean 69%) for Draw rectangle These two features are difficult to be distinguished CSMR 2010 - Madrid (Spain) 17
  18. 18. Inspection of the obtained segments Add class (ArgoUML) The approach split this long feature of 199 methods sequence into 5 segments related to sub-features (creation of objects, adding the project class, handling namespace, setting object properties, handling persistence of the diagram) Create note (ArgoUML) Only the first part (50 methods) of the trace composed of 88 calls was identified Problems related to multi-threading Problems related to collapsing (during compression) loops containing variants Cut rectangle (JHotDraw) Only the last 39 out of 172 calls were included in the segment Methods related to adding to the clipboard and showing the rectangle as “cut” First methods related to GUI events and split in many small segments Spawn window (JHotDraw) 72 out of 197 methods included The remaining ones were related to setting up menu command properties CSMR 2010 - Madrid (Spain) 18
  19. 19. Threats to Validity Construct validity (relation btw. theory and observation) Multi-threading can change the ordering of calls in multiple executions of the same scenario A better assessment of the actual content of the obtained segments is needed Internal validity (presence of confounding factors) Trace tagging may be imprecise, again due to multi-threading Noise due to utility methods GA intrinsic randomness External validity (generalization of findings) We analyzed two different systems, multiple traces As usual, further empirical evaluation is needed CSMR 2010 - Madrid (Spain) 19
  20. 20. Conclusions We proposed a search-based approach to automatically locate concepts in execution traces By splitting traces into conceptually cohesive and decoupled segments Empirical study on traces from JHotDraw and ArgoUML shows that The approach is stable Identified segments highly precise Finer-splitting wrt. high-level features Limitations due to: multi-threading, GUI events, feature adaptation.. Work-in-progress: Improve performance Use enhanced compression techniques Automatically label identified concepts Perform an extensive empirical validation CSMR 2010 - Madrid (Spain) 20
  21. 21. Thank You! Questions? CSMR 2010 - Madrid (Spain) 21