Experiments on Design Pattern Discovery Jing Dong and Yajing Zhao Department of Computer Science The University of Texas at Dallas {jdong, yxz045100}@utdallas.edu

Outline Introduction Related Work Our Approach Experiment Data Result and Discussion Precision and Recall Benchmark Conclusion and Future Work

Introduction

Related Work

Our Approach

Experiment Data

Result and Discussion

Precision and Recall

Benchmark

Conclusion and Future Work

Introduction Difficult to understand large computer-based systems No original architecture and design Patterns help on understanding systems Patterns embed future change of systems Need of design pattern discovery since patterns are lost in source code

Difficult to understand large computer-based systems

No original architecture and design

Patterns help on understanding systems

Patterns embed future change of systems

Need of design pattern discovery since patterns are lost in source code

Introduction (Cont’d) DP-Miner Experiment data: System source code JUnit JEdit JHotDraw Java.AWT Such experiments lead to benchmarks

DP-Miner

Experiment data: System source code

JUnit

JEdit

JHotDraw

Java.AWT

Such experiments lead to benchmarks

Related Work X Flyweight X Facade X X Chain of Responsibility X X Builder X X X Abstract Factory X X X X X X X X X X Balanyi 2003 X X Command X X X Proxy X X X X X X Composite X X X X X Decorator X X X X Factory Method X X X X Observer X X Prototype X X X X Singleton X X X X Strategy X X X Template Method X X X Visitor X X X X Bridge X X X X Adapter Gueheneuc 2006 Shi 2006 Blewitt 2001 Niere 2002 Heuzeroth 2003 Antoniol 1998 Tsantalis 2006 Authors Tools

Our Approach – DP-Miner Matrix and Weight Product of Prime Numbers Structural, Behavioral, and Semantic Analysis

Matrix and Weight

Product of Prime Numbers

Structural, Behavioral, and Semantic Analysis

Experiment Data JUnit: A regression test framework that helps developers to implement unit tests in Java. JEdit: A mature and easy-to-use text editor. JHotDraw: A two dimensional graphics frameworks for technical and structured drawing editors written in Java. Java.AWT: A library for developing graphical user interfaces for Java programs.

JUnit: A regression test framework that helps developers to implement unit tests in Java.

JEdit: A mature and easy-to-use text editor.

JHotDraw: A two dimensional graphics frameworks for technical and structured drawing editors written in Java.

Java.AWT: A library for developing graphical user interfaces for Java programs.

Why Those Four Systems as Data? At the time they were developed, the idea of design patterns is already mature and widely applied. Other works on design pattern discovery use one or more of these systems to evaluate their approaches, which allows us to compare and evaluate our experiment results.

At the time they were developed, the idea of design patterns is already mature and widely applied.

Other works on design pattern discovery use one or more of these systems to evaluate their approaches, which allows us to compare and evaluate our experiment results.

Experiment Data System information Version Number of Classes Number of Files 484 530 6.0 beta 1 JHotDraw 394 1001 4.2 JEdit 93 126 3.8.2 JUnit 345 570 JDK1.4.2 Java.AWT File # Class # Version Systems

System information

Version

Number of Classes

Number of Files

Experiment Result 64 24 6 76 Strategy 0 58 4 JHotDraw 0 24 17 JEdit 3 6 3 JUnit 3 65 21 Java.AWT Composite Bridge Adapter Systems

Experiment Result – Java.AWT Results for Each Analysis Phase N/A 76 65 N/A Semantic Analysis 3 3 92 Composite 76 76 100 Strategy 65 76 100 Bridge 21 21 57 Adapter Final Result Behavioral Analysis Structural Analysis Systems Java.AWT

Experiment Result – JUnit Results for Each Analysis Phase N/A 6 6 N/A Semantic Analysis 3 6 6 3 Final Result 3 9 Composite 6 6 Strategy 6 6 Bridge 3 15 Adapter Behavioral Analysis Structural Analysis Systems JUnit

Experiment Result – JEdit Results for Each Analysis Phase N/A 24 24 N/A Semantic Analysis 0 0 0 Composite 24 24 33 Strategy 24 24 33 Bridge 17 17 80 Adapter Final Result Behavioral Analysis Structural Analysis Systems JEdit

Experiment Result – JHotDraw Results for Each Analysis Phase N/A 64 58 N/A Semantic Analysis 0 0 0 Composite 64 64 74 Strategy 58 64 74 Bridge 4 4 27 Adapter Final Result Behavioral Analysis Structural Analysis Systems JHotDraw

Observations Behavioral analysis phase only deals with a small number of classes, since it’s based on the result of structural analysis phase. Significant reduction of the candidates between structural and behavioral analysis of Adapter and Composite pattern than those of Bridge and Strategy patterns, since the former have more behavioral characteristics than the latter do. Result of structural and behavioral analysis of the Bridge and Strategy patterns are the same, since they have same structural and behavioral characteristics and only differ in their intents.

Behavioral analysis phase only deals with a small number of classes, since it’s based on the result of structural analysis phase.

Significant reduction of the candidates between structural and behavioral analysis of Adapter and Composite pattern than those of Bridge and Strategy patterns, since the former have more behavioral characteristics than the latter do.

Result of structural and behavioral analysis of the Bridge and Strategy patterns are the same, since they have same structural and behavioral characteristics and only differ in their intents.

Recovery Precisions for JHotDraw Manually checked the results generated by our tool and see whether they are real pattern instances. TP: True Positive FP: False Positive 100% 0 0 Composite 90.63% 6 58 Strategy 91.38% 5 53 Bridge 100% 0 4 Adapter Precision FP TP JHotDraw

Manually checked the results generated by our tool and see whether they are real pattern instances.

TP: True Positive

FP: False Positive

Instances Missed Instances manually found but missed by DP-Miner Recall: 89.23% DrawingView SelectionTool [7] Locator LocatorConnector [6] Locator LocatorHandle [5] Locator PolygonHandle [4] Connector ConnectionTool [3] Connector ChangeConnectionHandle [2] Connector LineConnection [1] STRATEGY CONTEXT

Instances manually found but missed by DP-Miner

Recall: 89.23%

Reasons of Discrepancy Flexibility of design pattern Composite pattern: Collapse both the Component and Composite classes into a single class or not. Adapter pattern: Full matching all three roles, i.e. Target, Adapter, and Adaptee, or partial matching of only Adapter and Adaptee. Object-oriented programming languages provide special language constructs that greatly simplify the implementation of a design pattern. E.g., Java provides LinkedList, ArrayList, HashMap, and Hashtable, which make the implementation of aggregate elements in patterns easy, but the detection harder.

Flexibility of design pattern

Composite pattern: Collapse both the Component and Composite classes into a single class or not.

Adapter pattern: Full matching all three roles, i.e. Target, Adapter, and Adaptee, or partial matching of only Adapter and Adaptee.

Object-oriented programming languages provide special language constructs that greatly simplify the implementation of a design pattern. E.g., Java provides LinkedList, ArrayList, HashMap, and Hashtable, which make the implementation of aggregate elements in patterns easy, but the detection harder.

Benchmark Correct number of pattern instances and their locations are generally not available. Lacking benchmarks is the main impediment Hard to evaluate and compare design pattern discovery techniques. Hard to judge whether an approach discovers all instances and whether the discovered ones are correct. To calculate the precision and recall of a pattern matching result, it’s essential to know The number of correct pattern instances The locations of correct pattern instances JHotDraw pattern benchmark

Correct number of pattern instances and their locations are generally not available.

Lacking benchmarks is the main impediment

Hard to evaluate and compare design pattern discovery techniques.

Hard to judge whether an approach discovers all instances and whether the discovered ones are correct.

To calculate the precision and recall of a pattern matching result, it’s essential to know

The number of correct pattern instances

The locations of correct pattern instances

JHotDraw pattern benchmark

Conclusion We did a series of experiments on design pattern discovery from open-source systems using DP-Miner Patterns concerned: Adapter, Bridge, Strategy, Composite Experiment data: Java.AWT, JUnit, JEdit, JHotDraw We compared our experiment results with others and found several discrepancies. We analyzed the issues and discussed possible reasons. We argue for benchmarks for design pattern discovery.

We did a series of experiments on design pattern discovery from open-source systems using DP-Miner

Patterns concerned: Adapter, Bridge, Strategy, Composite

Experiment data: Java.AWT, JUnit, JEdit, JHotDraw

We compared our experiment results with others and found several discrepancies.

We analyzed the issues and discussed possible reasons.

We argue for benchmarks for design pattern discovery.

Future Work Improve our tool. Experiment on other systems Manually check results of other systems and perfect the benchmark Research on other object-oriented languages, such as C++.

Improve our tool.

Experiment on other systems

Manually check results of other systems and perfect the benchmark

Research on other object-oriented languages, such as C++.

Thank You!

Questions?

Overall Architecture of Our Approach System  Design Pattern match Source Code Intermediate Representation

Structural Analysis – Matrix

1 1 1 1 7 1 1 1 1 TextComponent 1 1 1 1 7 1 1 1 1 Scrollbar 1 1 1 1 7 1 1 1 1 Label 1 1 1 1 35 1 1 1 1 Container 1 1 1 5 1 1 1 1 1 Component 1 1 1 1 7 1 1 1 1 Choice 1 1 1 1 7 1 1 1 1 Checkbox 1 1 1 1 7 1 1 1 1 Canvas 1 1 1 1 7 1 1 1 1 Button TextComponent Scrollbar Label Container Component Choice Checkbox Canvas Button

1 1 1 1 7 1 1 1 1 TextComponent 1 1 1 1 7 1 1 1 1 Scrollbar 1 1 1 1 7 1 1 1 1 Label 1 1 1 1 35 1 1 1 1 Container 1 1 1 5 1 1 1 1 1 Component 1 1 1 1 7 1 1 1 1 Choice 1 1 1 1 7 1 1 1 1 Checkbox 1 1 1 1 7 1 1 1 1 Canvas 1 1 1 1 7 1 1 1 1 Button TextComponent Scrollbar Label Container Component Choice Checkbox Canvas Button 1 35 1 Composite 1 1 1 Component 1 7 1 Leaf Composite Component Leaf