Kevin Moran, profile picture
Uploaded byKevin Moran
PDF, PPTX886 views

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

The document discusses a machine learning-based approach for prototyping graphical user interfaces (GUIs) in mobile apps, addressing challenges such as market competitiveness and user feedback. It details a multi-phase process involving GUI component detection, classification, and hierarchy construction using deep learning techniques, particularly Convolutional Neural Networks (CNNs). An empirical study evaluates the accuracy of the CNN classifier in recognizing various GUI components, highlighting the project's industrial applicability.

Embed presentation

Download as PDF, PPTX
1 / 154
2 / 154
3 / 154
4 / 154
5 / 154
6 / 154
7 / 154
8 / 154
9 / 154
10 / 154
11 / 154
12 / 154
13 / 154
14 / 154
15 / 154
16 / 154
17 / 154
18 / 154
19 / 154
20 / 154
21 / 154
22 / 154
23 / 154
24 / 154
25 / 154
26 / 154
27 / 154
28 / 154
29 / 154
30 / 154
31 / 154
32 / 154
33 / 154
34 / 154
35 / 154
36 / 154
37 / 154
38 / 154
39 / 154
40 / 154
41 / 154
42 / 154
43 / 154
44 / 154
45 / 154
46 / 154
47 / 154
48 / 154
49 / 154
50 / 154
51 / 154
52 / 154
53 / 154
54 / 154
55 / 154
56 / 154
57 / 154
58 / 154
59 / 154
60 / 154
61 / 154
62 / 154
63 / 154
64 / 154
65 / 154
66 / 154
67 / 154
68 / 154
69 / 154
70 / 154
71 / 154
72 / 154
73 / 154
74 / 154
75 / 154
76 / 154
77 / 154
78 / 154
79 / 154
80 / 154
81 / 154
82 / 154
83 / 154
84 / 154
85 / 154
86 / 154
87 / 154
88 / 154
89 / 154
90 / 154
91 / 154
92 / 154
93 / 154
94 / 154
95 / 154
96 / 154
97 / 154
98 / 154
99 / 154
100 / 154
101 / 154
102 / 154
103 / 154
104 / 154
105 / 154
106 / 154
107 / 154
108 / 154
109 / 154
110 / 154
111 / 154
112 / 154
113 / 154
114 / 154
115 / 154
116 / 154
117 / 154
118 / 154
119 / 154
120 / 154
121 / 154
122 / 154
123 / 154
124 / 154
125 / 154
126 / 154
127 / 154
128 / 154
129 / 154
130 / 154
131 / 154
132 / 154
133 / 154
134 / 154
135 / 154
136 / 154
137 / 154
138 / 154
139 / 154
140 / 154
141 / 154
142 / 154
143 / 154
144 / 154
145 / 154
146 / 154
147 / 154
148 / 154
149 / 154
150 / 154
151 / 154
152 / 154
153 / 154
154 / 154
Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps IEEETransactions on Software Engineering Journal First Presentation ESEC/FSE’18 Thursday, November 8th, 2018 Kevin Moran, Carlos Bernal Cardenas Michael Curcio, Richard Bonett, & Denys Poshyvanyk
HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING?
HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING?
HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING?
HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING? Make the instantiation of concepts as frictionless as possible
HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING? Make the instantiation of concepts as frictionless as possible Facilitate growing software organically
WE NEED DEVELOPER-CENTRIC AUTOMATION
!7
!7
Video Source: MKBHD
Video Source: MKBHD
Video Source: Google
Video Source: Google
MOBILE DEVELOPMENT CHALLENGES
MOBILE DEVELOPMENT CHALLENGES Highly Competitive App Marketplaces
MOBILE DEVELOPMENT CHALLENGES Noisy Feedback from User Reviews Highly Competitive App Marketplaces
MOBILE DEVELOPMENT CHALLENGES Noisy Feedback from User Reviews Highly Competitive App Marketplaces Change/Fault Prone APIs
MOBILE DEVELOPMENT CHALLENGES Noisy Feedback from User Reviews Highly Competitive App Marketplaces Pressure for Frequent Releases Change/Fault Prone APIs
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam Sketch
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam Prototype GUI Code
RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam Prototype GUI Code
EXISTING WORK • Utilizes a combination of unsupervised ComputerVision Techniques to Detect Components • Binary Component Classification (Text or Image) • GUI-hierarchy generated using heuristics Reverse Engineering Mobile App Interfaces* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Neural MachineTranslation of Images to Code✦ ✦Tony Beltramelli, “pix2code: Generating code from a graphical user interface screenshot,” CoRR, vol. abs/1705.07962, 2017. [Online]. Available: http://arxiv.org/abs/1705.07962 • Utilizes a Deep-Learning Machine Translation Approach to convert Images into a DSL and then apps • Only tested on a small synthetic set of apps • Requires a DSL that must be maintained
EXISTING WORK Commercial Solutions
FIRST PRINCIPLES How does a developer translate a GUI mock-up into code?
FIRST PRINCIPLES 1. Detect or identify GUI-components that exist in a mock-up How does a developer translate a GUI mock-up into code?
FIRST PRINCIPLES 1. Detect or identify GUI-components that exist in a mock-up 2. Classify these GUI-components into their constituent types How does a developer translate a GUI mock-up into code?
FIRST PRINCIPLES 1. Detect or identify GUI-components that exist in a mock-up 2. Classify these GUI-components into their constituent types 3. Assemble the GUI-components into a hierarchy and stipulate styles How does a developer translate a GUI mock-up into code?
FIRST PRINCIPLES
FIRST PRINCIPLES Detection
FIRST PRINCIPLES ClassificationDetection
FIRST PRINCIPLES AssemblyClassificationDetection
FIRST PRINCIPLES AssemblyClassificationDetection— —
THE REDRAW FRAMEWORK
PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Input Image
PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Input Image
PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Input Image
PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Input Image
PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes Input Image
PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes GUI-Component Bounding Boxes Input Image
PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes GUI-Component Bounding Boxes Input Image Credits to: Steven Walker & William Hollingsworth
PHASE 1: GUI-COMPONENT DETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch
PHASE 1: GUI-COMPONENT DETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch Marketch Parsing Engine
PHASE 1: GUI-COMPONENT DETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch Marketch Parsing Engine GUI-Component Bounding Boxes
PHASE 2: GUI-COMPONENT CLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
PHASE 2: GUI-COMPONENT CLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ ConvNets or CNNs have revolutionized the task of image classification
PHASE 2: GUI-COMPONENT CLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ ConvNets or CNNs have revolutionized the task of image classification Advanced approaches have reached human-levels of classification accuracy
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j Calculate the Error According to Gradient Descent
ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j Calculate the Error According to Gradient Descent Backpropagate the Error through the network and update weights
PHASE 2: GUI-COMPONENT CLASSIFICATION How can we extract a large enough dataset of training images?
PHASE 2: GUI-COMPONENT CLASSIFICATION How can we extract a large enough dataset of training images? Mining Software Repositories
PHASE 2: GUI-COMPONENT CLASSIFICATION How can we extract a large enough dataset of training images? Mining Software Repositories Automated Dynamic Analysis +
PHASE 2: GUI-COMPONENT CLASSIFICATION
PHASE 2: GUI-COMPONENT CLASSIFICATION Software Repo Miner
PHASE 2: GUI-COMPONENT CLASSIFICATION Software Repo Miner Image Button Image Image Button Image Button TextView Image Image Button GUI Hierarchies Labeled ImagesExecution Engine
PHASE 2: GUI-COMPONENT CLASSIFICATION ReDraw’s CNN Architecture
PHASE 3: GUI-HIERARCHY CONSTRUCTION
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination Generated GUI-Hierarchy
PHASE 3:APPLICATION ASSEMBLY Labeled GUI Component Hierarchy
PHASE 3:APPLICATION ASSEMBLY GUI-Code Generation Color Quantization & Histogram Analysis Labeled GUI Component Hierarchy
PHASE 3:APPLICATION ASSEMBLY GUI-Code Generation Color Quantization & Histogram Analysis GUI-Code Generation Color Quantization & Histogram AnalysisLabeled GUI Component Hierarchy Functional Prototype Application
PHASE 3:APPLICATION ASSEMBLY Target Component
PHASE 3:APPLICATION ASSEMBLY Target Component 0 20 40 60 80 Component Colors
PHASE 3:APPLICATION ASSEMBLY Target Component 0 20 40 60 80 Component Colors
PHASE 3:APPLICATION ASSEMBLY Target Component 0 20 40 60 80 Component Colors
EMPIRICAL STUDY
EMPIRICAL STUDY • RQ1: CNN Accuracy? 

EMPIRICAL STUDY • RQ1: CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 

EMPIRICAL STUDY • RQ1: CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 
 • RQ3: Visual Similarity? 

EMPIRICAL STUDY • RQ1: CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 
 • RQ3: Visual Similarity? 
 • RQ4: Industrial Applicability?
STUDY 1: CNN CLASSIFIER ACCURACY Context 8,655 Applications 14,382 Unique Screens 191,300 Labeled GUI Component Images ReDraw CNN Classifier Supervised BoVW Baseline 15 of the most widely used GUI-componentTypes
STUDY 1: CNN CLASSIFIER ACCURACY Data Augmentation Automatically Generated Synthetic Applications Automatically Shift Color of Components through Hue Shifting
STUDY 1: CNN CLASSIFIER ACCURACY GUI Type Total # (C) Tr (O) Tr (O+S) Valid Test TextView 99,200 74,087 74,087 15,236 9,877 ImageView 53,324 39,983 39,983 7,996 5,345 Button 16,007 12,007 12,007 2,400 1,600 ImageButton 8,693 6,521 6,521 1,306 866 EditText 5,643 4,230 5,000 846 567 CheckedTextView 3,424 2,582 5,000 505 337 CheckBox 1,650 1,238 5,000 247 165 RadioButton 1,293 970 5,000 194 129 ProgressBar 406 307 5,000 60 39 SeekBar 405 304 5,000 61 40 NumberPicker 378 283 5,000 57 38 Switch 373 280 5,000 56 37 ToggleButton 265 199 5,000 40 26 RatingBar 219 164 5,000 33 22 Spinner 20 15 5,000 3 2 Total 191,300 143,170 187,598 29,040 19,090
STUDY 1:RESULTS Confusion Matrix for BoVW Baseline Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 59% 4% 9% 1% 6% 2% 8% 6% 0% 1% 2% 0% 1% 0% 2% IV 5345 4% 51% 4% 1% 2% 11% 2% 18% 1% 1% 3% 0% 2% 0% 2% Bt 1600 6% 6% 59% 1% 5% 4% 7% 4% 0% 1% 1% 0% 0% 3% 1% S 37 5% 0% 3% 65% 0% 0% 5% 22% 0% 0% 0% 0% 0% 0% 0% ET 567 6% 2% 4% 1% 62% 1% 4% 15% 0% 0% 1% 0% 0% 4% 1% IBt 866 2% 16% 3% 0% 2% 61% 1% 9% 1% 1% 2% 0% 1% 0% 3% CTV 337 3% 1% 7% 1% 3% 0% 81% 1% 0% 0% 2% 0% 0% 0% 2% PB 41 0% 24% 2% 0% 2% 5% 2% 54% 0% 0% 2% 2% 2% 0% 2% RB 22 0% 5% 0% 0% 0% 0% 0% 27% 68% 0% 0% 0% 0% 0% 0% TBt 26 7% 7% 19% 0% 0% 0% 11% 15% 0% 33% 0% 0% 0% 0% 7% CB 165 4% 2% 3% 1% 2% 1% 2% 12% 1% 0% 72% 0% 0% 0% 1% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 0% 5% 0% 0% 0% 0% 0% 18% 3% 0% 5% 0% 68% 0% 3% NP 40 3% 0% 5% 0% 3% 0% 5% 0% 0% 0% 0% 0% 0% 84% 0% RBt 129 6% 3% 5% 1% 3% 0% 6% 18% 0% 1% 1% 0% 1% 0% 55%
STUDY 1:RESULTS Confusion Matrix for BoVW Baseline Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 59% 4% 9% 1% 6% 2% 8% 6% 0% 1% 2% 0% 1% 0% 2% IV 5345 4% 51% 4% 1% 2% 11% 2% 18% 1% 1% 3% 0% 2% 0% 2% Bt 1600 6% 6% 59% 1% 5% 4% 7% 4% 0% 1% 1% 0% 0% 3% 1% S 37 5% 0% 3% 65% 0% 0% 5% 22% 0% 0% 0% 0% 0% 0% 0% ET 567 6% 2% 4% 1% 62% 1% 4% 15% 0% 0% 1% 0% 0% 4% 1% IBt 866 2% 16% 3% 0% 2% 61% 1% 9% 1% 1% 2% 0% 1% 0% 3% CTV 337 3% 1% 7% 1% 3% 0% 81% 1% 0% 0% 2% 0% 0% 0% 2% PB 41 0% 24% 2% 0% 2% 5% 2% 54% 0% 0% 2% 2% 2% 0% 2% RB 22 0% 5% 0% 0% 0% 0% 0% 27% 68% 0% 0% 0% 0% 0% 0% TBt 26 7% 7% 19% 0% 0% 0% 11% 15% 0% 33% 0% 0% 0% 0% 7% CB 165 4% 2% 3% 1% 2% 1% 2% 12% 1% 0% 72% 0% 0% 0% 1% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 0% 5% 0% 0% 0% 0% 0% 18% 3% 0% 5% 0% 68% 0% 3% NP 40 3% 0% 5% 0% 3% 0% 5% 0% 0% 0% 0% 0% 0% 84% 0% RBt 129 6% 3% 5% 1% 3% 0% 6% 18% 0% 1% 1% 0% 1% 0% 55% Overall Accuracy 65%
STUDY 1:RESULTS Confusion Matrix for ReDraw CNN Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 94% 3% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IV 5345 5% 93% 1% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% Bt 1600 11% 6% 81% 0% 1% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% S 37 5% 3% 0% 87% 0% 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% ET 567 14% 3% 2% 0% 81% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IBt 866 4% 23% 1% 0% 0% 72% 0% 0% 0% 0% 0% 0% 0% 0% 0% CTV 337 7% 0% 0% 0% 0% 0% 93% 0% 0% 0% 0% 0% 0% 0% 0% PB 41 15% 29% 0% 0% 0% 0% 0% 56% 0% 0% 0% 0% 0% 0% 0% RB 22 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% 0% 0% 0% TBt 26 19% 22% 7% 0% 0% 0% 0% 0% 0% 52% 0% 0% 0% 0% 0% CB 165 12% 7% 0% 0% 1% 0% 0% 0% 0% 0% 81% 0% 0% 0% 0% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 10% 13% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 78% 0% 0% NP 40 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 95% 0% RBt 129 4% 3% 2% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 89%
STUDY 1:RESULTS Confusion Matrix for ReDraw CNN Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 94% 3% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IV 5345 5% 93% 1% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% Bt 1600 11% 6% 81% 0% 1% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% S 37 5% 3% 0% 87% 0% 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% ET 567 14% 3% 2% 0% 81% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IBt 866 4% 23% 1% 0% 0% 72% 0% 0% 0% 0% 0% 0% 0% 0% 0% CTV 337 7% 0% 0% 0% 0% 0% 93% 0% 0% 0% 0% 0% 0% 0% 0% PB 41 15% 29% 0% 0% 0% 0% 0% 56% 0% 0% 0% 0% 0% 0% 0% RB 22 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% 0% 0% 0% TBt 26 19% 22% 7% 0% 0% 0% 0% 0% 0% 52% 0% 0% 0% 0% 0% CB 165 12% 7% 0% 0% 1% 0% 0% 0% 0% 0% 81% 0% 0% 0% 0% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 10% 13% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 78% 0% 0% NP 40 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 95% 0% RBt 129 4% 3% 2% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 89% Overall Accuracy 91%
STUDY 2:GUI HIERARCHY CONSTRUCTION Context ReDraw Pix2Code32 Applications 2-3 Unique Screens (Not used inTraining the CNN) REMAUI (Re-implementation)
STUDY 2:GUI HIERARCHY CONSTRUCTION Target GUI Hierarchy Prototype GUI-Hierarchy Edit Distance Calculation # of Insertions - 0 < Insertion Penalty < 1 # of Deletions - 0 < Deletion Penalty < 1 # of Substitutions - 0 < Substitution Penalty < 1
STUDY 2:GUI HIERARCHY CONSTRUCTION Target GUI Hierarchy Prototype GUI-Hierarchy Edit Distance Calculation # of Insertions - 0 < Insertion Penalty < 1 # of Deletions - 0 < Deletion Penalty < 1 # of Substitutions - 0 < Substitution Penalty < 1
STUDY 2: RESULTS Hierarchy Edit Distance to GroundTruth A) Insertion Edits B) Deletion Edits C) Substitution Edits Fraction of PenaltyFraction of PenaltyFraction of Penalty EditDistance EditDistance EditDistance
STUDY 3:VISUAL SIMILARITY OF APPS Context ReDraw Pix2Code32 Applications 2-3 Unique Screens (Not used inTraining the CNN) REMAUI (Re-implementation)
STUDY 3:VISUAL SIMILARITY OF APPS Target App Screenshot Generated App Screenshot Pixel-Based Mean Average Error (MAE) Mean Squared Error (MSE)
STUDY 3:VISUAL SIMILARITY OF APPS Target App Screenshot Generated App Screenshot Pixel-Based Mean Average Error (MAE) Mean Squared Error (MSE)
STUDY 3: RESULTS ReDraw−MU(MAE) ReDraw−CV(MAE) REMAUI(MAE) Pix2Code(MAE) ReDraw−MU(MSE) ReDraw−CV(MSE) REMAUI(MSE) Pix2Code(MSE) 0.0 0.2 0.4 0.6 0.8 MSE and MAE compared to GroundTruth Screenshots Mann Whitney U-Test & Cliff’s Delta MAE MSE Test p-Value d ReDrawMU vs. ReDrawCV 0.835 0.02 (Small) ReDrawMU vs. REMAUI 0.542 0.06 (Small) ReDrawMU vs. pix2code <0.0002 -0.34 (Medium) pix2code vs. ReDrawCV <0.0001 0.35 (Medium) pix2code vs. REMAUI <0.0001 0.39 (Medium) REMAUI vs. ReDrawCV 0.687 -0.04 (Small) Test p-Value d ReDrawMU vs. ReDrawCV 0.771 0.03 (Small) ReDrawMU vs. REMAUI <0.0001 0.45 (Medium) ReDrawMU vs. pix2code <0.003 -0.27 (Small) pix2code vs. ReDrawCV <0.002 0.28 (Small) pix2code vs. REMAUI <0.0001 0.61 (Large) REMAUI vs. ReDrawCV <0.0001 -0.42 (Medium)
APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (MockUp) Textgram
APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (MockUp) Zillow
APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (MockUp) Bank of America
APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (CV) Giphy
APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (CV) Netflix
STUDY4: DEVELOPER UTILITY Context Mobile Front-End Developer Mobile Designer 9 Screens from 8 Popular Apps Mobile Researcher
STUDY4: RESULTS
STUDY4: RESULTS “It’s a good starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]”
STUDY4: RESULTS “It’s a good starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]” “There are going to be edge cases for different layouts, but these are easily fixed after the fact”
STUDY4: RESULTS “It’s a good starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]” “There are going to be edge cases for different layouts, but these are easily fixed after the fact” “The key thing is fast iteration.A developer could generate the initial view [using ReDraw], clean up the layouts, and have a working app. In the future, if a designer could upload a screenshot, and without any other intervention [ReDraw] could update the [existing] xml this would be ideal.”
https://www.android-dev-tools.com/redraw/
https://www.android-dev-tools.com/redraw/
https://www.android-dev-tools.com/redraw/
Carlos Bernal Cardenas Michael Curcio Richard Bonett Denys Poshyvanyk
Any Questions? Thank you! Kevin Moran Ph.D. Post-Doctoral Fellow kpmoran@cs.wm.edu http://www.cs.wm.edu/~kpmoran
ADDITIONAL SLIDES
“The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation.” – Fredrick Brooks, No Silver Bullet – Essence and Accident in Software Engineering (1986)
DISTRIBUTION OF GUI COMPONENTS IN DATASET

More Related Content

PPTX
Software Measurement and Metrics.pptx
byubaidullah75790
 
PPTX
Software development life cycle (SDLC)
bySimran Kaur
 
PPT
Software Engineering (Software Process: A Generic View)
byShudipPal
 
PDF
SOFTWARE PROJECT PLANNING on Software Engineering
bywork90665
 
PPTX
Software scope
byShubham Dubey
 
PDF
Project report software major project
byAnnurajsingh
 
PPTX
Software Project Management - Staffing
byTanishqRongta1
 
PPTX
Staffing level estimation
bykavitha muneeshwaran
 
Software Measurement and Metrics.pptx
byubaidullah75790
 
Software development life cycle (SDLC)
bySimran Kaur
 
Software Engineering (Software Process: A Generic View)
byShudipPal
 
SOFTWARE PROJECT PLANNING on Software Engineering
bywork90665
 
Software scope
byShubham Dubey
 
Project report software major project
byAnnurajsingh
 
Software Project Management - Staffing
byTanishqRongta1
 
Staffing level estimation
bykavitha muneeshwaran
 

What's hot

PPTX
Software myths | Software Engineering Notes
byNavjyotsinh Jadeja
 
PPT
Software Engineering ppt
byshruths2890
 
PPTX
PROTOTYPE MODEL
byshenagarg44
 
PPTX
Software project management- Software Engineering
byMuhammad Yousuf Abdul Qadir
 
PPTX
Lect4 software economics
bymeena466141
 
PPT
Software Engineering (Process Models)
byShudipPal
 
PPT
Rad model
bySneha Chopra
 
PDF
Introduction to Multimedia Applications
bySanjaya Dharmadasa
 
PPT
Currency converter
bySyeda Khadizatul maria
 
PPT
Software estimation
byMd Shakir
 
PPT
Input devices in computer graphics
byAnu Garg
 
PPTX
Bezier curve computer graphics
byUniversity of Potsdam
 
PPT
2D transformation (Computer Graphics)
byTimbal Mayank
 
PPTX
Agile Unified Process (AUP).pptx
byKarim Monir
 
PPT
Graphics software
byMohd Arif
 
PPTX
Digital video
bylavanya marichamy
 
PPT
Animation Techniques
byMedia Studies
 
PPTX
Cohen sutherland line clipping
byMani Kanth
 
PPTX
Clipping computer graphics
byShaishavShah8
 
PPTX
Projections.pptx
byjyoti_lakhani
 
Software myths | Software Engineering Notes
byNavjyotsinh Jadeja
 
Software Engineering ppt
byshruths2890
 
PROTOTYPE MODEL
byshenagarg44
 
Software project management- Software Engineering
byMuhammad Yousuf Abdul Qadir
 
Lect4 software economics
bymeena466141
 
Software Engineering (Process Models)
byShudipPal
 
Rad model
bySneha Chopra
 
Introduction to Multimedia Applications
bySanjaya Dharmadasa
 
Currency converter
bySyeda Khadizatul maria
 
Software estimation
byMd Shakir
 
Input devices in computer graphics
byAnu Garg
 
Bezier curve computer graphics
byUniversity of Potsdam
 
2D transformation (Computer Graphics)
byTimbal Mayank
 
Agile Unified Process (AUP).pptx
byKarim Monir
 
Graphics software
byMohd Arif
 
Digital video
bylavanya marichamy
 
Animation Techniques
byMedia Studies
 
Cohen sutherland line clipping
byMani Kanth
 
Clipping computer graphics
byShaishavShah8
 
Projections.pptx
byjyoti_lakhani
 

Similar to Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

PDF
UI2code : A Neural Machine Translator to Bootstrap Mobile GUI Implementation
byChunyang Chen
 
PPTX
Scientists meet Entrepreneurs - AI & Machine Learning, Tambet Matiisen, Unive...
byMobileMonday Estonia
 
PDF
COMP 4026 Lecture4: Processing and Advanced Interface Technology
byMark Billinghurst
 
PDF
IRJET- Generation of HTML Code using Machine Learning Techniques from Mock-Up...
byIRJET Journal
 
PPTX
KnowiXML: A Knowledge-Based System Generating Multiple Abstract User Interfac...
byJean Vanderdonckt
 
PDF
Design document
byPiTechnologies
 
PDF
Creating Advanced GUIs for Low-power MCUs with Qt
byICS
 
PPTX
Multi-Dimensional Context-Aware Adaptation of Service Front-ends
byJean Vanderdonckt
 
PDF
How I came to build a prototype for mobile develop
byRice Tseng
 
PDF
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
byApplitools
 
PDF
Acceptance Testing of Web UI
byVladimir Tsukur
 
PDF
Modelling the User Interface
byPedro J. Molina
 
PPTX
Re-Engineering Graphical User Interfaces from their Resource Files with UsiRe...
byJean Vanderdonckt
 
KEY
UIAutomation + Mechanic.js
byjaykz52
 
PPTX
ICS3211 lecture 08
byVanessa Camilleri
 
PDF
ICS3211 Lecture 08 2020
byVanessa Camilleri
 
PPTX
Generative Testing in Clojure
byAlistair Roche
 
PPT
Proiect
byGalea Catalin
 
PDF
Back To The Future
byBill Scott
 
PDF
Copy Your Favourite Nokia App with Qt
byaccount inactive
 
UI2code : A Neural Machine Translator to Bootstrap Mobile GUI Implementation
byChunyang Chen
 
Scientists meet Entrepreneurs - AI & Machine Learning, Tambet Matiisen, Unive...
byMobileMonday Estonia
 
COMP 4026 Lecture4: Processing and Advanced Interface Technology
byMark Billinghurst
 
IRJET- Generation of HTML Code using Machine Learning Techniques from Mock-Up...
byIRJET Journal
 
KnowiXML: A Knowledge-Based System Generating Multiple Abstract User Interfac...
byJean Vanderdonckt
 
Design document
byPiTechnologies
 
Creating Advanced GUIs for Low-power MCUs with Qt
byICS
 
Multi-Dimensional Context-Aware Adaptation of Service Front-ends
byJean Vanderdonckt
 
How I came to build a prototype for mobile develop
byRice Tseng
 
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
byApplitools
 
Acceptance Testing of Web UI
byVladimir Tsukur
 
Modelling the User Interface
byPedro J. Molina
 
Re-Engineering Graphical User Interfaces from their Resource Files with UsiRe...
byJean Vanderdonckt
 
UIAutomation + Mechanic.js
byjaykz52
 
ICS3211 lecture 08
byVanessa Camilleri
 
ICS3211 Lecture 08 2020
byVanessa Camilleri
 
Generative Testing in Clojure
byAlistair Roche
 
Proiect
byGalea Catalin
 
Back To The Future
byBill Scott
 
Copy Your Favourite Nokia App with Qt
byaccount inactive
 

More from Kevin Moran

PDF
ICSE17 - Tool Demonstration - CrashScope A Practical Tool for the Automated T...
byKevin Moran
 
PDF
On-Device Bug Reporting for Android Applications
byKevin Moran
 
PDF
Assessing Test Case Prioritization on Real Faults and Mutants
byKevin Moran
 
PDF
Automated Reporting of GUI Design Violations for Mobile Apps
byKevin Moran
 
PDF
ICSE17Tech Briefing - Automated GUI Testing of Android Apps: From Research to...
byKevin Moran
 
PDF
Auto-completing Bug Reports for Android Applications
byKevin Moran
 
PDF
Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mob...
byKevin Moran
 
PDF
Automatically Discovering, Reporting and Reproducing Android Application Crashes
byKevin Moran
 
PDF
Discovering Flaws in Security-Focused Static Analysis Tools for Android using...
byKevin Moran
 
PDF
Automated GUI-Testing of Android Apps: From Research to Practice
byKevin Moran
 
PDF
How do Developers Test Android Applications?
byKevin Moran
 
PDF
MDroid+: A Mutation Testing Framework for Android
byKevin Moran
 
PDF
Detecting and Summarizing GUI Changes in Evolving Mobile Apps
byKevin Moran
 
PDF
A Large-Scale Empirical Comparison of Static and DynamicTest Case Prioritizat...
byKevin Moran
 
ICSE17 - Tool Demonstration - CrashScope A Practical Tool for the Automated T...
byKevin Moran
 
On-Device Bug Reporting for Android Applications
byKevin Moran
 
Assessing Test Case Prioritization on Real Faults and Mutants
byKevin Moran
 
Automated Reporting of GUI Design Violations for Mobile Apps
byKevin Moran
 
ICSE17Tech Briefing - Automated GUI Testing of Android Apps: From Research to...
byKevin Moran
 
Auto-completing Bug Reports for Android Applications
byKevin Moran
 
Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mob...
byKevin Moran
 
Automatically Discovering, Reporting and Reproducing Android Application Crashes
byKevin Moran
 
Discovering Flaws in Security-Focused Static Analysis Tools for Android using...
byKevin Moran
 
Automated GUI-Testing of Android Apps: From Research to Practice
byKevin Moran
 
How do Developers Test Android Applications?
byKevin Moran
 
MDroid+: A Mutation Testing Framework for Android
byKevin Moran
 
Detecting and Summarizing GUI Changes in Evolving Mobile Apps
byKevin Moran
 
A Large-Scale Empirical Comparison of Static and DynamicTest Case Prioritizat...
byKevin Moran
 

Recently uploaded

PDF
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
PDF
Chapter 3 Cryptography and encryption techniques.pdf
byGetnet Tigabie Askale -(GM)
 
PDF
apidays Australia 2025 | The Strategic Role of APIs in Digital Transformation
byapidays
 
PDF
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
PDF
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 
PDF
10 Best AI Tools for Fashion Brands in 2026
bymockitseo
 
PPTX
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
PDF
Quick Learn Laravel for Beginner from Zero to Survive
byiDev Semarang
 
PDF
B2SMB SaaS Philosophy Lab: Simplicity vs Options
byAnton Shulke
 
PDF
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 
PPTX
AI in Marine Insurance Future of Smarter Risk, Faster Claims & Safer Shipping...
byAutomationEdge Technologies
 
PDF
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
PPTX
CRM for the Insurance Industry Lessons for Insurance CIOs and Digital Leaders...
byKerry Millar
 
PDF
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
PDF
Introduction to Linux and Basic Commands
byiDev Semarang
 
PDF
How to Design Premium Health (Public Health) PPT Using AI? Top Strategies
byPublic Health Concern Nepal
 
PDF
LUXHUB: a detailed look at 2025...and fast forward to 2026
byalexandrekeilmann1
 
PDF
Agentic AI Roadmap 2026: Mastering Autonomous AI Workflows and Systems
byAeafat Ahmed Mubin
 
PPTX
Advance Computer Architecture Lecture 4.pptx
byBottshikanBottshikan
 
DOCX
Mathematical reviewer gor mmw in theofern
byyeojlevantinojesalva
 
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
Chapter 3 Cryptography and encryption techniques.pdf
byGetnet Tigabie Askale -(GM)
 
apidays Australia 2025 | The Strategic Role of APIs in Digital Transformation
byapidays
 
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 
10 Best AI Tools for Fashion Brands in 2026
bymockitseo
 
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
Quick Learn Laravel for Beginner from Zero to Survive
byiDev Semarang
 
B2SMB SaaS Philosophy Lab: Simplicity vs Options
byAnton Shulke
 
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 
AI in Marine Insurance Future of Smarter Risk, Faster Claims & Safer Shipping...
byAutomationEdge Technologies
 
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
CRM for the Insurance Industry Lessons for Insurance CIOs and Digital Leaders...
byKerry Millar
 
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
Introduction to Linux and Basic Commands
byiDev Semarang
 
How to Design Premium Health (Public Health) PPT Using AI? Top Strategies
byPublic Health Concern Nepal
 
LUXHUB: a detailed look at 2025...and fast forward to 2026
byalexandrekeilmann1
 
Agentic AI Roadmap 2026: Mastering Autonomous AI Workflows and Systems
byAeafat Ahmed Mubin
 
Advance Computer Architecture Lecture 4.pptx
byBottshikanBottshikan
 
Mathematical reviewer gor mmw in theofern
byyeojlevantinojesalva
 

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

  • 1.
    Machine Learning-Based Prototyping ofGraphical User Interfaces for Mobile Apps IEEETransactions on Software Engineering Journal First Presentation ESEC/FSE’18 Thursday, November 8th, 2018 Kevin Moran, Carlos Bernal Cardenas Michael Curcio, Richard Bonett, & Denys Poshyvanyk
  • 2.
    HOW CAN AUTOMATIONIMPROVE SOFTWARE ENGINEERING?
  • 3.
    HOW CAN AUTOMATIONIMPROVE SOFTWARE ENGINEERING?
  • 4.
    HOW CAN AUTOMATIONIMPROVE SOFTWARE ENGINEERING?
  • 5.
    HOW CAN AUTOMATIONIMPROVE SOFTWARE ENGINEERING? Make the instantiation of concepts as frictionless as possible
  • 6.
    HOW CAN AUTOMATIONIMPROVE SOFTWARE ENGINEERING? Make the instantiation of concepts as frictionless as possible Facilitate growing software organically
  • 7.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
    MOBILE DEVELOPMENT CHALLENGES HighlyCompetitive App Marketplaces
  • 18.
    MOBILE DEVELOPMENT CHALLENGES NoisyFeedback from User Reviews Highly Competitive App Marketplaces
  • 19.
    MOBILE DEVELOPMENT CHALLENGES NoisyFeedback from User Reviews Highly Competitive App Marketplaces Change/Fault Prone APIs
  • 20.
    MOBILE DEVELOPMENT CHALLENGES NoisyFeedback from User Reviews Highly Competitive App Marketplaces Pressure for Frequent Releases Change/Fault Prone APIs
  • 21.
  • 22.
  • 23.
    RESEARCH PROBLEM UI/UX DesignTeamDevelopmentTeam Sketch
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
    RESEARCH PROBLEM UI/UX DesignTeamDevelopmentTeam Prototype GUI Code
  • 36.
    RESEARCH PROBLEM UI/UX DesignTeamDevelopmentTeam Prototype GUI Code
  • 37.
    EXISTING WORK • Utilizesa combination of unsupervised ComputerVision Techniques to Detect Components • Binary Component Classification (Text or Image) • GUI-hierarchy generated using heuristics Reverse Engineering Mobile App Interfaces* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Neural MachineTranslation of Images to Code✦ ✦Tony Beltramelli, “pix2code: Generating code from a graphical user interface screenshot,” CoRR, vol. abs/1705.07962, 2017. [Online]. Available: http://arxiv.org/abs/1705.07962 • Utilizes a Deep-Learning Machine Translation Approach to convert Images into a DSL and then apps • Only tested on a small synthetic set of apps • Requires a DSL that must be maintained
  • 38.
  • 39.
    FIRST PRINCIPLES How doesa developer translate a GUI mock-up into code?
  • 40.
    FIRST PRINCIPLES 1. Detector identify GUI-components that exist in a mock-up How does a developer translate a GUI mock-up into code?
  • 41.
    FIRST PRINCIPLES 1. Detector identify GUI-components that exist in a mock-up 2. Classify these GUI-components into their constituent types How does a developer translate a GUI mock-up into code?
  • 42.
    FIRST PRINCIPLES 1. Detector identify GUI-components that exist in a mock-up 2. Classify these GUI-components into their constituent types 3. Assemble the GUI-components into a hierarchy and stipulate styles How does a developer translate a GUI mock-up into code?
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
    PHASE 1: GUI-COMPONENTDETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Input Image
  • 50.
    PHASE 1: GUI-COMPONENTDETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Input Image
  • 51.
    PHASE 1: GUI-COMPONENTDETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Input Image
  • 52.
    PHASE 1: GUI-COMPONENTDETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Input Image
  • 53.
    PHASE 1: GUI-COMPONENTDETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes Input Image
  • 54.
    PHASE 1: GUI-COMPONENTDETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes GUI-Component Bounding Boxes Input Image
  • 55.
    PHASE 1: GUI-COMPONENTDETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes GUI-Component Bounding Boxes Input Image Credits to: Steven Walker & William Hollingsworth
  • 56.
    PHASE 1: GUI-COMPONENTDETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch
  • 57.
    PHASE 1: GUI-COMPONENTDETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch Marketch Parsing Engine
  • 58.
    PHASE 1: GUI-COMPONENTDETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch Marketch Parsing Engine GUI-Component Bounding Boxes
  • 59.
    PHASE 2: GUI-COMPONENTCLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
  • 60.
    PHASE 2: GUI-COMPONENTCLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ ConvNets or CNNs have revolutionized the task of image classification
  • 61.
    PHASE 2: GUI-COMPONENTCLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ ConvNets or CNNs have revolutionized the task of image classification Advanced approaches have reached human-levels of classification accuracy
  • 62.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner w1 i…w1 j w2 i…w2 j
  • 63.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 64.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 65.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 66.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 67.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 68.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 69.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 70.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 71.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 72.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 73.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 74.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 75.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 76.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 77.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  • 78.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j Calculate the Error According to Gradient Descent
  • 79.
    ASIDE: CNN BASICS … … … ProgressBar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j Calculate the Error According to Gradient Descent Backpropagate the Error through the network and update weights
  • 80.
    PHASE 2: GUI-COMPONENTCLASSIFICATION How can we extract a large enough dataset of training images?
  • 81.
    PHASE 2: GUI-COMPONENTCLASSIFICATION How can we extract a large enough dataset of training images? Mining Software Repositories
  • 82.
    PHASE 2: GUI-COMPONENTCLASSIFICATION How can we extract a large enough dataset of training images? Mining Software Repositories Automated Dynamic Analysis +
  • 83.
    PHASE 2: GUI-COMPONENTCLASSIFICATION
  • 84.
    PHASE 2: GUI-COMPONENTCLASSIFICATION Software Repo Miner
  • 85.
    PHASE 2: GUI-COMPONENTCLASSIFICATION Software Repo Miner Image Button Image Image Button Image Button TextView Image Image Button GUI Hierarchies Labeled ImagesExecution Engine
  • 86.
    PHASE 2: GUI-COMPONENTCLASSIFICATION ReDraw’s CNN Architecture
  • 87.
  • 88.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 89.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 90.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 91.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 92.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 93.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 94.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 95.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 96.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 97.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 98.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination
  • 99.
    PHASE 3: GUI-HIERARCHYCONSTRUCTION KNN GUI-Hierarchy Determination Generated GUI-Hierarchy
  • 100.
    PHASE 3:APPLICATION ASSEMBLY LabeledGUI Component Hierarchy
  • 101.
    PHASE 3:APPLICATION ASSEMBLY GUI-Code Generation Color Quantization &Histogram Analysis Labeled GUI Component Hierarchy
  • 102.
    PHASE 3:APPLICATION ASSEMBLY GUI-Code Generation Color Quantization &Histogram Analysis GUI-Code Generation Color Quantization & Histogram AnalysisLabeled GUI Component Hierarchy Functional Prototype Application
  • 103.
  • 104.
    PHASE 3:APPLICATION ASSEMBLY TargetComponent 0 20 40 60 80 Component Colors
  • 105.
    PHASE 3:APPLICATION ASSEMBLY TargetComponent 0 20 40 60 80 Component Colors
  • 106.
    PHASE 3:APPLICATION ASSEMBLY TargetComponent 0 20 40 60 80 Component Colors
  • 107.
  • 108.
    EMPIRICAL STUDY • RQ1:CNN Accuracy? 

  • 109.
    EMPIRICAL STUDY • RQ1:CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 

  • 110.
    EMPIRICAL STUDY • RQ1:CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 
 • RQ3: Visual Similarity? 

  • 111.
    EMPIRICAL STUDY • RQ1:CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 
 • RQ3: Visual Similarity? 
 • RQ4: Industrial Applicability?
  • 112.
    STUDY 1: CNNCLASSIFIER ACCURACY Context 8,655 Applications 14,382 Unique Screens 191,300 Labeled GUI Component Images ReDraw CNN Classifier Supervised BoVW Baseline 15 of the most widely used GUI-componentTypes
  • 113.
    STUDY 1: CNNCLASSIFIER ACCURACY Data Augmentation Automatically Generated Synthetic Applications Automatically Shift Color of Components through Hue Shifting
  • 114.
    STUDY 1: CNNCLASSIFIER ACCURACY GUI Type Total # (C) Tr (O) Tr (O+S) Valid Test TextView 99,200 74,087 74,087 15,236 9,877 ImageView 53,324 39,983 39,983 7,996 5,345 Button 16,007 12,007 12,007 2,400 1,600 ImageButton 8,693 6,521 6,521 1,306 866 EditText 5,643 4,230 5,000 846 567 CheckedTextView 3,424 2,582 5,000 505 337 CheckBox 1,650 1,238 5,000 247 165 RadioButton 1,293 970 5,000 194 129 ProgressBar 406 307 5,000 60 39 SeekBar 405 304 5,000 61 40 NumberPicker 378 283 5,000 57 38 Switch 373 280 5,000 56 37 ToggleButton 265 199 5,000 40 26 RatingBar 219 164 5,000 33 22 Spinner 20 15 5,000 3 2 Total 191,300 143,170 187,598 29,040 19,090
  • 115.
    STUDY 1:RESULTS Confusion Matrixfor BoVW Baseline Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 59% 4% 9% 1% 6% 2% 8% 6% 0% 1% 2% 0% 1% 0% 2% IV 5345 4% 51% 4% 1% 2% 11% 2% 18% 1% 1% 3% 0% 2% 0% 2% Bt 1600 6% 6% 59% 1% 5% 4% 7% 4% 0% 1% 1% 0% 0% 3% 1% S 37 5% 0% 3% 65% 0% 0% 5% 22% 0% 0% 0% 0% 0% 0% 0% ET 567 6% 2% 4% 1% 62% 1% 4% 15% 0% 0% 1% 0% 0% 4% 1% IBt 866 2% 16% 3% 0% 2% 61% 1% 9% 1% 1% 2% 0% 1% 0% 3% CTV 337 3% 1% 7% 1% 3% 0% 81% 1% 0% 0% 2% 0% 0% 0% 2% PB 41 0% 24% 2% 0% 2% 5% 2% 54% 0% 0% 2% 2% 2% 0% 2% RB 22 0% 5% 0% 0% 0% 0% 0% 27% 68% 0% 0% 0% 0% 0% 0% TBt 26 7% 7% 19% 0% 0% 0% 11% 15% 0% 33% 0% 0% 0% 0% 7% CB 165 4% 2% 3% 1% 2% 1% 2% 12% 1% 0% 72% 0% 0% 0% 1% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 0% 5% 0% 0% 0% 0% 0% 18% 3% 0% 5% 0% 68% 0% 3% NP 40 3% 0% 5% 0% 3% 0% 5% 0% 0% 0% 0% 0% 0% 84% 0% RBt 129 6% 3% 5% 1% 3% 0% 6% 18% 0% 1% 1% 0% 1% 0% 55%
  • 116.
    STUDY 1:RESULTS Confusion Matrixfor BoVW Baseline Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 59% 4% 9% 1% 6% 2% 8% 6% 0% 1% 2% 0% 1% 0% 2% IV 5345 4% 51% 4% 1% 2% 11% 2% 18% 1% 1% 3% 0% 2% 0% 2% Bt 1600 6% 6% 59% 1% 5% 4% 7% 4% 0% 1% 1% 0% 0% 3% 1% S 37 5% 0% 3% 65% 0% 0% 5% 22% 0% 0% 0% 0% 0% 0% 0% ET 567 6% 2% 4% 1% 62% 1% 4% 15% 0% 0% 1% 0% 0% 4% 1% IBt 866 2% 16% 3% 0% 2% 61% 1% 9% 1% 1% 2% 0% 1% 0% 3% CTV 337 3% 1% 7% 1% 3% 0% 81% 1% 0% 0% 2% 0% 0% 0% 2% PB 41 0% 24% 2% 0% 2% 5% 2% 54% 0% 0% 2% 2% 2% 0% 2% RB 22 0% 5% 0% 0% 0% 0% 0% 27% 68% 0% 0% 0% 0% 0% 0% TBt 26 7% 7% 19% 0% 0% 0% 11% 15% 0% 33% 0% 0% 0% 0% 7% CB 165 4% 2% 3% 1% 2% 1% 2% 12% 1% 0% 72% 0% 0% 0% 1% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 0% 5% 0% 0% 0% 0% 0% 18% 3% 0% 5% 0% 68% 0% 3% NP 40 3% 0% 5% 0% 3% 0% 5% 0% 0% 0% 0% 0% 0% 84% 0% RBt 129 6% 3% 5% 1% 3% 0% 6% 18% 0% 1% 1% 0% 1% 0% 55% Overall Accuracy 65%
  • 117.
    STUDY 1:RESULTS Confusion Matrixfor ReDraw CNN Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 94% 3% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IV 5345 5% 93% 1% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% Bt 1600 11% 6% 81% 0% 1% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% S 37 5% 3% 0% 87% 0% 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% ET 567 14% 3% 2% 0% 81% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IBt 866 4% 23% 1% 0% 0% 72% 0% 0% 0% 0% 0% 0% 0% 0% 0% CTV 337 7% 0% 0% 0% 0% 0% 93% 0% 0% 0% 0% 0% 0% 0% 0% PB 41 15% 29% 0% 0% 0% 0% 0% 56% 0% 0% 0% 0% 0% 0% 0% RB 22 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% 0% 0% 0% TBt 26 19% 22% 7% 0% 0% 0% 0% 0% 0% 52% 0% 0% 0% 0% 0% CB 165 12% 7% 0% 0% 1% 0% 0% 0% 0% 0% 81% 0% 0% 0% 0% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 10% 13% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 78% 0% 0% NP 40 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 95% 0% RBt 129 4% 3% 2% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 89%
  • 118.
    STUDY 1:RESULTS Confusion Matrixfor ReDraw CNN Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 94% 3% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IV 5345 5% 93% 1% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% Bt 1600 11% 6% 81% 0% 1% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% S 37 5% 3% 0% 87% 0% 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% ET 567 14% 3% 2% 0% 81% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IBt 866 4% 23% 1% 0% 0% 72% 0% 0% 0% 0% 0% 0% 0% 0% 0% CTV 337 7% 0% 0% 0% 0% 0% 93% 0% 0% 0% 0% 0% 0% 0% 0% PB 41 15% 29% 0% 0% 0% 0% 0% 56% 0% 0% 0% 0% 0% 0% 0% RB 22 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% 0% 0% 0% TBt 26 19% 22% 7% 0% 0% 0% 0% 0% 0% 52% 0% 0% 0% 0% 0% CB 165 12% 7% 0% 0% 1% 0% 0% 0% 0% 0% 81% 0% 0% 0% 0% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 10% 13% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 78% 0% 0% NP 40 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 95% 0% RBt 129 4% 3% 2% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 89% Overall Accuracy 91%
  • 119.
    STUDY 2:GUI HIERARCHYCONSTRUCTION Context ReDraw Pix2Code32 Applications 2-3 Unique Screens (Not used inTraining the CNN) REMAUI (Re-implementation)
  • 120.
    STUDY 2:GUI HIERARCHYCONSTRUCTION Target GUI Hierarchy Prototype GUI-Hierarchy Edit Distance Calculation # of Insertions - 0 < Insertion Penalty < 1 # of Deletions - 0 < Deletion Penalty < 1 # of Substitutions - 0 < Substitution Penalty < 1
  • 121.
    STUDY 2:GUI HIERARCHYCONSTRUCTION Target GUI Hierarchy Prototype GUI-Hierarchy Edit Distance Calculation # of Insertions - 0 < Insertion Penalty < 1 # of Deletions - 0 < Deletion Penalty < 1 # of Substitutions - 0 < Substitution Penalty < 1
  • 122.
    STUDY 2: RESULTS HierarchyEdit Distance to GroundTruth A) Insertion Edits B) Deletion Edits C) Substitution Edits Fraction of PenaltyFraction of PenaltyFraction of Penalty EditDistance EditDistance EditDistance
  • 125.
    STUDY 3:VISUAL SIMILARITYOF APPS Context ReDraw Pix2Code32 Applications 2-3 Unique Screens (Not used inTraining the CNN) REMAUI (Re-implementation)
  • 126.
    STUDY 3:VISUAL SIMILARITYOF APPS Target App Screenshot Generated App Screenshot Pixel-Based Mean Average Error (MAE) Mean Squared Error (MSE)
  • 127.
    STUDY 3:VISUAL SIMILARITYOF APPS Target App Screenshot Generated App Screenshot Pixel-Based Mean Average Error (MAE) Mean Squared Error (MSE)
  • 128.
    STUDY 3: RESULTS ReDraw−MU(MAE) ReDraw−CV(MAE) REMAUI(MAE) Pix2Code(MAE) ReDraw−MU(MSE) ReDraw−CV(MSE) REMAUI(MSE) Pix2Code(MSE) 0.00.2 0.4 0.6 0.8 MSE and MAE compared to GroundTruth Screenshots Mann Whitney U-Test & Cliff’s Delta MAE MSE Test p-Value d ReDrawMU vs. ReDrawCV 0.835 0.02 (Small) ReDrawMU vs. REMAUI 0.542 0.06 (Small) ReDrawMU vs. pix2code <0.0002 -0.34 (Medium) pix2code vs. ReDrawCV <0.0001 0.35 (Medium) pix2code vs. REMAUI <0.0001 0.39 (Medium) REMAUI vs. ReDrawCV 0.687 -0.04 (Small) Test p-Value d ReDrawMU vs. ReDrawCV 0.771 0.03 (Small) ReDrawMU vs. REMAUI <0.0001 0.45 (Medium) ReDrawMU vs. pix2code <0.003 -0.27 (Small) pix2code vs. ReDrawCV <0.002 0.28 (Small) pix2code vs. REMAUI <0.0001 0.61 (Large) REMAUI vs. ReDrawCV <0.0001 -0.42 (Medium)
  • 129.
    APPLICATIONS GENERATED BYREDRAW A) Original Application B) ReDraw App (MockUp) Textgram
  • 130.
    APPLICATIONS GENERATED BYREDRAW A) Original Application B) ReDraw App (MockUp) Zillow
  • 131.
    APPLICATIONS GENERATED BYREDRAW A) Original Application B) ReDraw App (MockUp) Bank of America
  • 132.
    APPLICATIONS GENERATED BYREDRAW A) Original Application B) ReDraw App (CV) Giphy
  • 133.
    APPLICATIONS GENERATED BYREDRAW A) Original Application B) ReDraw App (CV) Netflix
  • 134.
    STUDY4: DEVELOPER UTILITY Context MobileFront-End Developer Mobile Designer 9 Screens from 8 Popular Apps Mobile Researcher
  • 135.
  • 136.
    STUDY4: RESULTS “It’s agood starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]”
  • 137.
    STUDY4: RESULTS “It’s agood starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]” “There are going to be edge cases for different layouts, but these are easily fixed after the fact”
  • 138.
    STUDY4: RESULTS “It’s agood starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]” “There are going to be edge cases for different layouts, but these are easily fixed after the fact” “The key thing is fast iteration.A developer could generate the initial view [using ReDraw], clean up the layouts, and have a working app. In the future, if a designer could upload a screenshot, and without any other intervention [ReDraw] could update the [existing] xml this would be ideal.”
  • 139.
  • 140.
  • 141.
  • 147.
    Carlos Bernal Cardenas MichaelCurcio Richard Bonett Denys Poshyvanyk
  • 148.
    Any Questions? Thank you! KevinMoran Ph.D. Post-Doctoral Fellow kpmoran@cs.wm.edu http://www.cs.wm.edu/~kpmoran
  • 150.
  • 151.
    “The essence ofa software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation.” – Fredrick Brooks, No Silver Bullet – Essence and Accident in Software Engineering (1986)
  • 154.
    DISTRIBUTION OF GUICOMPONENTS IN DATASET