Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

164 views

Published on

It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application’s inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91% and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw’s potential to improve real development workflows.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

  1. 1. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps IEEETransactions on Software Engineering Journal First Presentation ESEC/FSE’18 Thursday, November 8th, 2018 Kevin Moran, Carlos Bernal Cardenas Michael Curcio, Richard Bonett, & Denys Poshyvanyk
  2. 2. HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING?
  3. 3. HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING?
  4. 4. HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING?
  5. 5. HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING? Make the instantiation of concepts as frictionless as possible
  6. 6. HOW CAN AUTOMATION IMPROVE SOFTWARE ENGINEERING? Make the instantiation of concepts as frictionless as possible Facilitate growing software organically
  7. 7. WE NEED DEVELOPER-CENTRIC AUTOMATION
  8. 8. !7
  9. 9. !7
  10. 10. Video Source: MKBHD
  11. 11. Video Source: MKBHD
  12. 12. Video Source: Google
  13. 13. Video Source: Google
  14. 14. MOBILE DEVELOPMENT CHALLENGES
  15. 15. MOBILE DEVELOPMENT CHALLENGES Highly Competitive App Marketplaces
  16. 16. MOBILE DEVELOPMENT CHALLENGES Noisy Feedback from User Reviews Highly Competitive App Marketplaces
  17. 17. MOBILE DEVELOPMENT CHALLENGES Noisy Feedback from User Reviews Highly Competitive App Marketplaces Change/Fault Prone APIs
  18. 18. MOBILE DEVELOPMENT CHALLENGES Noisy Feedback from User Reviews Highly Competitive App Marketplaces Pressure for Frequent Releases Change/Fault Prone APIs
  19. 19. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  20. 20. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  21. 21. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam Sketch
  22. 22. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  23. 23. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  24. 24. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  25. 25. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  26. 26. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  27. 27. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  28. 28. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  29. 29. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  30. 30. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  31. 31. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  32. 32. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam
  33. 33. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam Prototype GUI Code
  34. 34. RESEARCH PROBLEM UI/UX DesignTeam DevelopmentTeam Prototype GUI Code
  35. 35. EXISTING WORK • Utilizes a combination of unsupervised ComputerVision Techniques to Detect Components • Binary Component Classification (Text or Image) • GUI-hierarchy generated using heuristics Reverse Engineering Mobile App Interfaces* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Neural MachineTranslation of Images to Code✦ ✦Tony Beltramelli, “pix2code: Generating code from a graphical user interface screenshot,” CoRR, vol. abs/1705.07962, 2017. [Online]. Available: http://arxiv.org/abs/1705.07962 • Utilizes a Deep-Learning Machine Translation Approach to convert Images into a DSL and then apps • Only tested on a small synthetic set of apps • Requires a DSL that must be maintained
  36. 36. EXISTING WORK Commercial Solutions
  37. 37. FIRST PRINCIPLES How does a developer translate a GUI mock-up into code?
  38. 38. FIRST PRINCIPLES 1. Detect or identify GUI-components that exist in a mock-up How does a developer translate a GUI mock-up into code?
  39. 39. FIRST PRINCIPLES 1. Detect or identify GUI-components that exist in a mock-up 2. Classify these GUI-components into their constituent types How does a developer translate a GUI mock-up into code?
  40. 40. FIRST PRINCIPLES 1. Detect or identify GUI-components that exist in a mock-up 2. Classify these GUI-components into their constituent types 3. Assemble the GUI-components into a hierarchy and stipulate styles How does a developer translate a GUI mock-up into code?
  41. 41. FIRST PRINCIPLES
  42. 42. FIRST PRINCIPLES Detection
  43. 43. FIRST PRINCIPLES ClassificationDetection
  44. 44. FIRST PRINCIPLES AssemblyClassificationDetection
  45. 45. FIRST PRINCIPLES AssemblyClassificationDetection— —
  46. 46. THE REDRAW FRAMEWORK
  47. 47. PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Input Image
  48. 48. PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Input Image
  49. 49. PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Input Image
  50. 50. PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Input Image
  51. 51. PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes Input Image
  52. 52. PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes GUI-Component Bounding Boxes Input Image
  53. 53. PHASE 1: GUI-COMPONENT DETECTION ComputerVision-Based Detection* *Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15) Canny’s Edge Detection Edge Dilation Contour Bounding Boxes GUI-Component Bounding Boxes Input Image Credits to: Steven Walker & William Hollingsworth
  54. 54. PHASE 1: GUI-COMPONENT DETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch
  55. 55. PHASE 1: GUI-COMPONENT DETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch Marketch Parsing Engine
  56. 56. PHASE 1: GUI-COMPONENT DETECTION Parsing Mock-Up Metadata Input Image SketchSketch MarkEtch Marketch Parsing Engine GUI-Component Bounding Boxes
  57. 57. PHASE 2: GUI-COMPONENT CLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
  58. 58. PHASE 2: GUI-COMPONENT CLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ ConvNets or CNNs have revolutionized the task of image classification
  59. 59. PHASE 2: GUI-COMPONENT CLASSIFICATION Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ ConvNets or CNNs have revolutionized the task of image classification Advanced approaches have reached human-levels of classification accuracy
  60. 60. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner w1 i…w1 j w2 i…w2 j
  61. 61. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  62. 62. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  63. 63. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  64. 64. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  65. 65. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  66. 66. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  67. 67. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  68. 68. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  69. 69. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  70. 70. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  71. 71. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  72. 72. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  73. 73. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  74. 74. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  75. 75. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j
  76. 76. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j Calculate the Error According to Gradient Descent
  77. 77. ASIDE: CNN BASICS … … … Progress Bar TextView Image Button Spinner 1% 85 % 5% 9% 12 14 45 46 78 45 23 12 45 78 90 76 09 67 56 34 09 34 23 67 89 23 12 45 w1 i…w1 j w2 i…w2 j Calculate the Error According to Gradient Descent Backpropagate the Error through the network and update weights
  78. 78. PHASE 2: GUI-COMPONENT CLASSIFICATION How can we extract a large enough dataset of training images?
  79. 79. PHASE 2: GUI-COMPONENT CLASSIFICATION How can we extract a large enough dataset of training images? Mining Software Repositories
  80. 80. PHASE 2: GUI-COMPONENT CLASSIFICATION How can we extract a large enough dataset of training images? Mining Software Repositories Automated Dynamic Analysis +
  81. 81. PHASE 2: GUI-COMPONENT CLASSIFICATION
  82. 82. PHASE 2: GUI-COMPONENT CLASSIFICATION Software Repo Miner
  83. 83. PHASE 2: GUI-COMPONENT CLASSIFICATION Software Repo Miner Image Button Image Image Button Image Button TextView Image Image Button GUI Hierarchies Labeled ImagesExecution Engine
  84. 84. PHASE 2: GUI-COMPONENT CLASSIFICATION ReDraw’s CNN Architecture
  85. 85. PHASE 3: GUI-HIERARCHY CONSTRUCTION
  86. 86. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  87. 87. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  88. 88. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  89. 89. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  90. 90. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  91. 91. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  92. 92. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  93. 93. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  94. 94. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  95. 95. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  96. 96. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination
  97. 97. PHASE 3: GUI-HIERARCHY CONSTRUCTION KNN GUI-Hierarchy Determination Generated GUI-Hierarchy
  98. 98. PHASE 3:APPLICATION ASSEMBLY Labeled GUI Component Hierarchy
  99. 99. PHASE 3:APPLICATION ASSEMBLY GUI-Code Generation Color Quantization & Histogram Analysis Labeled GUI Component Hierarchy
  100. 100. PHASE 3:APPLICATION ASSEMBLY GUI-Code Generation Color Quantization & Histogram Analysis GUI-Code Generation Color Quantization & Histogram AnalysisLabeled GUI Component Hierarchy Functional Prototype Application
  101. 101. PHASE 3:APPLICATION ASSEMBLY Target Component
  102. 102. PHASE 3:APPLICATION ASSEMBLY Target Component 0 20 40 60 80 Component Colors
  103. 103. PHASE 3:APPLICATION ASSEMBLY Target Component 0 20 40 60 80 Component Colors
  104. 104. PHASE 3:APPLICATION ASSEMBLY Target Component 0 20 40 60 80 Component Colors
  105. 105. EMPIRICAL STUDY
  106. 106. EMPIRICAL STUDY • RQ1: CNN Accuracy? 

  107. 107. EMPIRICAL STUDY • RQ1: CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 

  108. 108. EMPIRICAL STUDY • RQ1: CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 
 • RQ3: Visual Similarity? 

  109. 109. EMPIRICAL STUDY • RQ1: CNN Accuracy? 
 • RQ2: Hierarchy Similarity? 
 • RQ3: Visual Similarity? 
 • RQ4: Industrial Applicability?
  110. 110. STUDY 1: CNN CLASSIFIER ACCURACY Context 8,655 Applications 14,382 Unique Screens 191,300 Labeled GUI Component Images ReDraw CNN Classifier Supervised BoVW Baseline 15 of the most widely used GUI-componentTypes
  111. 111. STUDY 1: CNN CLASSIFIER ACCURACY Data Augmentation Automatically Generated Synthetic Applications Automatically Shift Color of Components through Hue Shifting
  112. 112. STUDY 1: CNN CLASSIFIER ACCURACY GUI Type Total # (C) Tr (O) Tr (O+S) Valid Test TextView 99,200 74,087 74,087 15,236 9,877 ImageView 53,324 39,983 39,983 7,996 5,345 Button 16,007 12,007 12,007 2,400 1,600 ImageButton 8,693 6,521 6,521 1,306 866 EditText 5,643 4,230 5,000 846 567 CheckedTextView 3,424 2,582 5,000 505 337 CheckBox 1,650 1,238 5,000 247 165 RadioButton 1,293 970 5,000 194 129 ProgressBar 406 307 5,000 60 39 SeekBar 405 304 5,000 61 40 NumberPicker 378 283 5,000 57 38 Switch 373 280 5,000 56 37 ToggleButton 265 199 5,000 40 26 RatingBar 219 164 5,000 33 22 Spinner 20 15 5,000 3 2 Total 191,300 143,170 187,598 29,040 19,090
  113. 113. STUDY 1:RESULTS Confusion Matrix for BoVW Baseline Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 59% 4% 9% 1% 6% 2% 8% 6% 0% 1% 2% 0% 1% 0% 2% IV 5345 4% 51% 4% 1% 2% 11% 2% 18% 1% 1% 3% 0% 2% 0% 2% Bt 1600 6% 6% 59% 1% 5% 4% 7% 4% 0% 1% 1% 0% 0% 3% 1% S 37 5% 0% 3% 65% 0% 0% 5% 22% 0% 0% 0% 0% 0% 0% 0% ET 567 6% 2% 4% 1% 62% 1% 4% 15% 0% 0% 1% 0% 0% 4% 1% IBt 866 2% 16% 3% 0% 2% 61% 1% 9% 1% 1% 2% 0% 1% 0% 3% CTV 337 3% 1% 7% 1% 3% 0% 81% 1% 0% 0% 2% 0% 0% 0% 2% PB 41 0% 24% 2% 0% 2% 5% 2% 54% 0% 0% 2% 2% 2% 0% 2% RB 22 0% 5% 0% 0% 0% 0% 0% 27% 68% 0% 0% 0% 0% 0% 0% TBt 26 7% 7% 19% 0% 0% 0% 11% 15% 0% 33% 0% 0% 0% 0% 7% CB 165 4% 2% 3% 1% 2% 1% 2% 12% 1% 0% 72% 0% 0% 0% 1% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 0% 5% 0% 0% 0% 0% 0% 18% 3% 0% 5% 0% 68% 0% 3% NP 40 3% 0% 5% 0% 3% 0% 5% 0% 0% 0% 0% 0% 0% 84% 0% RBt 129 6% 3% 5% 1% 3% 0% 6% 18% 0% 1% 1% 0% 1% 0% 55%
  114. 114. STUDY 1:RESULTS Confusion Matrix for BoVW Baseline Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 59% 4% 9% 1% 6% 2% 8% 6% 0% 1% 2% 0% 1% 0% 2% IV 5345 4% 51% 4% 1% 2% 11% 2% 18% 1% 1% 3% 0% 2% 0% 2% Bt 1600 6% 6% 59% 1% 5% 4% 7% 4% 0% 1% 1% 0% 0% 3% 1% S 37 5% 0% 3% 65% 0% 0% 5% 22% 0% 0% 0% 0% 0% 0% 0% ET 567 6% 2% 4% 1% 62% 1% 4% 15% 0% 0% 1% 0% 0% 4% 1% IBt 866 2% 16% 3% 0% 2% 61% 1% 9% 1% 1% 2% 0% 1% 0% 3% CTV 337 3% 1% 7% 1% 3% 0% 81% 1% 0% 0% 2% 0% 0% 0% 2% PB 41 0% 24% 2% 0% 2% 5% 2% 54% 0% 0% 2% 2% 2% 0% 2% RB 22 0% 5% 0% 0% 0% 0% 0% 27% 68% 0% 0% 0% 0% 0% 0% TBt 26 7% 7% 19% 0% 0% 0% 11% 15% 0% 33% 0% 0% 0% 0% 7% CB 165 4% 2% 3% 1% 2% 1% 2% 12% 1% 0% 72% 0% 0% 0% 1% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 0% 5% 0% 0% 0% 0% 0% 18% 3% 0% 5% 0% 68% 0% 3% NP 40 3% 0% 5% 0% 3% 0% 5% 0% 0% 0% 0% 0% 0% 84% 0% RBt 129 6% 3% 5% 1% 3% 0% 6% 18% 0% 1% 1% 0% 1% 0% 55% Overall Accuracy 65%
  115. 115. STUDY 1:RESULTS Confusion Matrix for ReDraw CNN Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 94% 3% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IV 5345 5% 93% 1% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% Bt 1600 11% 6% 81% 0% 1% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% S 37 5% 3% 0% 87% 0% 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% ET 567 14% 3% 2% 0% 81% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IBt 866 4% 23% 1% 0% 0% 72% 0% 0% 0% 0% 0% 0% 0% 0% 0% CTV 337 7% 0% 0% 0% 0% 0% 93% 0% 0% 0% 0% 0% 0% 0% 0% PB 41 15% 29% 0% 0% 0% 0% 0% 56% 0% 0% 0% 0% 0% 0% 0% RB 22 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% 0% 0% 0% TBt 26 19% 22% 7% 0% 0% 0% 0% 0% 0% 52% 0% 0% 0% 0% 0% CB 165 12% 7% 0% 0% 1% 0% 0% 0% 0% 0% 81% 0% 0% 0% 0% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 10% 13% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 78% 0% 0% NP 40 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 95% 0% RBt 129 4% 3% 2% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 89%
  116. 116. STUDY 1:RESULTS Confusion Matrix for ReDraw CNN Total TV IV Bt S ET IBt CTV PB RB TBt CB Sp SB NP RBt TV 9877 94% 3% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IV 5345 5% 93% 1% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% Bt 1600 11% 6% 81% 0% 1% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% S 37 5% 3% 0% 87% 0% 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% ET 567 14% 3% 2% 0% 81% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% IBt 866 4% 23% 1% 0% 0% 72% 0% 0% 0% 0% 0% 0% 0% 0% 0% CTV 337 7% 0% 0% 0% 0% 0% 93% 0% 0% 0% 0% 0% 0% 0% 0% PB 41 15% 29% 0% 0% 0% 0% 0% 56% 0% 0% 0% 0% 0% 0% 0% RB 22 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% 0% 0% 0% TBt 26 19% 22% 7% 0% 0% 0% 0% 0% 0% 52% 0% 0% 0% 0% 0% CB 165 12% 7% 0% 0% 1% 0% 0% 0% 0% 0% 81% 0% 0% 0% 0% Sp 2 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 100% 0% 0% 0% SB 39 10% 13% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 78% 0% 0% NP 40 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 95% 0% RBt 129 4% 3% 2% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 89% Overall Accuracy 91%
  117. 117. STUDY 2:GUI HIERARCHY CONSTRUCTION Context ReDraw Pix2Code32 Applications 2-3 Unique Screens (Not used inTraining the CNN) REMAUI (Re-implementation)
  118. 118. STUDY 2:GUI HIERARCHY CONSTRUCTION Target GUI Hierarchy Prototype GUI-Hierarchy Edit Distance Calculation # of Insertions - 0 < Insertion Penalty < 1 # of Deletions - 0 < Deletion Penalty < 1 # of Substitutions - 0 < Substitution Penalty < 1
  119. 119. STUDY 2:GUI HIERARCHY CONSTRUCTION Target GUI Hierarchy Prototype GUI-Hierarchy Edit Distance Calculation # of Insertions - 0 < Insertion Penalty < 1 # of Deletions - 0 < Deletion Penalty < 1 # of Substitutions - 0 < Substitution Penalty < 1
  120. 120. STUDY 2: RESULTS Hierarchy Edit Distance to GroundTruth A) Insertion Edits B) Deletion Edits C) Substitution Edits Fraction of PenaltyFraction of PenaltyFraction of Penalty EditDistance EditDistance EditDistance
  121. 121. STUDY 3:VISUAL SIMILARITY OF APPS Context ReDraw Pix2Code32 Applications 2-3 Unique Screens (Not used inTraining the CNN) REMAUI (Re-implementation)
  122. 122. STUDY 3:VISUAL SIMILARITY OF APPS Target App Screenshot Generated App Screenshot Pixel-Based Mean Average Error (MAE) Mean Squared Error (MSE)
  123. 123. STUDY 3:VISUAL SIMILARITY OF APPS Target App Screenshot Generated App Screenshot Pixel-Based Mean Average Error (MAE) Mean Squared Error (MSE)
  124. 124. STUDY 3: RESULTS ReDraw−MU(MAE) ReDraw−CV(MAE) REMAUI(MAE) Pix2Code(MAE) ReDraw−MU(MSE) ReDraw−CV(MSE) REMAUI(MSE) Pix2Code(MSE) 0.0 0.2 0.4 0.6 0.8 MSE and MAE compared to GroundTruth Screenshots Mann Whitney U-Test & Cliff’s Delta MAE MSE Test p-Value d ReDrawMU vs. ReDrawCV 0.835 0.02 (Small) ReDrawMU vs. REMAUI 0.542 0.06 (Small) ReDrawMU vs. pix2code <0.0002 -0.34 (Medium) pix2code vs. ReDrawCV <0.0001 0.35 (Medium) pix2code vs. REMAUI <0.0001 0.39 (Medium) REMAUI vs. ReDrawCV 0.687 -0.04 (Small) Test p-Value d ReDrawMU vs. ReDrawCV 0.771 0.03 (Small) ReDrawMU vs. REMAUI <0.0001 0.45 (Medium) ReDrawMU vs. pix2code <0.003 -0.27 (Small) pix2code vs. ReDrawCV <0.002 0.28 (Small) pix2code vs. REMAUI <0.0001 0.61 (Large) REMAUI vs. ReDrawCV <0.0001 -0.42 (Medium)
  125. 125. APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (MockUp) Textgram
  126. 126. APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (MockUp) Zillow
  127. 127. APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (MockUp) Bank of America
  128. 128. APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (CV) Giphy
  129. 129. APPLICATIONS GENERATED BY REDRAW A) Original Application B) ReDraw App (CV) Netflix
  130. 130. STUDY4: DEVELOPER UTILITY Context Mobile Front-End Developer Mobile Designer 9 Screens from 8 Popular Apps Mobile Researcher
  131. 131. STUDY4: RESULTS
  132. 132. STUDY4: RESULTS “It’s a good starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]”
  133. 133. STUDY4: RESULTS “It’s a good starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]” “There are going to be edge cases for different layouts, but these are easily fixed after the fact”
  134. 134. STUDY4: RESULTS “It’s a good starting point... From a development standpoint, the thing I would appreciate most is getting a lot of the boilerplate code done [automatically]” “There are going to be edge cases for different layouts, but these are easily fixed after the fact” “The key thing is fast iteration.A developer could generate the initial view [using ReDraw], clean up the layouts, and have a working app. In the future, if a designer could upload a screenshot, and without any other intervention [ReDraw] could update the [existing] xml this would be ideal.”
  135. 135. https://www.android-dev-tools.com/redraw/
  136. 136. https://www.android-dev-tools.com/redraw/
  137. 137. https://www.android-dev-tools.com/redraw/
  138. 138. Carlos Bernal Cardenas Michael Curcio Richard Bonett Denys Poshyvanyk
  139. 139. Any Questions? Thank you! Kevin Moran Ph.D. Post-Doctoral Fellow kpmoran@cs.wm.edu http://www.cs.wm.edu/~kpmoran
  140. 140. ADDITIONAL SLIDES
  141. 141. “The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation.” – Fredrick Brooks, No Silver Bullet – Essence and Accident in Software Engineering (1986)
  142. 142. DISTRIBUTION OF GUI COMPONENTS IN DATASET

×