This document presents Screen2Vec, a technique for generating semantic embeddings of GUI screens and components using text, visual design, layout patterns, and app metadata. It has a two-level architecture that learns embeddings at the component and screen levels. The technique is self-supervised and does not require labeled data. It was trained on the RICO dataset and can be used for tasks like nearest neighbor retrieval and representing mobile tasks. Sample results showed Screen2Vec generated more comprehensive representations compared to baselines using only text or layout.

1. Screen2Vec: Semantic Embedding of GUI
Screens and GUI Components
Toby Li, Lindsay Popowski, Tom M Mitchell, Brad A. Myers
2021 CHI Conference on Human Factors in Computing Systems

2. Background
• Existing approaches of representing GUI screens are limited
 Capturing only text on the screen
• Missing information encoded in the layout and design pattern
 Focusing on the visual design patterns and GUI layouts
• Not capturing the content in the GUI
• Prior approaches use supervised learning with large datasets for specific task
objectives
 Requiring labeling efforts
 Inapplicable in different downstream tasks
1
Semantic representations of GUI screens and components

3. Contribution
• Presenting a self-supervised technique, not requiring human-labeled data
• Generating more comprehensive semantic embeddings of GUI screens and
components using
 Textual content
 Visual design
 Layout patterns
 App meta-data
• Training an open-sourced GUI embedding model using Screen2Vec with RICO
dataset
• Providing sample downstream tasks such as
 Nearest neighbor retrieval
 Composability-based retrieval
 Representing mobile tasks
2

4. Architecture of Screen2Vec
3
GUI Component level
GUI Screen Level
• Two-level architecture

5. Architecture of Screen2Vec
• Input
 768-dimensional embedding vector of the text label of the GUI component
• Encoded using a pre-trained Sentence-BERT
 6-dimensional class embedding vector
• Representing the class type of the GUI component
• Optimizing weights in the class embeddings and weights in the linear layer (text + class)
• Output
 768-dimensional embedding vector
4
GUI Component Level

6. Architecture of Screen2Vec
1) Collection of GUI component embedding vector
 Combined into a 768-dimensional vector using RNN
2) 64-dimensional layout embedding vector
 Encoding the screen’s visual layout
3) 768-dimensional embedding vector of the textual App Store description
 Encoded with a pre-trained Sentence-BERT model
• GUI(1) and layout(2) vectors are combined using a linear layer  768-dimensional
embedding vector
• After training, description(3) vector is concatenated  1536-dimensional embedding
vector
• Weights of RNN and weights of the linear layer trained on a Continuous Bag of Word
prediction 5
GUI Screen Level

7. Dataset
• RICO Dataset
 Containing interaction traces on 66,261 unique GUI screens
 From 9,384 free Android apps
• Specifics
 Each dataset with a screenshot image
 Screen’s “view hierarchy” (e.g., DOM tree in HTML) in a JSON file
• Each node including
• Class type
• Textual content
• Location as the bounding box on the screen
• Properties such as whether it is clickable, focused, or scrollable
 Each interaction trace represented as a sequence of GUI screens
• Which location is clicked or swiped
6

8. Implementation Details
• Encoding 26 class categories into a vector space
• Mapping each of the categories into a continuous 6-dimensional vector
• Optimizing embedding vector value by training GUI component prediction task
 Categories semantically similar, close in the vector space
7
GUI Class Type Embeddings

9. Implementation Details
• Defining the context of a component as its 16 nearest components
• Measures of screen distance for determining the context
 Euclidean : straight-line minimal distance on the screen
• In pixel
 Hierarchical : distance between 2 GUI components on the hierarchical view tree
• Parent and children : 1
8
GUI Component Context

10. Implementation Details
• Combining multiple vectors into a lower-dimension vector
• GUI component level
 Concatenating 768-dimension with 6-dimension
 Shrinking down to 768-dimension
 Creating 774 x 768 weights
• GUI screen level
 Combining 768-dimension and 64-dimension
 Producing 768-dimension for screen content and layout
9
Linear Layer

11. Implementation Details
• Use a pre-trained Sentence-BERT language model
 Using SNLI and Multi-Genre NLI datasets with mean-pooling
• Encoding the text label of description to 768-dimensional vectors
• Deriving semantically similar sentences and phrases
10
Text Embeddings

12. Implementation Details
• Extracting the layout from a screenshot
• Differentiating between text and non-text GUI components
• Using autoencoder to encode each image into 64-dimensional embedding vector
• Encoder’s input dimension : 11,200
• Two hidden layers of 2,048 and 256
• Applying RLU to eliminate negative input
• Loss determined by MSE
11
Layout Embeddings

13. Implementation Details
• Combining embedding vectors of multiple GUI components
• GUI components embeddings fed into the RNN
 In the pre-order traversal order of hierarchy tree
• Starting with hidden state of zero, fed into a linear layer with 𝑛 − 1 𝑡ℎ output
12
GUI Embedding Combining Layer

14. Training Configuration
• Training: 90% of the data; validation: 10%
• Cross entropy loss function with Adam optimizer
• Learning rate: 0.001; batch size: 256
• GUI component model: 120 epochs; GUI screen model: 80-120 epochs
• Total loss
 Component
• Total Loss = Loss(text prediction) + Loss(class type prediction)
 Screen
• Negative sampling
• Prediction compared to the correct screen and a sample of negative data
• Random sampling of other screens with size 128 on the same app
• To differentiate different screens on the same app
13

15. Baselines
• Text Embedding Only (similar textual context)
 Screen embedding method used in SOVITE
 Computed by averaging the text embedding vectors for all the text in the screen
• Layout Embedding Only (similar layout)
 Screen embedding method used in the original RICO paper
 Computed by the layout autoencoder to represent the screen
• Visual Embedding Only (similar visual)
 Direct screen shot of image instead of layout
 Inspired by VASTA, Sikuli, and HILC
14

16. Results
• Predicting each GUI screen in all the GUI interaction traces in the RICO dataset using its
context
 3 versions to compare
• EUCLIDEAN with locations of GUI components and the screen layouts
• HIERARCHICAL with above spatial info
• EUCLIDEAN without spatial info
15

17. Sample Downstream Tasks
• The main purpose is to produce distributed vector representations that encode useful
semantic, layout, design properties
• Compare similarity between the nearest neighbor results by different models
Methods
• Select 50 screens from apps and app domains
• Retrieve top-5 most similar screens using each of 3 models
• 79 Mechanical Turk workers participated
• Each worker saw top-5 most similar screens of 5 source screens produced by 3 models
• Questionnaires include followings
 (1) App similarity (2) Screen type similarity (3) Content similarity
16
Nearest Neighbors

18. Sample Downstream Tasks
Results
• The differences between the mean ratings of the Screen2Vec model and both TextOnly
and LayoutOnly model are significant (non-parametric Mann-Whitney U test)
• Retrieve top-5 most similar screens using each of 3 models
17
Nearest Neighbors

19. Sample Downstream Tasks
Observation
• Screen2Vec generate more comprehensive
representations
 “Request ride” in Lyft
• “Get direction” in Uber Driver
• “Select navigation type” in Waze app
• “Request ride” in Free Now
 MapView taking majority
 All feature a menu/information card at the
bottom 1/3 – 1/4
• TextOnly generated results are semantically
similar to “payment”
• LayoutOnly generated results show lower score
in the content and app-context similarity
18
Nearest Neighbors

20. Sample Downstream Tasks
Word2Vec
• “Man is to woman as brother is to sister”
• (brother - man + woman) results in an
embedding vector representing sister
Screen2Vec
• Marriott app ’s “hotel booking” screen +
(Cheapoair app’s “search result” screen
− Cheapoair app’s “hotel booking”
screen))
• The top result is the “search result”
screen in the Marriott app
19
Embedding Composability

21. Sample Downstream Tasks
• Preliminary evaluation on the effectiveness of embedding mobile tasks as
sequences of Screen2Vec screen embedding vectors
• Recording scripts of completing 10 common smartphone tasks
• Representing each task as the average of Screen2Vec vectors
• Querying for the nearest neighbor within 20 task variations and get 18/20
accuracy
 TextOnly : 14/20 accuracy
20
Screen Embedding Sequences for Representing Mobile Tasks

22. Potential Application
• Designers query for example designs that display similar content or screens in
apps of a similar domain
• Composability helps to find a specific page for the app
 Suppose a designer searches for checkout page for app A
 A’s order page + (App B’s checkout page – App B’s order page
• LayoutGAN can generate realistic GUI layouts based on user-specified
constraints
 Applying Screen2Vec to incorporate the semantics of GUIs and the context
of user interaction
21

23. Limitation
• Only trained and test on Android app GUIs
• RICO dataset
 contains interaction traces within single apps  need to generalize multiple app
 Does not contain paid apps
• Screen2Vec does not encode the semantics of graphic icons that have no textual
information
22