Paper title: Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python The paper is published on the technical track of ICSE'22. Abstract: Dynamic languages, such as Python and Javascript, trade static typing for developer flexibility and productivity. Lack of static typing can cause run-time exceptions and is a major factor for weak IDE support. To alleviate these issues, PEP 484 introduced optional type annotations for Python. As retrofitting types to existing codebases is error-prone and laborious, machine learning (ML)-based approaches have been proposed to enable automatic type inference based on existing, partially annotated codebases. However, previous ML-based approaches are trained and evaluated on human-provided type annotations, which might not always be sound, and hence this may limit the practicality for real-world usage. In this paper, we present Type4Py, a deep similarity learning-based hierarchical neural network model. It learns to discriminate between similar and dissimilar types in a high-dimensional space, which results in clusters of types. Likely types for arguments, variables, and return values can then be inferred through the nearest neighbor search. Unlike previous work, we trained and evaluated our model on a \emph{type-checked} dataset and used mean reciprocal rank (MRR) to reflect the performance perceived by users. The obtained results show that Type4Py achieves an MRR of 77.1%, which is a substantial improvement of 8.1% and 16.7% over the state-of-the-art approaches Typilus and TypeWriter, respectively. Finally, to aid developers with retrofitting types, we released a Visual Studio Code extension, which uses Type4Py to provide ML-based type auto-completion for Python.

1. Type4Py
Practical Deep Similarity Learning-Based Type Inference for
Python
Amir M. Mir (PhD Student, TU Delft)
Evaldas Latoskinas (Software Eng., Amazon)
Sebastian Proksch (Assistant Prof., TU Delft)
Georgios Gousios (Associate Prof., TU Delft)
44th International Conference on Software Engineering (ICSE 2022)
May 2022

2. Talk Outline
● Intro
○ Motivation and challenges
○ Related Work
● Proposed Approach (Type4Py)
○ Type hints & Vector embeddings
○ Neural Model
● Dataset
○ Data Augmentation and Type-checking
○ Pre-processing
● Evaluation
○ Research questions
○ Obtained results
● Type4Py in Practice
○ Deployment & Telemetry in VS Code
● Conclusion & Future Work
2

3. Motivation
3

4. Motivation
● PEP 484
○ Optional type annotations
● Adding types manually is laborious
○ Static type inference
○ ML-based type prediction
4

5. SOTA ML-based Type Inference
5
● Typilus (PLDI’20)
○ GNN-based model
● TypeWriter (FSE’20)
○ HNN-based model

6. Current Problems and Challenges
● Suggesting the correct type in the Top-1
○ Developers tend to use the first suggestion by a tool
● Developer-provided type annotations may not always be valid for training and
evaluation
6

7. Type4Py
● Based on Deep Similarity Learning and Hierarchical Neural Network
● A type-checked dataset with 5.1K Python projects and 1.2M type annotations.
● A Visual Studio Code extension, which provides ML-based type auto-completion for
Python.
7

8. Type4Py vs. other ML-based approaches
8

9. Proposed Approach
9

10. Overview of Type4Py
10

11. Type hints/features
● Natural Information
○ Identifiers: names of parameters, functions, and variables
● Code Context
○ Usage of parameters and variables inside a function
● Visible Type Hints (VTH)
○ Recursive extraction of import statements from a module and its transitive
dependencies
11

12. Type hints/features
12
Identifiers
Visible type hints
Code Context

13. Vector Embeddings
● Applying NLP pre-processing techniques
○ tokenization, stop words removal, lemmatization
● Training a Word2Vec model to generate token embeddings:
○ Identifiers & code context
13
file
name
path

14. Neural Model
● Hierarchical Neural Network
○ 2x Recurrent Neural Networks (LSTM cells)
○ Linear layer
14

15. Neural Model
● Triplet loss
○ Learning to discriminate between similar and dissimilar types
15

16. Neural Model
● Using KNN search to suggest types from learned Type Clusters
16

17. Model Training
17
● Used Dropout to avoid overfitting
● Adam optimizer to minimize Triplet loss
● PyTorch’s data parallelism
● A single epoch takes ~4 minutes

18. Dataset
18

19. Code de-duplication
● Used our CD4Py tool
○ TF-IDF to represent source code files a vector
○ KNN search to find clusters of similar duplicate files
○ Similarity is transitive
■ keep a file from each cluster and remove all other identified duplicate files
○ Removed ~400K duplicate files
19

20. Dataset Augmentation
● Pyre, static type inference tool
● Inferring type of variables
20

21. Type Checking
● Developer-provided types rarely type-check
● Performed basic type-checking procedure:
○ Used mypy to type-check 288K source files in the dataset. 64% of them
type-checked.
○ Ignored source code files with syntax errors, which amounts to 64K files
○ Given 40K files w/ type errors, remove one type annotation at a time from a file
■ If it type-checks, include the file.
■ Otherwise, we continue this step up to 10 times.
○ Fixed 16.8K files with type errors
21

22. Dataset Characteristics
● ~201K source code files (11.9M LoC)
○ ~1.24M type annotations in total
● Split dataset by files
○ 70% training, 10% validation, 20% test
22

23. Top-10 frequent types
23

24. Pre-processing
● Excluded trivial functions such as __str__ and __len__
● Excluded Any and None type annotations
● Resolved type aliases, e.g., [] to List.
● Resolved qualified names of type annotations, e.g., np.array
● Rewrote the components of a base type whose nested level is greater than 2
to Any.
○ E.g, List[List[Tuple[int]]] to List[List[Any]]]
24

25. Evaluation
25

26. Evaluation Metrics
26
● Exact match
○ List[str] and List[str] would be an exact match
● Base type match
○ List[str] and List[int] would be a match

27. Evaluation Metrics
27
● Mean Reciprocal Rank (MRR)
● Differentiating between types:
○ Ubiquitous: {str, int, list, bool, float}
○ Common: seen >= 100 times in the train set
○ Rare: seen <100 times in the train set

28. Research Questions
28
● RQ1: What is the general type prediction performance of Type4Py?
● RQ2: How does Type4Py perform while considering different predictions
tasks?
● RQ3: How do each proposed type hint and the size of type vocabulary
contribute to the performance of Type4Py?

29. Type Prediction Performance (RQ1)
29
● Considering Top-1 and exact match, Type4Py achieves 75.8% and
outperforms:
○ Typilus by 9.7%
○ TypeWriter by 19.7%
● Considering MRR, Type4Py obtains 77.1% and outperforms:
○ Typilus by 8.1%
○ TypeWriter by 16.7%
● Type4Py achieves 100% exact match at Top-1 for the ubiquitous types

30. Different prediction tasks (RQ2)
30
● Three prediction tasks
○ Arguments
○ Return
○ Variables
● Considering MRR score
○ Exact match
Arguments
● Type4Py obtains 64.2% vs.
○ 5.5% higher than Typilus
○ 1% higher than TypeWriter
Return
● Type4Py obtains 57.9% vs.
○ 11.9% higher than Typilus
○ 3.7% higher than TypeWriter
Variables
● Type4Py obtains 81.4% vs.
○ 7.7% higher than Typilus

31. Efficacy of type hints (RQ3)
31
● Code context has the most impact on the model’s performance
○ MRR score drops by 7.4%
● By ignoring Visible Type Hints:
○ MRR score for ubiquitous types drops from 100% to 86%.
● By Ignoring identifiers:
○ MRR score drops by 3.3%
● By limiting type vocabulary to 1,000 types:
○ MRR score of rare types drops by 7.2%

32. Type4Py in Practice
32

33. Deployment
33

34. Deployment
34
● Converting pre-trained PyTorch Model to ONNX Model
○ Faster inference
● A small Flask application
○ Predict endpoint (POST)
○ Handles concurrent type prediction requests
○ Nginx as a proxy

35. VSCode Extension
35
https://marketplace.visualstudio.com/items?itemName=saltud.type4py

36. ML-based Type Auto-completion
36

37. VSCode Telemetry
37

38. Telemetry Stats
38
Total no. of prediction requests 93,559
No. of prediction requests from VS Code 65,162
Approx. no. of all users (VS Code) 127
No. of accepted types: 119
No. of rejected types: 49
Avg. processing time of prediction requests 786 ms
No. of failed prediction requests 10,147
NOTE: Telemetry data is considered from Jul. ‘21 to Apr. ‘22

39. Concluding Remarks
39
● Using a type-checked dataset to train and evaluate ML-based type
prediction model
● Considering the MRR metric, Type4Py significantly outperforms Typilus and
TypeWriter
● Type Clusters enables a much larger type vocabulary
● Type4Py cannot synthesize types
● Our VSCode extension helps to improve developers’ productivity

40. Future Work
40
● Exploring pointer networks to suggest new types
● Studying how to improve Type4Py from its usage in VSCode and telemetry
data
● Complenting Type4Py with static type inference

41. Open Science!
41
https://arxiv.org/abs/2101.04470
https://github.com/saltudelft/type4py
Paper
Code
https://doi.org/10.5281/zenodo.4044635
Dataset

42. Contact Info
42
@amir_mir93
s.a.m.mir@tudelft.nl