Slides from my thesis defense at the Data Science Master's Program of the Ukrainian Catholic University (UCU)

1. Aspects of software naturalness through
the generation of IdentifierNames
Oleksandr Zaytsev
Ukrainian Catholic University
oleks@ucu.edu.ua
Stéphane Ducasse
Inria Lille - Nord Europe
stephane.ducasse@inria.fr
Alexandre Bergel
DCC, Universidad de Chile
abergel@dcc.uchile.cl

2. 1. Introduction (problem statement)
2. Data collection
3. What makes code different from English?
4. Translating code to English
5. Evaluation
6. Conclusion
Contents
2
Each dot indicates
a chapter

3. 1. Introduction
3

4. Developers spend
of time
reading
code90%
up to *
* According to Martin (2009)
4

5. Identifier names
* According to Deissenboeck and Pizka (2006)
of source
code
72%
can take
up to
*
5

6. Bad names
A >> b: c
| d |
d := self e f: c; g.
^ self h i: d j
6
*
* It actually works

7. Good names
Collection >> union: aCollection
| set |
set := self asSet addAll: aCollection; yourself.
^ self species withAll: set asArray
7
*
* That’s the same method

8. Problem
1. As software evolves, names become obsolete
2. They need to be maintained and updated
3. This requires complete understanding of the whole
system at all times (impossible for big systems)
➡ We need automated tools to assist humans
8

9. Pharo
9
• Dialect of Smalltalk
• Object-oriented
• Created by Stéphane
and Marcus in 2008
What?
• Outside C-family
• Syntax for kids
• Access to community
Why?

10. Challenge
Suggest descriptive names for Pharo
methods based on their source code
10

11. This is hard because...
We name methods based on how they
should be used, not how they are
implemented
11

12. Software naturalness
• Programming is a form of communication
• Source code written by people has similar regularities
as texts in natural languages
➡ We can apply NLP to source code
• We take this a bit further:
In certain applications and after good preprocessing we
can treat code as a natural language
12

13. Research questions
RQ1. Is source code of Pharo natural enough to allow us
to use its regularities in machine learning models?
RQ2. Does source code of a method contain enough
semantic information to generate a name that
would express the purpose of that method?
13

14. • Collected datasets of Pharo and Java methods from
selected projects
• Compared source code of Pharo and Java to English
• Built a deep learning model that infers the name of
a method from its body
Contributions
14

15. To our knowledge, this is …
• First study of software naturalness outside the
C-family of programming languages
• First application of machine learning and natural
language processing to source code of Smalltalk
for building a tool in Pharo IDE
15

16. 2. Collecting the data
16

17. We have collected…
Pharo methods
132, 046 136, 811
Java methods
17

18. …and compared them to
natural English
sentences from
Brown corpus
57,340 98,552
sentences from
Gutenberg corpus
18

19. Word count in datasets
0
1,750,000
3,500,000
5,250,000
7,000,000
Brown Gutenberg Pharo Java
Natural English
Programming languages
19

20. Tokenizing source code
20
text := string asRopedText.
attributes := (anEditorElement editor text attributesAt: aBrText end)
reject: [ :each | each = self ].
text
attributes: attributes;
foreground: BrGlamorousColors defaultButtonTextColor;
bold.
^ (BlTextElement text: text)
background: (BrGlamorousColors errorBackgroundColor alpha: 0.5);
padding: (BlInsets top: 3 left: 0 bottom: 3 right: 0);
yourself .

21. Tokenizing source code
21
text := string asRopedText.
attributes := (anEditorElement editor text attributesAt: aBrText end)
reject: [ :each | each = self ].
text
attributes: attributes;
foreground: BrGlamorousColors defaultButtonTextColor;
bold.
^ (BlTextElement text: text)
background: (BrGlamorousColors errorBackgroundColor alpha: 0.5);
padding: (BlInsets top: 3 left: 0 bottom: 3 right: 0);
yourself .

22. Tokenizing source code
22
text := string asRopedText .
attributes := ( anEditorElement editor text attributesAt: aBrText end )
reject: [ :each | each = self ] .
text
attributes: attributes ;
foreground: BrGlamorousColors defaultButtonTextColor ;
bold .
^ ( BlTextElement text: text )
background: ( BrGlamorousColors errorBackgroundColor alpha: 0.5 ) ;
padding: ( BlInsets top: 3 left: 0 bottom: 3 right: 0 ) ;
yourself .

23. Tokenizing source code
23
text := string asRopedText .
attributes := ( anEditorElement editor text attributesAt: aBrText end )
reject: [ :each | each = self ] .
text
attributes: attributes ;
foreground: BrGlamorousColors defaultButtonTextColor ;
bold .
^ ( BlTextElement text: text )
background: ( BrGlamorousColors errorBackgroundColor alpha: 0.5 ) ;
padding: ( BlInsets top: 3 left: 0 bottom: 3 right: 0 ) ;
yourself .

24. Tokenizing source code
24
text := string asRopedText .
attributes := ( anEditorElement editor text attributesAt : aBrText end )
reject : [ : each | each = self ] .
text
attributes : attributes ;
foreground : BrGlamorousColors defaultButtonTextColor ;
bold .
^ ( BlTextElement text : text )
background : ( BrGlamorousColors errorBackgroundColor alpha : 0.5 ) ;
padding : ( BlInsets top : 3 left : 0 bottom : 3 right : 0 ) ;
yourself .

25. Tokenizing source code
25
text := string asRopedText .
attributes := ( anEditorElement editor text attributesAt : aBrText end )
reject : [ : each | each = self ] .
text
attributes : attributes ;
foreground : BrGlamorousColors defaultButtonTextColor ;
bold .
^ ( BlTextElement text : text )
background : ( BrGlamorousColors errorBackgroundColor alpha : 0.5 ) ;
padding : ( BlInsets top : 3 left : 0 bottom : 3 right : 0 ) ;
yourself .

26. Tokenizing source code
26
text := string as roped text .
attributes := ( an editor element editor text attributes at : a br text end )
reject : [ : each | each = self ] .
text
attributes : attributes ;
foreground : br glamorous colors default button text color ;
bold .
^ ( bl text element text : text )
background : ( br glamorous colors error background color alpha : 0.5 ) ;
padding : ( bl insets top : 3 left : 0 bottom : 3 right : 0 ) ;
yourself .

27. Tokenizing source code
27
text := string as roped text .
attributes := ( an editor element editor text attributes at : a br text end )
reject : [ : each | each = self ] .
text
attributes : attributes ;
foreground : br glamorous colors default button text color ;
bold .
^ ( bl text element text : text )
background : ( br glamorous colors error background color alpha : 0.5 ) ;
padding : ( bl insets top : 3 left : 0 bottom : 3 right : 0 ) ;
yourself .

28. Tokenizing source code
28
text := string as roped text .
attributes := ( an editor element editor text attributes at : a br text end )
reject : [ : each | each = self ] .
text
attributes : attributes ;
foreground : br glamorous colors default button text color ;
bold .
^ ( bl text element text : text )
background : ( br glamorous colors error background color alpha : <num> ) ;
padding : ( bl insets top : <num> left : <num> bottom : <num> right : <num> ) ;
yourself .

29. Tokenizing source code
29
text := string as roped text .P
attributes := ( an editor element editor text attributes at : a br text end )P
reject : [ : each | each = self ] .P
P
textP
attributes : attributes ;P
foreground : br glamorous colors default button text color ;P
bold .P
P
^ ( bl text element text : text )P
background : ( br glamorous colors error background color alpha : <num> ) ;P
padding : ( bl insets top : <num> left : <num> bottom : <num> right : <num> ) ;P
yourself .

30. 30
text := string as roped text . attributes := ( an editor element editor text
attributes at : a br text end ) reject : [ : each | each = self ] . text attributes
: attributes ; foreground : br glamorous colors default button text color ; bold .
^ ( bl text element text : text ) background : ( br glamorous colors error
background color alpha : <num> ) ; padding : ( bl insets top : <num> left : <num>
bottom : <num> right : <num> ) ; yourself .
Tokenizing source code

31. Words
text := string as roped text . attributes := ( an editor element editor text
attributes at : a br text end ) reject : [ : each | each = self ] . text attributes
: attributes ; foreground : br glamorous colors default button text color ; bold .
^ ( bl text element text : text ) background : ( br glamorous colors error
background color alpha : <num> ) ; padding : ( bl insets top : <num> left : <num>
bottom : <num> right : <num> ) ; yourself .
Tokenizing source code
31
Paragraph with 4 sentences
Sentence

32. 3. What makes code
different from English?
32

33. Limited vocabulary
6,415 50,286
Unique words
in Pharo code
Unique words
in Gutenberg corpus
33

34. Specialized vocabulary
Top 5 words from…
Bible
the
and
of
to
that
34
Reuters
the
of
to
in
and
Pharo
self
a
if
new
assert
Java
get
new
string
if
return

35. 4. Translating source
code to English
35

36. 36
Method name as short
English sentence
sumOfIntegerNumbers — method name
sum of integer numbers — short English sentence

37. • Sequence to sequence neural network
• Attention-based decoder
• Teacher forcing
• GRU cells
Model
37

38. Translating
English to French
38

39. Translating method’s
body to name
39

40. Example of generated
method names
40
self assert: self newNode isComment.
Real name: test is comment
Generated name: test is comment

41. Example of generated
method names
41
aVisitor
visitDraggableInteractreion: self
with: args.
Real name: accept with
Generated name: accept

42. 5. Evaluation
42

43. Idea
• Real method name is ground truth.
• Good model will on average generate
names that are close to real names
43

44. Random baseline*
Idea: Select 3 random words from vocabulary
of words used in method names
44
* Lowest baseline

45. TF-IDF baseline*
Idea: Select 3 words that appear often in
source code of this method and are
very rare in other methods (keywords)
45
* Simple, fast and surprisingly good

46. Evaluation on
validation set
46

47. Evaluation on test set
Exact match score
0%
4%
7%
11%
14%
Random model TF-IDF Our model
47
13.82%

48. Evaluation on test set
Precision score
0%
15%
30%
45%
60%
Random model TF-IDF Our model
48

49. Evaluation on test set
Recall score
0%
10%
20%
30%
40%
Random model TF-IDF Our model
49

50. 6. Conclusion
50

51. RQ1. Can we apply NLP and
ML to source code of Pharo?
51
• We can build tools for Pharo IDE using same
machine learning models that are applied to
natural texts
• Similarly to natural languages, code has regularities
that can be exploited by machine learning models
YES

52. RQ2. Can we infer method
names only from source code?
52
• With no additional features we were able to
generate descriptive method names based on
source code
• We explain this by semantic information carried by
identifier names inside code
YES

53. Questions?
53