The document presents data from a study involving 6,000 software projects at Saarland University, focusing on the challenges of formal specifications in programming and the properties of various programming languages. It highlights the significant number of PhD advisors and students involved, alongside details about graduates, stipends, and the complexity of verifying secure protocols. The findings also cover the use of temporal properties and the prevalence of defects or code smells in software development.

1. Learning from
6,000 Projects
Mining Models in the Large
Andreas Zeller
Saarland University

5. Saarbrücken

6. Saarbrücken

7. Saarbrücken

8. Saarbrücken

9. Saarbrücken

10. Saarbrücken

11. Saarbrücken
®
Visual
Computing
Institute

12. Saarbrücken

13. Some numbers

14. Some numbers
• ~70 PhD advisors in computer science

15. Some numbers
• ~70 PhD advisors in computer science
• ≥ 300 PhD students in computer science

16. Some numbers
• ~70 PhD advisors in computer science
• ≥ 300 PhD students in computer science
• ~60 new PhD graduates per year

17. Some numbers
• ~70 PhD advisors in computer science
• ≥ 300 PhD students in computer science
• ~60 new PhD graduates per year
• ~60 new MSc graduates per year

18. Some numbers
• ~70 PhD advisors in computer science
• ≥ 300 PhD students in computer science
• ~60 new PhD graduates per year
• ~60 new MSc graduates per year
• 800–1400 € per month as a PhD stipend
(+ laptop & office • starting right after BSc • all courses in English)

19. Two Graduates
Michael Backes Andrej Rybalchenko
TR35 in 2009 TR35 in 2010

20. Michael Backes Andrej Rybalchenko

21. secure protocols Andrej Rybalchenko

22. secure protocols loop termination

23. secure protocols loop termination
hard to verify

24. secure protocols loop termination
hard to verify

25. information ow
secure protocols loop termination
hard to verify

26. information ow liveness
secure protocols loop termination
hard to verify

27. buffer over ow
information ow liveness
secure protocols loop termination
hard to verify

28. buffer over ow resource leaks
information ow liveness
secure protocols loop termination
hard to verify

29. buffer over ow resource leaks
information ow liveness
secure protocols loop termination
easy to specify
hard to verify

30. hard to specify

31. sorting
hard to specify

32. ∀i ∈ {0, . . . , |x |} : x [i] < x [i + 1]
|x| = |x |
∀i ∈ {0, . . . , |x|} : ιi ∈ {0, . . . , |x |} : x[i] = x [i ]
∀i ∈ {0, . . . , |x |} : ιi ∈ {0, . . . , |x|} : x [i ] = x[i]
hard to specify

33. ∀i ∈ {0, . . . , |x |} : x [i] < x [i + 1]
|x| = |x |
∀i ∈ {0, . . . , |x|} : ιi ∈ {0, . . . , |x |} : x[i] = x [i ]
∀i ∈ {0, . . . , |x |} : ιi ∈ {0, . . . , |x|} : x [i ] = x[i]
easy to verify
hard to specify

34. is-sorted(x ) ∧ is-permutation(x, x )
still hard to specify

36. microsoft word

37. microsoft word
travel booking

38. microsoft word
travel booking
airplane control

39. microsoft word mobile phones
travel booking
airplane control

40. microsoft word mobile phones
travel booking operating systems
airplane control

41. microsoft word mobile phones
travel booking operating systems
airplane control banking systems

42. microsoft word mobile phones
travel booking operating systems
airplane control banking systems
hard to specify

43. microsoft word mobile phones
travel booking operating systems
airplane control banking systems
easy to verify
hard to specify

44. hard to specify

45. hard to specify
new language • duplicate effort • can’t abstract from details

46. speci cation crisis

47. mine speci cations

48. mine speci cations

49. mine speci cations
from 6,000 projects

50. Speci cations
∀i ∈ {0, . . . , |x |} : x [i] < x [i + 1]
|x| = |x |
∀i ∈ {0, . . . , |x|} : ιi ∈ {0, . . . , |x |} : x[i] = x [i ]
∀i ∈ {0, . . . , |x |} : ιi ∈ {0, . . . , |x|} : x [i ] = x[i]
pre- and postconditions

51. Speci cations
auth()!
<init>()
openPort()
socket: null socket: ¬null
state: NOT_CON state: PLAIN
quit() auth()
socket: ¬null
state: AUTH
nite state models

52. OP-Miner

53. OP-Miner
Program

54. OP-Miner
Usage Models
Program iter.hasNext () iter.next ()

55. OP-Miner
Usage Models Temporal Properties
hasNext ≺ next
Program hasNext ≺ hasNext
iter.hasNext () iter.next () next ≺ hasNext
next ≺ next

56. OP-Miner
Usage Models Temporal Properties
hasNext ≺ next
Program hasNext ≺ hasNext
iter.hasNext () iter.next () next ≺ hasNext
next ≺ next
Patterns
hasNext ≺ next
hasNext ≺ hasNext

57. OP-Miner
Usage Models Temporal Properties
hasNext ≺ next
Program hasNext ≺ hasNext
iter.hasNext () iter.next () next ≺ hasNext
next ≺ next
Anomalies Patterns
hasNext ≺ next
✓ hasNext ≺ hasNext hasNext ≺ next
hasNext ≺ next hasNext ≺ hasNext
✗ hasNext ≺ hasNext

58. OP-Miner
Usage Models Temporal Properties
hasNext ≺ next
Program hasNext ≺ hasNext
iter.hasNext () iter.next () next ≺ hasNext
next ≺ next
Anomalies Patterns
hasNext ≺ next
✓ hasNext ≺ hasNext hasNext ≺ next
hasNext ≺ next hasNext ≺ hasNext
✗ hasNext ≺ hasNext

60. public Stack createStack () {
Random r = new Random ();
int n = r.nextInt ();
Stack s = new Stack ();
int i = 0;
while (i < n) {
s.push (rand (r));
i++;
}
s.push (-1);
return s;
}

61. public Stack createStack () {
Random r = new Random ();
int n = r.nextInt ();
Stack s = new Stack ();
int i = 0;
while (i < n) {
s.push (rand (r));
i++;
}
s.push (-1);
return s;
}

62. Random r = new Random ();
public Stack createStack () {
Random r = new Random ();
int n = r.nextInt ();
Stack s = new Stack ();
int i = 0;
while (i < n) {
s.push (rand (r));
i++;
}
s.push (-1);
return s;
}

63. Random r = new Random ();
public Stack createStack () {
Random r = new Random ();
int n = r.nextInt (); int n = r.nextInt ();
Stack s = new Stack ();
int i = 0; Stack s = new Stack ();
while (i < n) {
s.push (rand (r));
i++; int i = 0;
}
s.push (-1);
return s;
}

64. Random r = new Random ();
public Stack createStack () {
Random r = new Random ();
int n = r.nextInt (); int n = r.nextInt ();
Stack s = new Stack ();
int i = 0; Stack s = new Stack ();
while (i < n) {
s.push (rand (r));
i++; int i = 0;
}
s.push (-1); i < n
return s; i++;
}
s.push (rand (r));

65. Random r = new Random ();
public Stack createStack () {
Random r = new Random ();
int n = r.nextInt (); int n = r.nextInt ();
Stack s = new Stack ();
int i = 0; Stack s = new Stack ();
while (i < n) {
s.push (rand (r));
i++; int i = 0;
}
s.push (-1); i < n i < n
return s; i++;
}
s.push (-1); s.push (rand (r));

66. Random r = new Random ();
public Stack createStack () {
Random r = new Random ();
int n = r.nextInt (); int n = r.nextInt ();
Stack s = new Stack ();
int i = 0; Stack s = new Stack ();
while (i < n) {
s.push (rand (r));
i++; int i = 0;
}
s.push (-1); i < n i < n
return s; i++;
}
s.push (-1); s.push (rand (r));

67. Random r = new Random ();
int n = r.nextInt ();
Stack s = new Stack ();
int i = 0;
i < n i < n
i++;
s.push (-1); s.push (rand (r));

68. Stack s = new Stack ();
s.push (-1); s.push (rand (r));

69. s.<init>()
s.push (_)
s.push (_)

70. Random r = new Random ();
int n = r.nextInt ();
Stack s = new Stack ();
int i = 0;
i < n i < n
i++;
s.push (-1); s.push (rand (r));

71. Random r = new Random ();
int n = r.nextInt ();
s.push (rand (r));

72. r.<init> ()
r.nextInt ()
Utils.rand (r)

73. OP-Miner
Usage Models Temporal Properties
hasNext ≺ next
Program hasNext ≺ hasNext
iter.hasNext () iter.next () next ≺ hasNext
next ≺ next
Anomalies Patterns
hasNext ≺ next
✓ hasNext ≺ hasNext hasNext ≺ next
hasNext ≺ next hasNext ≺ hasNext
✗ hasNext ≺ hasNext

74. OP-Miner
Usage Models Temporal Properties
hasNext ≺ next
Program hasNext ≺ hasNext
iter.hasNext () iter.next () next ≺ hasNext
next ≺ next
Anomalies Patterns
hasNext ≺ next
✓ hasNext ≺ hasNext hasNext ≺ next
hasNext ≺ next hasNext ≺ hasNext
✗ hasNext ≺ hasNext

75. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
Methods

76. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
get()
Methods

77. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
get()
Methods
open()

78. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
get()
Methods
open()
hello()

79. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
get()
Methods
open()
hello()
parse()

80. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
get()
Methods
open()
hello()
parse()

81. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
get()
Methods
open()
hello()
parse()

82. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
Pattern
get()
Methods
open()
hello()
parse()

83. Methods vs. Properties
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
Pattern
get()
Methods
open()
hello()
parse()
Support

84. Discovering Anomalies
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
get()
Methods
open()
hello()
parse()

85. Discovering Anomalies
Temporal Properties
start ≺ lock ≺ eof ≺
stop unlock close
Anomaly
get() ✘
Methods
open()
hello()
parse()

86. AspectJ

87. for (Iterator iter = itdFields.iterator();
iter.hasNext();) {
...
for (Iterator iter2 = worthRetrying.iterator();
iter.hasNext();) {
...
}
}

88. for (Iterator iter = itdFields.iterator();
iter.hasNext();) {
...
for (Iterator iter2 = worthRetrying.iterator();
iter.hasNext();) {
... should be iter2
}
}

89. public void visitNEWARRAY (NEWARRAY o) {
byte t = o.getTypecode ();
if (!((t == Constants.T_BOOLEAN) ||
(t == Constants.T_CHAR) ||
...
(t == Constants.T_LONG))) {
constraintViolated (o, "(...) '+t+' (...)");
}
}

90. public void visitNEWARRAY (NEWARRAY o) {
byte t = o.getTypecode ();
if (!((t == Constants.T_BOOLEAN) ||
(t == Constants.T_CHAR) ||
...
(t == Constants.T_LONG))) {
constraintViolated (o, "(...) '+t+' (...)");
}
} should be double quotes

91. Name internalNewName (String[] identifiers)
...
for (int i = 1; i < count; i++) {
SimpleName name = new SimpleName(this);
name.internalSetIdentifier(identifiers[i]);
...
}
...
}

92. Name internalNewName (String[] identifiers)
...
for (int i = 1; i < count; i++) {
SimpleName name = new SimpleName(this);
name.internalSetIdentifier(identifiers[i]);
...
} should stay as is
...
}

93. public String getRetentionPolicy ()
{
...
for (Iterator it = ...; it.hasNext();)
{
... = it.next();
...
return retentionPolicy;
}
...
}

94. public String getRetentionPolicy ()
{
...
for (Iterator it = ...; it.hasNext();)
{
... = it.next();
...
return retentionPolicy;
}
... should be xed
}

96. 44% of violations
are defects or code smells

97. mine speci cations

98. mine speci cations
across thousands of projects

99. Wisdom of the crowds
Francis
Galton
Nein, links auch nicht

100. Wisdom of the crowds
Francis
Galton
Nein, links auch nicht

102. lightweight parsing

103. Target Languages
Java C++ C PHP Javascript

104. Target Languages
Java C++ C PHP Javascript
Similar syntax
{...} ; foo()

105. Target Languages
Java C++ C PHP Javascript
Similar syntax
{...} ; foo()
Similar keywords
while if switch return

106. Lightweight Parser
Abstract Temporal
Source Code
Representation Properties

107. Lightweight Parser
Abstract Temporal
Source Code
Representation Properties
}
language-independent
lightweight parsing

108. Abstract Temporal
Source Code
Representation Properties

109. Abstract Temporal
Source Code
Representation Properties
int j;
int fA;
int fB = open(“newFile”);
fA = open(“myFile”);
j = 7;
while (j > 3) {
read(fA);
write(fB, “Hello”);
j--;
}
close(fA);
close(fB);

110. Abstract Temporal
Source Code
Representation Properties
int j; fB: open(CONST)
int fA;
int fB = open(“newFile”);
fA: open(CONST)
fA = open(“myFile”);
j = 7;
while (j > 3) { Loop:
read(fA); read(fA)
write(fB, “Hello”); write(fB, CONST)
j--;
}
close(fA)
close(fA);
close(fB); close(fB)

111. Abstract Temporal
Source Code
Representation Properties
fB: open(CONST)
fA: open(CONST)
Loop:
read(fA)
write(fB, CONST)
close(fA)
close(fB)

112. Abstract Temporal
Source Code
Representation Properties
fA: open(CONST)
fB: open(CONST)
read(fA)
fA: open(CONST)
close(fA)
Loop:
read(fA)
write(fB, CONST)
close(fA)
close(fB)

113. Abstract Temporal
Source Code
Representation Properties
fA: open(CONST)
fB: open(CONST)
read(fA)
fA: open(CONST)
close(fA)
Loop:
read(fA)
write(fB, CONST)
fB: open(CONST)
close(fA)
write(fB, CONST)
close(fB) close(fB)

114. Abstract Temporal
Source Code
Representation Properties
fA: open(CONST)
fB: open(CONST) open() < read()
read(fA)
fA: open(CONST)
close(fA)
Loop:
read(fA)
write(fB, CONST)
fB: open(CONST)
close(fA)
write(fB, CONST)
close(fB) close(fB)

115. Abstract Temporal
Source Code
Representation Properties
fA: open(CONST)
fB: open(CONST) open() < read()
open() < close()
read(fA)
fA: open(CONST)
close(fA)
Loop:
read(fA)
write(fB, CONST)
fB: open(CONST)
close(fA)
write(fB, CONST)
close(fB) close(fB)

116. Abstract Temporal
Source Code
Representation Properties
fA: open(CONST)
fB: open(CONST) open() < read()
open() < close()
read(fA)
read() < read()
fA: open(CONST)
close(fA)
Loop:
read(fA)
write(fB, CONST)
fB: open(CONST)
close(fA)
write(fB, CONST)
close(fB) close(fB)

117. Abstract Temporal
Source Code
Representation Properties
fA: open(CONST)
fB: open(CONST) open() < read()
open() < close()
read(fA)
read() < read()
fA: open(CONST)
close(fA) read() < close()
Loop:
read(fA)
write(fB, CONST)
fB: open(CONST)
close(fA)
write(fB, CONST)
close(fB) close(fB)

118. Abstract Temporal
Source Code
Representation Properties
fA: open(CONST)
fB: open(CONST) open() < read()
open() < close()
read(fA)
read() < read()
fA: open(CONST)
close(fA) read() < close()
Loop:
read(fA)
write(fB, CONST)
fB: open(CONST) open() < write()
close(fA) open() < close()
write(fB, CONST) write() < write()
close(fB) close(fB)
write() < close()

120. thousands of projects

122. 8,000
6,000
4,000
2,000
0
C projects

123. 8,000
6,097
6,000
4,000
2,000
0
C projects

124. 200,000,000 8,000
6,097
150,000,000 6,000
100,000,000 4,000
50,000,000 2,000
0 0
Lines of code C projects

125. 201,321,237
200,000,000 8,000
6,097
150,000,000 6,000
100,000,000 4,000
50,000,000 2,000
0 0
Lines of code C projects

126. 6,097 C projects

127. 201,321,237 lines of code

128. 5,985,193 functions

129. 15,803,766 properties (“f < g”)

130. 6 GB database

131. 18 hours analysis time
single core

132. 11 million lines of code per hour

133. 11 seconds per project

134. static int dcc_listen_init (…) {
dcc->sok = socket(…);
if (…) {
while (…) {
… = bind (dcc->sok, …);
}
/* with a small port range, reUseAddr is needed */
setsockopt (dcc->sok, …, SO_REUSEADDR, …);
}
listen (dcc->sok, …);
}

135. static int dcc_listen_init (…) {
dcc->sok = socket(…);
if (…) {
while (…) {
… = bind (dcc->sok, …);
}
/* with a small port range, reUseAddr is needed */
setsockopt (dcc->sok, …, SO_REUSEADDR, …);
}
listen (dcc->sok, …); should be called before bind()
}

136. static int find_file (…)
{
DIR *dirp;
struct dirent *dirinfo;
…
dirp = opendir(".");
if (dirp == NULL)
{
…
}
while ((dirinfo = readdir(dirp)) != NULL)
{
…
}
rewinddir(dirp);
return 1;
}

137. static int find_file (…)
{
DIR *dirp;
struct dirent *dirinfo;
…
dirp = opendir(".");
if (dirp == NULL)
{
…
}
while ((dirinfo = readdir(dirp)) != NULL)
{
…
}
rewinddir(dirp);
return 1; should call closedir() instead
}

139. Platform

142. Check my Code
• Check your code against
the wisdom of Linux
• Builds on millions of
mined speci cations
• Detects problems no
other tool can detect
www.checkmycode.org

143. Check my Code
• Check your code against
the wisdom of Linux
Dat abase
• Builds on millions of
ilable
mined speci cations ava
fo r dow nload
• Detects problems no
other tool can detect
www.checkmycode.org

144. speci cation crisis

145. speci cation crisis

146. microsoft word mobile phones
travel booking operating systems
airplane control banking systems

147. microsoft word mobile phones
travel booking operating systems
airplane control banking systems
easy to mine

148. Challenges

149. Challenges
• Mining complete speci cations

150. Challenges
• Mining complete speci cations
• Finding relevant abstractions

151. Challenges
• Mining complete speci cations
• Finding relevant abstractions
• Producing readable speci cations

152. Challenges
• Mining complete speci cations
• Finding relevant abstractions
• Producing readable speci cations
• Integrating speci cation mining
and programming

153. Andrzej Wasylkowski Christian Lindig Natalie Gruska

154. Summary

155. Summary

156. Summary

157. Summary

158. Summary

159. Summary