Software developers rely on a fast and correct build system to compile their source code changes and produce modified deliverables for testing and deployment. The scale and complexity of the PostgreSQL build process makes build performance an important topic to discuss and address. In this talk, we will introduce a new build performance analysis technique that identifies "build hotspots", i.e., files that are slow to rebuild (by analyzing a build dependency graph), yet change often (by analyzing version control history). We will discuss the identified hotspots in the 9.2.4 release of PostgreSQL.

1. Identifying Hotspots in the
PostgreSQL Build Processes
Shane
McIntosh
Bram
Adams
Meiyappan
Nagappan
Ahmed E.
Hassan
@shane_mcintosh
shanemcintosh@acm.org

2. What is a build system?
Source
code
!2

3. What is a build system?
Source
code
Deliverable
!2

4. Build systems describe how sources are
translated into deliverables
.c
.o
.cc
.o
.exe
.tex
.dvi
.pdf
.a
.deb
!3

5. All builds are equal…
(1)
Think
(2)
Edit
The
developer’s
work cycle
(4)
Test
(3)
Build
!4

6. …But some builds are more
equal than others
“
http://xkcd.com/303/
Why is the gtk build so abysmally slow?
This hinders our ability to run gtk bots
and keep the gtk build green. :(
”
!5

7. Refactoring can improve
build performance
deliverable1
del1.o
del1.c
deliverable2
library.a
util1.o
util2.o
util1.c
del2.o
del2.c
util2.c
!6

8. Refactoring can improve
build performance
deliverable1
del1.o
del1.c
deliverable2
library.a
util1.o
util2.o
util1.c
del2.o
del2.c
util2.c
!6

9. Refactoring can improve
build performance
deliverable1
del1.o
del1.c
deliverable2
library.a
util1.o
util2.o
util1.c
del2.o
del2.c
util2.c
!6

10. Refactoring can improve
build performance
deliverable1
del1.o
del1.c
deliverable2
library.a
util1.o
util2.o
util1.c
del2.o
del2.c
util2.c
!6

11. Refactoring can improve
build performance
deliverable1
del1.o
del1.c
deliverable2
library.a
util1.o
util2.o
util1.c
del2.o
del2.c
util2.c
!6

12. Refactoring can improve
build performance
deliverable1
deliverable2
Before refactoring:!
4 commands! del2.o
del1.o
library.a
triggered
del1.c
del2.c
util1.o
util2.o
util1.c
util2.c
!6

13. Refactoring can improve
build performance
deliverable1
del1.o
del1.c
deliverable2
library.a
util1.o
util2.o
util1.c
del2.o
del2.c
util2.c
!6

14. Refactoring can improve
build performance
deliverable1
deliverable2
del1.o
util1.o
util2.o
del2.o
del1.c
util1.c
util2.c
del2.c
!7

15. Refactoring can improve
build performance
deliverable1
deliverable2
del1.o
util1.o
util2.o
del2.o
del1.c
util1.c
util2.c
del2.c
!7

16. Refactoring can improve
build performance
deliverable1
deliverable2
del1.o
util1.o
util2.o
del2.o
del1.c
util1.c
util2.c
del2.c
!7

17. Refactoring can improve
build performance
deliverable1
deliverable2
del1.o
util1.o
util2.o
del2.o
del1.c
util1.c
util2.c
del2.c
!7

18. Refactoring can improve
build performance
deliverable1
deliverable2
After refactoring:!
2 commands! del2.o
del1.o
util1.o
util2.o
triggered
del1.c
del2.c
util1.c
util2.c
!7

19. …But where should we focus
refactoring effort?
Files that rebuild
the slowest?
!8

20. (1)
Dependency
Graph
Construction
Dep. Graph
(2)
Dependency
Graph
Analysis
Graph
analysis
results
Version
Control
System
(3)
Build
Hotspot
Detection
File Churn
An approach to detect slowly
rebuilding files
B
Qua
!9

21. MAKAO tool extracts build
dependency graphs
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis
Design recovery and maintenance of build systems
B. Adams, Herman Tromp, Kris De Schutter, Wolfgang De Meuter
[ICSM 2007]
!10

22. The cost of traversing an edge is
derived by timing its build command
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis
Median of 10
repetitions
!11

23. Open source case studies!
!
!
!
!12

24. Build dependency graph
properties
# Nodes
# Edges
38,102
2,752,225
0
750,000
1,500,000
2,250,000
3,000,000
3,375
121,710
5,131
60,170
0
35,000
70,000
105,000
140,000
!13

25. 100
50
Build Time (s)
8
6
4
0
2
0
Build Time (s)
10
12
150
Incremental build
performance
0
100
200
300
400
500
600
0
File ID
100
200
File ID
.c files
.h files
!14
300
400

26. 150
50
100
Build Time (s)
1.5
1.0
0.5
0
0.0
Build Time (s)
2.0
Incremental build
performance
0
200
400
600
800
1000
File ID
0
100
200
300
File ID
.c files
.h files
!15
400
500
600

27. 6000
0
2000
4000
Build Time (s)
100
50
0
Build Time (s)
150
8000
Incremental build
performance
0
2000
4000
6000
8000 10000
0
File ID
2000
4000
6000
File ID
.c files
.h files
!16
8000
10000

28. 100
50
0
Build Time (s)
150
So, refactoring
should target the slow files!
0
100
200
300
File ID
!17
400
500
600

29. 100
50
0
Build Time (s)
150
So, refactoring
should target the slow files!
0
100
200
300
File ID
!17
400
500
600

30. 100
50
0
Build Time (s)
150
So, refactoring
should target the slow files!
0
100
200
300
File ID
!17
400
500
600

31. 100
50
0
Build Time (s)
150
So, refactoring
should target the slow files!
0
100
200
300
File ID
!17
400
500
600

32. Let’s see what
developers think!
> Please answer the following questions for each
source file listed above:!
> (1) Is this source code file a performance
bottleneck? Y/N!
!
For all of them no...
!18

33. Let’s see what
developers think!
> Please answer the following questions for each
source file listed above:!
> (1) Is this source code file a performance
bottleneck? Y/N!
!
For all of them no...!
!
> (2) Why do you believe this file is or is not a
performance bottleneck?!
!
...because none of these headers change often.
!19

34. …But where should we focus
refactoring effort?
Files that rebuild
the slowest?
!20

35. …But where should we focus
refactoring effort?
Files that rebuild
the slowest?
…but they may
only rarely change!
!20

36. …But where should we focus
refactoring effort?
Files that rebuild
the slowest?
Files that change
the most often?
…but they may
only rarely change!
!20

37. Mining frequently changing files
from version control
Terminal
$ git clone …
…
$ git checkout <some_tag>
…
$ git log --oneline <some_file>
!21

38. Mining frequently changing files
from version control
But many of these files
$ git clone …
…
already build quickly!
Terminal
$ git checkout <some_tag>
…
$ git log --oneline <some_file>
!21

39. …But where should we focus
refactoring effort?
Files that rebuild
the slowest?
Files that change
the most often?
…but they may
only rarely change!
!22

40. …But where should we focus
refactoring effort?
Files that rebuild
the slowest?
Files that change
the most often?
…but they may
only rarely change!
…but they may
already be optimal!
!22

41. …But where should we focus
refactoring effort?
Files that rebuild
the slowest?
Files that change
the most:often?
ots
…but they may
only rarely change!
…but they may
already be optimal!
otsp
h
ft e n
uild
ge o
on b
han
cus
nd c
Fo
ly a
slow
build
at re
es th
il
F
!22

42. (1)
Dependency
Graph
Construction
Dep. Graph
(2)
Dependency
Graph
Analysis
Graph
analysis
results
Version
Control
System
(3)
Build
Hotspot
Detection
File Churn
An approach to detect slowly
rebuilding files
B
Qua
!23

43. (1)
Dependency
Graph
Construction
Dep. Graph
(2)
Dependency
Graph
Analysis
Graph
analysis
results
Version
Control
System
(3)
Build
Hotspot
Detection
File Churn
Hotspot detection approach
Build Activity
Quadrant Plot
!24

44. Quadrant plots highlight build
hotspots
Number of changes
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis
(3)
Build
Hotspot
Detection
Rebuild cost
!25

45. Quadrant plots highlight build
hotspots
Number of changes
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis
(3)
Build
Hotspot
Detection
Rebuild cost
!25

46. Quadrant plots highlight build
hotspots
Number of changes
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis
(3)
Build
Hotspot
Detection
Rebuild cost
!25

47. Number of changes
Quadrant plots highlight build
hotspots
factor
Re
e first!
thes
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis
(3)
Build
Hotspot
Detection
Rebuild cost
!25

48. Open source case studies!
Thresholds
!
Rebuild cost
90 seconds
Number of changes
!
Median
!
!26

49. 1.00
●
●
●
●
● ●
●
●
Number of changes
(Normalized)
Normalized File Churn
●
●
●
●
●
0.25
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●● ● ●
●
●
●
●
●● ● ●
●
●
●
●
●
● ●
●
●● ●
● ●
●
●
●
●
●
● ●● ●
● ●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
65 hotspots!
(7% of
source files)
●
●
●
●
0.75
0.50
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●● ●
●
●
●●
●
●
●●
●●
●
●
●
● ● ●
●●
●
● ●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●● ●
●
● ●
●
●●
●
●
●●
●
●
●
●
●● ● ● ●
●● ●
●
●
●
●
●●
●●
●
●
●
●
●
●
●● ● ●
●● ●
●
●
●
●
●
● ●● ●
●
● ● ●
●
●●
●
●
●
●●●●●
●● ●
●
●
●
●
●
●
●
●
●
●● ● ●
●● ●
●
●
●
●
0
●●
●●
●
●
glib/glib.h
glib/glib-object.h
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
Main culprits
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●●
●
●
●
●●
● ●
●
● ●●
● ●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
50
●
● ●●
●
Build Time
●
●
●
● ●
100
150
Rebuild cost
(seconds)
!27

50. 1.00
●
●
●
●
●
●
●
Number of changes
(Normalized)
Normalized File Churn
●
0.75
0.50
0.25
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
● ● ●●
●
●
●● ●
●●
●
●
●
●
●● ● ●
● ●
●● ●
●
●
●
●
●●
●●
●● ●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●● ●
●
● ● ● ● ● ●● ●
● ●●●
●
●
●● ● ●● ● ●
●● ● ●
● ●
●
●●●● ●● ●
●●●●
●●
●●
●●
● ●
●●● ●●● ●
●●● ●
●●
●
●
●
●●●● ●● ● ●
●●●● ●
●● ●
●● ●
●●
●
●
●
●●●●●●●● ●
●●● ●●●●
●●● ● ●
● ●● ●
●●
●●
●
732 hotspots!
(8% of source files)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
● ●
●
●●
●
●●
●●
●
●● ●
● ●
●
●
●
●
●
●●
● ●
● ●●
● ●
●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
● ●
●
●
●
●
●
● ●
● ●●
●
●● ● ●●● ● ●
●● ●● ● ●
●● ●●
●●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●● ● ●● ●
●● ● ●
●● ●
●● ●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●●●●●● ● ● ● ●
●● ● ●● ●
●
●
● ●●
●
●
●
●
●
●
●
Main culprits
tmtm
●
●
●
●
●
● ●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
● ●
●
● ●
●
● ●●
●
● ● ● ●
● ●
● ●● ●
● ●
●
●
●
●● ● ●
●
●● ● ●
●
●●
●
●●● ● ●
●●
●●
●●
●
● ●● ●
●
●
●●● ● ● ●
●● ● ●
●● ●
●
●
●
●
●
●
●
●
●
●
● ● ●
●●● ● ●● ● ●
●● ● ● ●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●●
●●●● ●● ● ●
●●
●●
●●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●●●●
●●
● ●
●
●●●●●●●●●● ● ●
●●● ● ●●●● ● ●
● ●●● ●●●●
● ●● ● ●
●●
●
●●● ● ● ●●
●●● ● ● ●
●●● ●
●●●
●
●
●
●
● ●●
● ●
●
●
●
●
●●●●●●●● ● ● ●
●●●●●● ●● ●
●●● ●● ●●
●●●● ● ●
●●● ●
●●
●●● ●●●
● ●● ● ●●
● ●●
●●
● ●
●
●
●
●
●
High hotspot!
concentration
qtbase/…/corelib
qtxmlpatterns
●●●●●● ● ●
●●●● ●
●●●●
● ●●●
●●●
●
0
2000
4000
Build Time (s)
6000
8000
Rebuild cost
(seconds)
!28

51. 1.00
●
●
●
●
●
●●
●
●
●
Number of changes
(Normalized)
Normalized File Churn
●
●
●
●
●
●
0.75
0.50
0.25
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
● ●● ●
●
●
● ● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
● ●
● ● ●● ●
●
●
●
●
●
●
●
●
●
● ●● ● ●
●
●
●
●
● ●
●
●
●
●
●
●
● ● ●
●
● ●
●
●
●
● ● ●
●
●
●
● ●●● ●
●●
●
●●●●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●● ●●● ● ●
●
●
● ● ●
●
● ●
●
● ●●
●
● ●● ●●
●
● ●●
●
● ●●
●●
●
●
●
●
●
●●● ●● ●
●
●
●
●● ●
●
●
●
●● ● ●●● ●
●
●
●
●
●
●● ●
●●
●
●
●● ● ●
●●
●
●
●● ● ●
●
●
●●
●● ● ●
●●
●
●
●● ● ●
●●
●
●● ●
● ●●
●
●
●
●
●
●● ● ● ●● ● ●●
●● ● ● ● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●● ●●
●●
●
●● ●
● ●
●
●
● ●
●●●
●● ●
●
●
●●
●●
●
● ●
●●
● ● ●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
● ●
● ●
●
●
● ● ●●
●
● ●
●
●
●
●
●
●
●●● ●
●● ●
●
●
●●
●● ● ●
●●
●
●● ●
●●
●●
●● ● ●
●
●
● ●
●
●●
●●
●●
●
●
●● ●
●●
●
●
●
●● ● ● ●
●● ●
● ●●
●●
●●● ●●
● ●●●
●●
● ●
●
●
●●
●●
●●
●
●
● ●● ●
●●
●●
●
●
27 hotspots!
(2% of source
files)
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
Main culprits
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
postgres.h
access/htup.h
access/genam.h
access/xact.h
utils/builtins.h
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●
●
●
●●● ●
●●●
● ●
●
●
●● ●
● ●
●
●
●
●●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●
0
●
50
100
Build Time (s)
150
200
Rebuild cost
(seconds)
!29

52. General trends
Transitive property of build hotspots
1
2
...
n-1
n
hot.o
hot.c
!30

53. General trends
Transitive property of build hotspots
1
2
...
n-1
n
hot.o
hot.c
!30

54. General trends
Transitive property of build hotspots
1
2
...
n-1
n
hot.o
hot.c
x.h
!30

55. General trends
Transitive property of build hotspots
1
2
...
n-1
n
hot.o
hot.c
x.h
!30

56. 1
2
...
n-1
n
General trends
hot.o
Transitive property of build hotspots
hot.c
x.h
!31

57. 1
2
...
n-1
n
General trends
hot.o
Transitive property of build hotspots
hot.c
x.h
!31

58. 1
2
...
n-1
n
General trends
hot.o
Transitive property of build hotspots
hot.c
x.h
hub.h
!31

59. 1
2
...
n-1
n
General trends
hot.o
Transitive property of build hotspots
hot.c
a.h
b.h
x.h
y.h
z.h
hub.h
!31

60. 1
2
...
n-1
n
General trends
hot.o
Transitive property of build hotspots
hot.c
a.h
b.h
x.h
y.h
z.h
hub.h
!31

61. 1
2
...
n-1
n
General trends
hot.o
Transitive property of build hotspots
hot.c
a.h
b.h
x.h
y.h
z.h
hub.h
!31

62. 1
2
...
n-1
n
General trends
hot.o
Transitive property of build hotspots
hot.c
a.h
b.h
x.h
y.h
z.h
hub.h
Limit internal use of “header file hubs”
as much as possible!
!31

63. 2nd International Workshop on Release Engineering
RELENG 2014
?
?
http://releng.polymtl.ca

64. 2nd International Workshop on Release Engineering
RELENG 2014
keynotes
?
?
http://releng.polymtl.ca
talks on
research
and
practice
poster
session
and panel
Deadline: February 28, 2014
Event: April 11, 2014

65. 2nd International Workshop on Release Engineering
RELENG 2014
keynotes
http://releng.polymtl.ca
?
?
hosted in
Mountainview (CA, US)
by
talks on
research
and
practice
poster
session
and panel
Deadline: February 28, 2014
Event: April 11, 2014

66. 2nd International Workshop on Release Engineering
RELENG 2014
keynotes
McNutt
Dinah le)
(Goog
hosted in
Mountainview (CA, US)
by
?
http://releng.polymtl.ca
talks on
research
and
practice
poster
session
and panel
Deadline: February 28, 2014
Event: April 11, 2014

73. @shane_mcintosh
shanemcintosh@acm.org

74. Bonus!
!34

75. postgres.h
src/include/c.h
src/include/port.h
src/include/utils/elog.h
src/include/utils/palloc.h
src/include/pg_config_manual.h
!35

76. htup_details.h
src/include/access/htup.h
src/include/storage/itemptr.h
src/include/storage/itemid.h
src/include/storage/bufpage.h
src/include/access/tupdesc.h
src/include/catalog/pg_attribute.h
src/include/access/tupmacs.h
!36

77. genam.h
src/include/access/skey.h
src/include/storage/lock.h
src/include/storage/shmem.h
src/include/utils/hsearch.h
src/include/storage/lwlock.h
src/include/utils/relcache.h
!37

78. xact.h
src/include/access/xlog.h
src/include/access/xlogdefs.h
src/include/nodes/pg_list.h
src/include/lib/stringinfo.h
!38

79. builtins.h
src/include/fmgr.h
src/include/nodes/parsenodes.h
src/include/nodes/primnodes.h
src/include/nodes/pg_list.h
src/include/nodes/nodes.h
!39