Identifying Hotspots in Software Build Processes

Identifying Hotspots in
Software Build Processes

Shane
McIntosh

Bram
Adams

Meiyappan
Nagappan

Ahmed E.
Hassan

@shane_mcintosh
shanemcintosh@acm.org

What is a build system?
Source
code

!2

What is a build system?
Source
code

Deliverable

!2

Build systems describe how sources are
translated into deliverables
.c

.o

.cc

.o

.exe

.tex

.dvi

.pdf

.a
.deb

!3

All builds are equal…
(1)
Think

(2)
Edit
The
developer’s
work cycle

(4)
Test

(3)
Build
!4

…But some builds are more
equal than others

“

http://xkcd.com/303/

Why is the gtk build so abysmally slow?
This hinders our ability to run gtk bots
and keep the gtk build green. :(

”

!5

Refactoring can improve
build performance
deliverable1
del1.o
del1.c

deliverable2
library.a

util1.o

util2.o

util1.c

del2.o
del2.c

util2.c

!6

build performance
deliverable1
deliverable2
Before refactoring:!
4 commands! del2.o
del1.o
library.a
triggered
del1.c
del2.c
util1.o
util2.o
util1.c

util2.c

!6

build performance
deliverable1

deliverable2

del1.o

util1.o

util2.o

del2.o

del1.c

util1.c

util2.c

del2.c

!7

build performance
deliverable1
deliverable2
After refactoring:!
2 commands! del2.o
del1.o
util1.o
util2.o
triggered
del1.c
del2.c
util1.c
util2.c

!7

…But where should we focus
refactoring effort?
Files that rebuild
the slowest?

!8

refactoring effort?
Files that rebuild
the slowest?

…but they may
only rarely change!
!8

refactoring effort?
Files that rebuild
the slowest?

Files that change
the most often?

…but they may
only rarely change!
!8

refactoring effort?
Files that rebuild
the slowest?

Files that change
the most often?

…but they may
only rarely change!

…but they may
already be optimal!
!8

refactoring effort?
Files that rebuild
the slowest?

Files that change
the most:often?
ots

…but they may
only rarely change!

…but they may
already be optimal!

otsp
h
ft e n
uild
ge o
on b
han
cus
nd c
Fo
ly a
slow
build
at re
es th
il

F

!8

(1)
Dependency
Graph
Construction

Dep. Graph

(2)
Dependency
Graph
Analysis

Graph
analysis
results

Version
Control
System

(3)
Build
Hotspot
Detection

File Churn

Hotspot detection approach

Build Activity

Quadrant Plot

!9

MAKAO tool extracts build
dependency graphs
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis

(3)
Build
Hotspot
Detection

Design recovery and maintenance of build systems
B. Adams, Herman Tromp, Kris De Schutter, Wolfgang De Meuter

[ICSM 2007]
!10

The cost of traversing an edge is
derived by timing its build command
(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis

(3)
Build
Hotspot
Detection

Median of 10
repetitions
!11

Mining frequently changing ﬁles
from version control
(1)
Dependency
Graph
Construction

Terminal

$ git clone …
…
$ git checkout <some_tag>
…
$ git log --oneline <some_file>

!12

(2)
Dependency
Graph
Analysis

(3)
Build
Hotspot
Detection

Quadrant plots highlight build
hotspots
Number of changes

(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis

(3)
Build
Hotspot
Detection

Rebuild cost
!13

Number of changes

Quadrant plots highlight build
hotspots
factor
Re
e ﬁrst!
thes

(1)
Dependency
Graph
Construction
(2)
Dependency
Graph
Analysis

(3)
Build
Hotspot
Detection

Rebuild cost
!13

Open source case studies!
Thresholds
!

Rebuild cost
90 seconds

Number of changes
!
Median
!

!14

Build dependency graph
properties
# Nodes
# Edges

38,102

2,752,225

0

750,000

1,500,000

2,250,000

3,000,000

3,375
121,710

5,131
60,170

0

35,000

70,000

105,000

140,000

!15

1.00

●
●
●

●
● ●
●
●

Number of changes
(Normalized)
Normalized File Churn

●
●

●

●
●

0.25

●
●
●
●
●
●
●
●

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●

●

●
●
●
●
●
●
●

●

●

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●● ● ●
●
●
●
●
●● ● ●
●
●
●
●
●
● ●
●
●● ●
● ●
●
●
●
●
●
● ●● ●
● ●
●
●● ●

●
●

●

●
●

●

●
●
●

●

●

●
●
●
●
●
●
●

●
●

●

●

●

●
●
●
●
●
●
●
●
●

●
●

65 hotspots!
(7% of
source ﬁles)
●

●

●
●

0.75

0.50

●
●
●
●
●
●●
●
●
●
●

●

Main culprits

●

●

●

●

●
●
●
●

●

●●
●●

●
●

●
●
●

●
●
●

●

●
●
●
●
●●
●
●
● ●
●
●
●
●

●

●
●●
●
●
●●● ●
●
●
●●
●

●
●●

●●
●
●
●

● ● ●
●●

●

● ●● ●
●

●

●

●

●

●
●
●

●
●
●

glib/glib.h
glib/glib-object.h

●
●

●

●

●●
●●
●

●

●●
●●
●

●

●●
●
●

● ●●
●
●

●
●
●

● ●●
●

●
●
●

●
●

●
●

●● ●
●
● ●
●

●●
●
●

●●
●

●
●
●

●● ● ● ●
●● ●
●
●

●
●

●●
●●
●
●

●
●
●
●

●● ● ●
●● ●
●
●

●
●
●

● ●● ●
●
● ● ●
●

●●
●
●
●

●●●●●
●● ●
●
●
●

●
●

●
●
●
●

●● ● ●
●● ●
●
●
●

●

0

●

●

●
●

●

●

●
●

●
●

●

●●

●

●

●

●

● ●
●

●●
●
●
●

●●
● ●

●

● ●●
● ●
●
●

●
●
●
●

50

Max: 148 seconds

●

●●●
●●
●
●
●

●

●

● ●●

●

Build Time

●

●
●

● ●

100

150

Rebuild cost
(seconds)
!16

1.00

●

●

●
●
●
●●

●

●
●

Number of changes
(Normalized)

●
●
●
●
●
●

0.75

0.50

0.25

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
● ●● ●
●
●
● ● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
● ●
● ● ●● ●
●
●
●
●
●
●
●
●
●
● ●● ● ●
●
●
●
●
● ●
●
●
●
●
●
●
● ● ●
●
● ●
●
●
●
● ● ●
●
●
●
● ●●● ●
●●
●
●●●●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●● ●●● ● ●
●
●
● ● ●
●
● ●
●
● ●●
●
● ●● ●●
●
● ●●
●
● ●●
●●
●
●
●
●
●
●●● ●● ●
●
●
●
●● ●
●
●
●
●● ● ●●● ●
●
●
●
●
●
●● ●
●●
●
●
●● ● ●
●●
●
●
●● ● ●
●
●
●●
●● ● ●
●●
●
●
●● ● ●
●●
●
●● ●
● ●●
●
●
●
●
●
●● ● ● ●● ● ●●
●● ● ● ● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●● ●●
●●
●
●● ●
● ●
●
●
● ●
●●●
●● ●
●
●
●●
●●
●
● ●
●●
● ● ●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
● ●
● ●
●
●
● ● ●●
●
● ●
●
●
●
●
●
●
●●● ●
●● ●
●
●
●●
●● ● ●
●●
●
●● ●
●●
●●
●● ● ●
●
●
● ●
●
●●
●●
●●
●
●
●● ●
●●
●
●
●
●● ● ● ●
●● ●
● ●●
●●
●●● ●●
● ●●●
●●
● ●
●
●
●●
●●
●●
●
●
● ●● ●
●●
●●
●
●

27 hotspots!
(2% of source
ﬁles)

●

●

●
●

●

●
●

●

●

●

●

●

●

●●

●

●

Main culprits

●
●

●

●

●

●

●
●

●
●

●

●

●

●

postgres.h
access/htup.h
access/genam.h
access/xact.h
utils/builtins.h

● ●
●
●

●

●

●

●

●
●

●

● ●
●

●
●
●
●

●

● ●
●

●

●

●
●

●
●
●

●
●

●

●
●

●

●●● ●
●●
●
●
●●● ●
●●●
● ●
●

●

●● ●
● ●
●
●

●

●●
●●
●
●
●●
●●
●●
●
●

●
●

●
●

●

●●

●
●

0

Max: 191 seconds

●

50

100

Build Time (s)

150

200

Rebuild cost
(seconds)
!17

1.00

●

●

●

●
●
●
●

Number of changes
(Normalized)

●

0.75

0.50

0.25

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
● ● ●●
●
●
●● ●
●●
●
●
●
●
●● ● ●
● ●
●● ●
●
●
●
●
●●
●●
●● ●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●● ●
●
● ● ● ● ● ●● ●
● ●●●
●
●
●● ● ●● ● ●
●● ● ●
● ●
●
●●●● ●● ●
●●●●
●●
●●
●●
● ●
●●● ●●● ●
●●● ●
●●
●
●
●
●●●● ●● ● ●
●●●● ●
●● ●
●● ●
●●
●

●
●

●●●●●●●● ●
●●● ●●●●
●●● ● ●
● ●● ●
●●
●●

●

732 hotspots!
(8% of source ﬁles)

●

●

●

●

●
●

●

●

●

●

●

●

●

●

●
●
●

●

●
●

●

●

●
●

●

● ●
●

●

●
●

●

●●

● ●

● ●●
● ●
●

●
●
●

● ●
● ●
●
●

●
●

●

●●

●

●● ●
●●
●

●

●
●●
●

●

●
●

●

●
● ●

●

●

●

●

●

●●
● ●
●

●

●

●

●

● ●●
●

●● ● ●●● ● ●
●● ●● ● ●
●● ●●
●●
●●
●

●
●

●
●

●

●

●

●●
●
●

●● ● ●● ●
●● ● ●
●● ●
●● ●
●

●

●

● ● ●

●
●

● ●

●
●
●

● ●

●
●
●
●
●
● ●●
●
●
●
●
●
●

●
●

● ●

●● ●
●

●
●

●
●

●
●

●

●
●

●

●
●
●

●●
●

●

●

●
●
●

●

●

●

Main culprits
tmtm

●

●
●●
●
●
●●
● ●
●
●●
●
●●
●●
●
●● ●

●
●

●
●

●

●

●

●●
●
●●
● ●
●
● ●
●
● ●●
●
● ● ● ●
● ●
● ●● ●
● ●
●
●
●
●● ● ●
●
●● ● ●
●
●●
●
●●● ● ●
●●
●●
●●
●
● ●● ●
●
●
●●● ● ● ●
●● ● ●
●● ●
●
●

●
●
●

●
●

●

●
●

●

●

● ●●
●

●●● ● ●● ● ●
●● ● ● ●
●●
●●
●
●
●

●
●
●

●

●

●●●●●● ● ● ● ●
●● ● ●● ●
●
●
● ●●
●
●

●

●

●

●

●
●

●

●

●

●●

●●●● ●● ● ●
●●
●●
●●
●
●
●

●
●
●
●

●

● ●●
●

●
●

●

●

●●●●
●●
● ●
●

●●●●●●●●●● ● ●
●●● ● ●●●● ● ●
● ●●● ●●●●
● ●● ● ●
●●
●

●●● ●●●
● ●● ● ●●
● ●●
●●
● ●
●

●
●

● ●●
● ●
●
●
●
●

●●●●●●●● ● ● ●
●●●●●● ●● ●
●●● ●● ●●
●●●● ● ●
●●● ●
●●

●●● ● ● ●●
●●● ● ● ●
●●● ●
●●●
●
●

●

●
●

●
●
●

●

●

●

High hotspot!
concentration
qtbase/…/corelib
qtxmlpatterns

●●●●●● ● ●
●●●● ●
●●●●
● ●●●
●●●
●

0

2000

4000

Build Time (s)

6000

8000

Max: 2hrs 19min

Rebuild cost
(seconds)
!18

General trends
Transitive property of build hotspots
1

2

...

n-1

n

hot.o

hot.c

!19

General trends
1

2

...

n-1

n

hot.o

hot.c

x.h
!19

1

2

...

n-1

n

General trends
hot.o

hot.c

x.h

!20

1

2

...

n-1

n

General trends
hot.o

hot.c

x.h

hub.h

!20

1

2

...

n-1

n

General trends
hot.o

hot.c

a.h

b.h

x.h

y.h

z.h

hub.h

!20

1

2

...

n-1

n

General trends
hot.o

hot.c

a.h

b.h

x.h

y.h

z.h

hub.h

Limit internal use of “header ﬁle hubs”
as much as possible!
!20

@shane_mcintosh
shanemcintosh@acm.org

Identifying Hotspots in Software Build Processes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Identifying Hotspots in Software Build Processes

Similar to Identifying Hotspots in Software Build Processes (20)

Recently uploaded

Recently uploaded (20)

Identifying Hotspots in Software Build Processes