This document discusses parallel builds in the Eclipse IDE workspace. It describes how builds can be parallelized by creating one job per build and throttling the number of threads. Parallel builds provide benefits like shorter completion times and faster availability of individual projects. However, parallelization must account for dependencies between projects and conflicting scheduling rules used by builders. The document recommends that builders use the narrowest possible scheduling rule to minimize conflicts and allow more parallelism. It analyzes specific cases like long build chains and provides examples to illustrate how scheduling rules impact parallelization.

1. Parallel builds in Eclipse IDE Workspace
Value, making-of and optimal usages
Mickael Istria - @mickaelistria CC0 – No rights reserved

2. Advertisement: OpenShift.io

3. Parallel builds in Eclipse IDE Workspace
Value, making-of and optimal usages
Mickael Istria - @mickaelistria CC0 – No rights reserved

4. Demo scenario
6 projects with WaitBuilder waiting for some time (Thread.sleep())
– Projects deps directly configured in .project
– Builder duration configured in waitBuilder.properties (default 1000ms)
– Builder schedulingRule configured in waitBuilder.properties (default current
project) can be set to workspace root or some other project.

5. The basics: currently (pre-Photon)
Builds are sequential; most basic case: no dep,
not conflicting scheduling rule

6. The basics: Parallelizing
● Create 1 job per build
● Throttle on limited amount of
threads using JobGroup API
● Preference used by ”legacy”
Workspace.build() API
● Default in Photon is
disabled (value == 1)

7. The basics: Parallelized
No dependencies, isolated scheduling rules

8. The basics: Benefits
● Shorter full time to completion (& less energy
consumed)
● Faster availability of each project
– A long project build doesn’t block for too long
– If you have handlers on project builds completion,
they’re called way earlier

9. Usable progress View

10. Dependency Graph: no parallel
Dependency graph is computed, and turned
into a sequence (via topological sort)

11. Dependency graph: Parallel (4 threads)
Graph is used directly and jobs are scheduled
for builds when all ancestors are built

12. Scheduling Rules
● 2 operations using conflicting scheduling rules
rule1.isConflicting(rule2) can NOT run at the same time,
even in different threads.
●
Scheduling rules are usually Resource Path which conflict when
one path is a parent of the other ( a/b and a/b/c are
conflicting;
a/b and a/c are not conflicting)
●
Builders are expected to implement
IncrementalProjectBuilder.getRule(…)
Default is workspace.getRoot()
Here, p1 build rule conflicts with all other build rules

13. Scheduling rules don’t matter for
scheduling in non-parallel
As it’s sequential, no issue about conflicting
scheduling rules

14. So why scheduling rules on builders?
Safety: Scheduling rule are here to prevent some other
thread to change critical content while builder is
running. The scheduling rule defines what must NOT
be changed externally while builder is running.
GOLDEN RULE: To minimize conflicts and allow
more parallelism, set the narrowest rule your
builder can deal with.

15. (Bad) State of art
Many builders do not specify a scheduling rule
(defaulting to workspace root, conflicting with
every workspace operation)
→ Building a project locks the whole workspace!
Builder uses root as SR, conflicts with whole WS
Change file in another
unrelated Project + Save

16. (Bad) examples
● JDT Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=531554
● PDE Builder https://bugs.eclipse.org/bugs/show_bug.cgi?id=531555
● M2E Builder
…
Use default workspace scheduling rule,
preventing parallel execution and blocking user
actions in unrelated files!

17. With finer scheduling rule...
More specific scheduling rule, only necessary
content is locked by builder.
For example, just using getProject() as
scheduling rule can often be enough.
Builder uses project as SR, conflicts with only project content
Change file in another
unrelated Project + Save File just saved!

18. Back to parallel builds
Not conflicting builder execution, no bad results

19. Current impl does not pass rules to schedule jobs
and use rules at job exec. Some jobs are started for
nothing ( )
But could do better

20. Now you got it
● 1 project build → 1 job
● Throttling with JobGroup
● Scheduled according to dependency graph
● Honoring scheduling rules
GOLDEN RULE: To minimize conflicts and allow
more parallelism, set the narrowest rule your
builder can deal with.

21. Highlighted cases
● Long build chain vs standalone projet
● Depedency cycles
● 1 project builder uses workspace root as
scheduling rule

22. Long chain vs standalone project
1 thread
4 threads
p6 is available much much earlier

23. With cycles, sequential
Compute a best sequential order (using “Strongly
connected component”) and apply it

24. With cycles, parallel
1. Process graph until no node is ready (symptom of cycle)
2. Then process the 1st
waiting node in computed sequential order
3. Then back to regular graph processing.
1 2 3

25. Workspace root scheduling rule
● For backward compatibility, if any builder use workspace
root scheduling rule, disable parallel builds (as the
scheduler will use this rule in its thread)
● Anyway, workspace root would conflict with about
everything…
● So most builders currently prevents parallel builds

26. GOLDEN RULE:
To minimize conflicts and allow more parallelism,
set the narrowest rule your builder can deal with.

27. Case studies
1. dotnet build builder
2. JavaBuilder (JDT)

28. Case Study #1
6 independent C# projects building with dotnet
build (builder scheduling root == current project)
Sequential build takes 39s
Parallel on 4 threads takes 26s
Actual gain: 33% (cool, but...)
Expected gain: 66%
?

29. Case study #1 lesson learned
Most real world operation have bottlenecks (I/O,
network…) making concurrent executions slower.
Sequential
4 threads
But still a major win!

30. Case study #2: JDT
JavaBuilder do use default workspace root
scheduling rule → Disable/prevents parallel builds

31. GOLDEN RULE:
To minimize conflicts and allow more parallelism,
set the narrowest rule your builder can deal with.

32. Case study #2: JavaBuilder’s scheduling rule
Allows independent non-JDT builds in parallel of JDT ones for
more immediate availability.
Hopefully part of Eclipse JDT 4.9 (September 2018)
Possible greedy
& safe
scheduling rule:
all JDT projects

33. Finding the minimal scheduling rule (ie apply the GOLDEN
RULE) would require a thourough analysis of JavaBuilder.
– What can be done in parallel?
– What needs to be synchronized and how?
– ...
Probably very worth the effort as it’d speed up build in IDE and
JDT.LS, but maybe not if there are major bottlenecks...
Case study #2: JavaBuilder’s scheduling rule

34. Many builders similar to JDT
All the same issues and possible improvements
applies to other parts of the IDE (PDE, m2e,
WTP, PDT...)

35. Mickael Istria - @mickaelistria CC0 – No rights reserved
And remember GOLDEN RULE:
To minimize conflicts and allow more parallelism,
set the narrowest rule your builder can deal with.
Questions?