Presented at GDCE 2013, video should be available on GDCVault

1. Toolchain Independent
Distributed Compilation
Dietmar Hauser
Head of Console Technology
Sproing Interactive Media GmbH

2. http://xkcd.com/303/

3. Is it possible?
Input
●
1 cpp/c source file
●
Many h/hpp header files
“→ Compilation Unit” (“CU”)
Output
●
1 obj/o binary file
●
(Misc. helper files)

4. Research previous solutions
IncrediBuild
●
The de-facto standard
●
Easy setup, works well
●
Pretty, but pricey
●
Limited platforms
●
Coordinated load balancing
distcc
●
Free, but narrow focus
●
Needs homogenous setup
●
Two methods:
●
Preprocess & distribute
●
Analyse & distribute

5. The Plan
Send input files
●
Start with .cpp/.c
●
Find required .h/.inl/...
●
Precompiled Header (PCH)
●
Compiler executable(s)
●
Calculate hash for every file
Receive output files
●
Main .obj binary file
●
Misc .ti/.sbr/.d/... files
●
Log output
●
Cache inputs using hash
●
Cache output by
combining input hashes

6. Finding all input files
Preprocess Method
●
Run Preprocessor locally
●
Distribute Result
●
Easy to do
●
Less parallelisation
●
Input file cache not possible
●
No PCH support possible
Analyse Method
●
Analyse .cpp file
●
Find all dependencies
●
Sounds simple, but is tricky
●
Slightly more data to send
●
Good cache behaviour
●
PCH support possible

7. Precompiled Header Files
●
Speeds up compilation
●
Contains “global” includes
●
Is included in every CU
●
Proprietary format
●
Often very big
●
Contains extra dependency
information
●
May not be deterministic
Source: http://www.ogre3d.org

8. Fun with preprocessor directives
●
Directory search order
●
<> vs. “” includes
●
Multi line includes
●
Case mismatches
●
Conditional includes
●
PP constants in includes
●
PP macros in includes
●
Conservative approach
●
Find all possible dependencies
●
Reasonable overhead
●
Cache dependencies locally
●
Still a world of pain
●
Trial & Error

9. Putting it all together
●
Collect all input files
●
Send them if needed (zipped)
●
Build directory structure in Temp
●
Cache & Compile
●
Collect all output files
●
Cache & Send back (zipped)
●
???
●
Profit!

10. It kinda works...
Little problems:
●
PCH files don't work
●
Long & deeply nested file
names
●
Absolute paths
●
Some compilers need registry
●
Issues with parallel jobs
●
And some more...
Big problem:
●
Debug info stores absolute
paths to source files!

11. Sandboxie to the rescue
●
Virtual file system
●
Recreate original paths
●
No concurrency issues
●
Simple clean up
●
Virtual registry
●
Not free (~€10-25 per user)
●
Does not solve all problems
●
But it's good enough!

12. Miscellaneous titbits
●
“Screen Saver” mode
●
Automatic server updates
●
Output file cache (~ccache)
●
Data compression woes
●
100 MBit/s vs. 1 Gbit/s
●
Local compilation server
●
Parallel local compilation
●
Parallel linking experiment

13. So, is it worth the hassle?
●
Measuring this is tricky
●
Real projects
●
In a live environment
●
34 servers, ~17 available
●
Maximum speed up: ~17
●
Uncached: 0.6 – 6.68
●
Cached: 1.06 – 13.13

14. Sproing's Codebase (21 Projects)

15. Schlag den Raab 2 (1 Project)

16. 3rd
Party Codebase (64 Projects)

17. Conclusions
It's possible distribute compilation with any compiler
Speed up is highly dependent on the environment and use case
Speed up is almost always positive, often greatly

18. What's next?
●
Get other developers involved?
●
Leverage an external cloud?
●
Distribute other processes? (Asset conversion,...)
●
Find a better solution for PCH?
●
Improve or unify front end with LLVM & Clang?
●
Distributed linking?

19. Thank you for your attention!
dietmar.hauser@sproing.com
@Rattenhirn
http://www.sproing.com
http://fb.me/sproing
Questions?