Successfully reported this slideshow.
Your SlideShare is downloading. ×

Toolchain Independent Distributed Compilation

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 19 Ad
Advertisement

More Related Content

Slideshows for you (20)

Advertisement

Similar to Toolchain Independent Distributed Compilation (20)

Advertisement

Recently uploaded (20)

Toolchain Independent Distributed Compilation

  1. 1. Toolchain Independent Distributed Compilation Dietmar Hauser Head of Console Technology Sproing Interactive Media GmbH
  2. 2. http://xkcd.com/303/
  3. 3. Is it possible? Input ● 1 cpp/c source file ● Many h/hpp header files “→ Compilation Unit” (“CU”) Output ● 1 obj/o binary file ● (Misc. helper files)
  4. 4. Research previous solutions IncrediBuild ● The de-facto standard ● Easy setup, works well ● Pretty, but pricey ● Limited platforms ● Coordinated load balancing distcc ● Free, but narrow focus ● Needs homogenous setup ● Two methods: ● Preprocess & distribute ● Analyse & distribute
  5. 5. The Plan Send input files ● Start with .cpp/.c ● Find required .h/.inl/... ● Precompiled Header (PCH) ● Compiler executable(s) ● Calculate hash for every file Receive output files ● Main .obj binary file ● Misc .ti/.sbr/.d/... files ● Log output ● Cache inputs using hash ● Cache output by combining input hashes
  6. 6. Finding all input files Preprocess Method ● Run Preprocessor locally ● Distribute Result ● Easy to do ● Less parallelisation ● Input file cache not possible ● No PCH support possible Analyse Method ● Analyse .cpp file ● Find all dependencies ● Sounds simple, but is tricky ● Slightly more data to send ● Good cache behaviour ● PCH support possible
  7. 7. Precompiled Header Files ● Speeds up compilation ● Contains “global” includes ● Is included in every CU ● Proprietary format ● Often very big ● Contains extra dependency information ● May not be deterministic Source: http://www.ogre3d.org
  8. 8. Fun with preprocessor directives ● Directory search order ● <> vs. “” includes ● Multi line includes ● Case mismatches ● Conditional includes ● PP constants in includes ● PP macros in includes ● Conservative approach ● Find all possible dependencies ● Reasonable overhead ● Cache dependencies locally ● Still a world of pain ● Trial & Error
  9. 9. Putting it all together ● Collect all input files ● Send them if needed (zipped) ● Build directory structure in Temp ● Cache & Compile ● Collect all output files ● Cache & Send back (zipped) ● ??? ● Profit!
  10. 10. It kinda works... Little problems: ● PCH files don't work ● Long & deeply nested file names ● Absolute paths ● Some compilers need registry ● Issues with parallel jobs ● And some more... Big problem: ● Debug info stores absolute paths to source files!
  11. 11. Sandboxie to the rescue ● Virtual file system ● Recreate original paths ● No concurrency issues ● Simple clean up ● Virtual registry ● Not free (~€10-25 per user) ● Does not solve all problems ● But it's good enough!
  12. 12. Miscellaneous titbits ● “Screen Saver” mode ● Automatic server updates ● Output file cache (~ccache) ● Data compression woes ● 100 MBit/s vs. 1 Gbit/s ● Local compilation server ● Parallel local compilation ● Parallel linking experiment
  13. 13. So, is it worth the hassle? ● Measuring this is tricky ● Real projects ● In a live environment ● 34 servers, ~17 available ● Maximum speed up: ~17 ● Uncached: 0.6 – 6.68 ● Cached: 1.06 – 13.13
  14. 14. Sproing's Codebase (21 Projects)
  15. 15. Schlag den Raab 2 (1 Project)
  16. 16. 3rd Party Codebase (64 Projects)
  17. 17. Conclusions It's possible distribute compilation with any compiler Speed up is highly dependent on the environment and use case Speed up is almost always positive, often greatly
  18. 18. What's next? ● Get other developers involved? ● Leverage an external cloud? ● Distribute other processes? (Asset conversion,...) ● Find a better solution for PCH? ● Improve or unify front end with LLVM & Clang? ● Distributed linking?
  19. 19. Thank you for your attention! dietmar.hauser@sproing.com @Rattenhirn http://www.sproing.com http://fb.me/sproing Questions?

Editor's Notes

  •  
    PC
    Xbox 360
    Wii
    PS3
    3DS
    WiiU
    iOS
    Non-distributive
    170
    161
    348
    466
    533
    1400
    665
    Clean Rebuild
    281 (0,6)
    223 (0,72)
    230 (1,51)
    362 (1,29)
    224 (2,38)
    392 (3,57)
    260 (2,56)
    Rebuild
    160 (1,06)
    115 (1,4)
    125 (2,78)
    215 (2,17)
    122 (4,37)
    245 (5,71)
    141 (4,72)
  •  
    PC
    PS3
    3DS
    Non-distributive
    81
    287
    394
    Clean Rebuild
    47 (1,72)
    59 (4,86)
    59 (6,68)
    Rebuild
    18 (4,5)
    22 (13,05)
    30 (13,13)
  •  
    XBox
    Non-distributive
    18320
    Clean Rebuild
    6667 (2,75)
    Rebuild
    4903 (3,74)

×