SlideShare a Scribd company logo
1 of 19
Toolchain Independent
Distributed Compilation
Dietmar Hauser
Head of Console Technology
Sproing Interactive Media GmbH
http://xkcd.com/303/
Is it possible?
Input
●
1 cpp/c source file
●
Many h/hpp header files
“→ Compilation Unit” (“CU”)
Output
●
1 obj/o binary file
●
(Misc. helper files)
Research previous solutions
IncrediBuild
●
The de-facto standard
●
Easy setup, works well
●
Pretty, but pricey
●
Limited platforms
●
Coordinated load balancing
distcc
●
Free, but narrow focus
●
Needs homogenous setup
●
Two methods:
●
Preprocess & distribute
●
Analyse & distribute
The Plan
Send input files
●
Start with .cpp/.c
●
Find required .h/.inl/...
●
Precompiled Header (PCH)
●
Compiler executable(s)
●
Calculate hash for every file
Receive output files
●
Main .obj binary file
●
Misc .ti/.sbr/.d/... files
●
Log output
●
Cache inputs using hash
●
Cache output by
combining input hashes
Finding all input files
Preprocess Method
●
Run Preprocessor locally
●
Distribute Result
●
Easy to do
●
Less parallelisation
●
Input file cache not possible
●
No PCH support possible
Analyse Method
●
Analyse .cpp file
●
Find all dependencies
●
Sounds simple, but is tricky
●
Slightly more data to send
●
Good cache behaviour
●
PCH support possible
Precompiled Header Files
●
Speeds up compilation
●
Contains “global” includes
●
Is included in every CU
●
Proprietary format
●
Often very big
●
Contains extra dependency
information
●
May not be deterministic
Source: http://www.ogre3d.org
Fun with preprocessor directives
●
Directory search order
●
<> vs. “” includes
●
Multi line includes
●
Case mismatches
●
Conditional includes
●
PP constants in includes
●
PP macros in includes
●
Conservative approach
●
Find all possible dependencies
●
Reasonable overhead
●
Cache dependencies locally
●
Still a world of pain
●
Trial & Error
Putting it all together
●
Collect all input files
●
Send them if needed (zipped)
●
Build directory structure in Temp
●
Cache & Compile
●
Collect all output files
●
Cache & Send back (zipped)
●
???
●
Profit!
It kinda works...
Little problems:
●
PCH files don't work
●
Long & deeply nested file
names
●
Absolute paths
●
Some compilers need registry
●
Issues with parallel jobs
●
And some more...
Big problem:
●
Debug info stores absolute
paths to source files!
Sandboxie to the rescue
●
Virtual file system
●
Recreate original paths
●
No concurrency issues
●
Simple clean up
●
Virtual registry
●
Not free (~€10-25 per user)
●
Does not solve all problems
●
But it's good enough!
Miscellaneous titbits
●
“Screen Saver” mode
●
Automatic server updates
●
Output file cache (~ccache)
●
Data compression woes
●
100 MBit/s vs. 1 Gbit/s
●
Local compilation server
●
Parallel local compilation
●
Parallel linking experiment
So, is it worth the hassle?
●
Measuring this is tricky
●
Real projects
●
In a live environment
●
34 servers, ~17 available
●
Maximum speed up: ~17
●
Uncached: 0.6 – 6.68
●
Cached: 1.06 – 13.13
Sproing's Codebase (21 Projects)
Schlag den Raab 2 (1 Project)
3rd
Party Codebase (64 Projects)
Conclusions
It's possible distribute compilation with any compiler
Speed up is highly dependent on the environment and use case
Speed up is almost always positive, often greatly
What's next?
●
Get other developers involved?
●
Leverage an external cloud?
●
Distribute other processes? (Asset conversion,...)
●
Find a better solution for PCH?
●
Improve or unify front end with LLVM & Clang?
●
Distributed linking?
Thank you for your attention!
dietmar.hauser@sproing.com
@Rattenhirn
http://www.sproing.com
http://fb.me/sproing
Questions?

More Related Content

What's hot

Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open sourceThomas Alrin
 
Introduction to protocol buffer
Introduction to protocol bufferIntroduction to protocol buffer
Introduction to protocol bufferTim (文昌)
 
[POSS 2019] OVirt and Ceph: Perfect Combination.?
[POSS 2019] OVirt and  Ceph: Perfect Combination.?[POSS 2019] OVirt and  Ceph: Perfect Combination.?
[POSS 2019] OVirt and Ceph: Perfect Combination.?Worteks
 
Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016Gluster.org
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/awaitC4Media
 
Object Compaction in Cloud for High Yield
Object Compaction in Cloud for High YieldObject Compaction in Cloud for High Yield
Object Compaction in Cloud for High YieldScyllaDB
 
Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0Ceph Community
 
Fluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUFluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUN Masahiro
 
Redis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRedis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRed Hat Developers
 
Sdc 2012-challenges
Sdc 2012-challengesSdc 2012-challenges
Sdc 2012-challengesGluster.org
 
.NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov).NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov)ITCamp
 
Fluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconFluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconN Masahiro
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?ScyllaDB
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
 
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017Codemotion
 
Whoops! I Rewrote It in Rust
Whoops! I Rewrote It in RustWhoops! I Rewrote It in Rust
Whoops! I Rewrote It in RustScyllaDB
 

What's hot (20)

Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open source
 
Introduction to protocol buffer
Introduction to protocol bufferIntroduction to protocol buffer
Introduction to protocol buffer
 
[POSS 2019] OVirt and Ceph: Perfect Combination.?
[POSS 2019] OVirt and  Ceph: Perfect Combination.?[POSS 2019] OVirt and  Ceph: Perfect Combination.?
[POSS 2019] OVirt and Ceph: Perfect Combination.?
 
Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016
 
Rust Primer
Rust PrimerRust Primer
Rust Primer
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/await
 
Object Compaction in Cloud for High Yield
Object Compaction in Cloud for High YieldObject Compaction in Cloud for High Yield
Object Compaction in Cloud for High Yield
 
Rust programming-language
Rust programming-languageRust programming-language
Rust programming-language
 
Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0
 
Fluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUFluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EU
 
Redis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRedis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech Talk
 
Sdc 2012-challenges
Sdc 2012-challengesSdc 2012-challenges
Sdc 2012-challenges
 
.NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov).NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov)
 
Fluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconFluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at Kubecon
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uring
 
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
 
Fluentd 101
Fluentd 101Fluentd 101
Fluentd 101
 
Kraken mesoscon 2018
Kraken mesoscon 2018Kraken mesoscon 2018
Kraken mesoscon 2018
 
Whoops! I Rewrote It in Rust
Whoops! I Rewrote It in RustWhoops! I Rewrote It in Rust
Whoops! I Rewrote It in Rust
 

Viewers also liked (11)

Surat peryataan
Surat peryataanSurat peryataan
Surat peryataan
 
Marmorarias manual de_referencia
Marmorarias manual de_referenciaMarmorarias manual de_referencia
Marmorarias manual de_referencia
 
Dara puspita
Dara puspitaDara puspita
Dara puspita
 
Marketing your Startup....things you have to do
Marketing your Startup....things you have to doMarketing your Startup....things you have to do
Marketing your Startup....things you have to do
 
รายได้ 4 ทางจาก 4life
รายได้ 4 ทางจาก 4lifeรายได้ 4 ทางจาก 4life
รายได้ 4 ทางจาก 4life
 
Diseño material-educativo-digital
Diseño material-educativo-digitalDiseño material-educativo-digital
Diseño material-educativo-digital
 
Cytokines and obesity
Cytokines and obesityCytokines and obesity
Cytokines and obesity
 
Pitch Deck-Format-NinzaBiz.com | investment Deck for Fund Raising
Pitch Deck-Format-NinzaBiz.com | investment Deck for Fund RaisingPitch Deck-Format-NinzaBiz.com | investment Deck for Fund Raising
Pitch Deck-Format-NinzaBiz.com | investment Deck for Fund Raising
 
persentasi melalui power point
persentasi melalui power pointpersentasi melalui power point
persentasi melalui power point
 
Liderazgo centrado en Principios (parte 01)
Liderazgo centrado en Principios (parte 01)Liderazgo centrado en Principios (parte 01)
Liderazgo centrado en Principios (parte 01)
 
Microsoft azure edited
Microsoft azure   editedMicrosoft azure   edited
Microsoft azure edited
 

Similar to Toolchain Independent Distributed Compilation

Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmKernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmAnne Nicolas
 
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)어형 이
 
Cache in Chromium: Disk Cache
Cache in Chromium: Disk CacheCache in Chromium: Disk Cache
Cache in Chromium: Disk CacheChang W. Doh
 
Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...VMware Tanzu
 
Manage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachManage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachinovex GmbH
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan BlueDatabricks
 
Continuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritageContinuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritageRan Levy
 
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelUnikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelThe Linux Foundation
 
It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)Jarek Potiuk
 
Real-world Experiences in Scala
Real-world Experiences in ScalaReal-world Experiences in Scala
Real-world Experiences in ScalaAmir Karimi
 
Php Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The CookbookPhp Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The CookbookVladimir Reznichenko
 
Gears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesGears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesPerforce
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmersmrsabo
 
LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013dotCloud
 
LXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software DeliveryLXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software DeliveryDocker, Inc.
 
Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)Ange Albertini
 
Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013Docker, Inc.
 

Similar to Toolchain Independent Distributed Compilation (20)

Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmKernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
 
Mongodb meetup
Mongodb meetupMongodb meetup
Mongodb meetup
 
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
 
Cache in Chromium: Disk Cache
Cache in Chromium: Disk CacheCache in Chromium: Disk Cache
Cache in Chromium: Disk Cache
 
Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...
 
Manage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachManage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approach
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan Blue
 
Continuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritageContinuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritage
 
Hg for bioinformatics, second part
Hg for bioinformatics, second partHg for bioinformatics, second part
Hg for bioinformatics, second part
 
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelUnikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
 
It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)
 
Real-world Experiences in Scala
Real-world Experiences in ScalaReal-world Experiences in Scala
Real-world Experiences in Scala
 
Php Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The CookbookPhp Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The Cookbook
 
Gears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesGears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development Challenges
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmers
 
Go in Production
Go in ProductionGo in Production
Go in Production
 
LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013
 
LXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software DeliveryLXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software Delivery
 
Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)
 
Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013
 

More from Dietmar Hauser

The Case Against Human Readability
The Case Against Human ReadabilityThe Case Against Human Readability
The Case Against Human ReadabilityDietmar Hauser
 
More Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code CompletionMore Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code CompletionDietmar Hauser
 
Going Rogue - 8 Months On My Own
Going Rogue - 8 Months On My OwnGoing Rogue - 8 Months On My Own
Going Rogue - 8 Months On My OwnDietmar Hauser
 
The Rocky Road to KISS Rock City
The Rocky Road to KISS Rock CityThe Rocky Road to KISS Rock City
The Rocky Road to KISS Rock CityDietmar Hauser
 
A Half Life in Game Development
A Half Life in Game DevelopmentA Half Life in Game Development
A Half Life in Game DevelopmentDietmar Hauser
 
The Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for RaiosThe Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for RaiosDietmar Hauser
 
The Difficulty of Going Mobile
The Difficulty of Going MobileThe Difficulty of Going Mobile
The Difficulty of Going MobileDietmar Hauser
 
Handling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development TeamHandling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development TeamDietmar Hauser
 

More from Dietmar Hauser (12)

The Case Against Human Readability
The Case Against Human ReadabilityThe Case Against Human Readability
The Case Against Human Readability
 
More Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code CompletionMore Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code Completion
 
Data Compression 2020
Data Compression 2020Data Compression 2020
Data Compression 2020
 
The Abstraction Trap
The Abstraction TrapThe Abstraction Trap
The Abstraction Trap
 
The Settlers Returns
The Settlers ReturnsThe Settlers Returns
The Settlers Returns
 
Going Rogue - 8 Months On My Own
Going Rogue - 8 Months On My OwnGoing Rogue - 8 Months On My Own
Going Rogue - 8 Months On My Own
 
The Rocky Road to KISS Rock City
The Rocky Road to KISS Rock CityThe Rocky Road to KISS Rock City
The Rocky Road to KISS Rock City
 
A Half Life in Game Development
A Half Life in Game DevelopmentA Half Life in Game Development
A Half Life in Game Development
 
Devil Dentist
Devil DentistDevil Dentist
Devil Dentist
 
The Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for RaiosThe Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for Raios
 
The Difficulty of Going Mobile
The Difficulty of Going MobileThe Difficulty of Going Mobile
The Difficulty of Going Mobile
 
Handling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development TeamHandling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development Team
 

Recently uploaded

cpct NetworkING BASICS AND NETWORK TOOL.ppt
cpct NetworkING BASICS AND NETWORK TOOL.pptcpct NetworkING BASICS AND NETWORK TOOL.ppt
cpct NetworkING BASICS AND NETWORK TOOL.pptrcbcrtm
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureImplementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureDinusha Kumarasiri
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based projectAnoyGreter
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf31events.com
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Matt Ray
 
PREDICTING RIVER WATER QUALITY ppt presentation
PREDICTING  RIVER  WATER QUALITY  ppt presentationPREDICTING  RIVER  WATER QUALITY  ppt presentation
PREDICTING RIVER WATER QUALITY ppt presentationvaddepallysandeep122
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...Technogeeks
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Natan Silnitsky
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmSujith Sukumaran
 
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsUnveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsAhmed Mohamed
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEEVICTOR MAESTRE RAMIREZ
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样umasea
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesPhilip Schwarz
 
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024Andreas Granig
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaHanief Utama
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalLionel Briand
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxTier1 app
 

Recently uploaded (20)

cpct NetworkING BASICS AND NETWORK TOOL.ppt
cpct NetworkING BASICS AND NETWORK TOOL.pptcpct NetworkING BASICS AND NETWORK TOOL.ppt
cpct NetworkING BASICS AND NETWORK TOOL.ppt
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureImplementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based project
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
 
PREDICTING RIVER WATER QUALITY ppt presentation
PREDICTING  RIVER  WATER QUALITY  ppt presentationPREDICTING  RIVER  WATER QUALITY  ppt presentation
PREDICTING RIVER WATER QUALITY ppt presentation
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
 
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsUnveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML Diagrams
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
 
2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a series
 
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
 

Toolchain Independent Distributed Compilation

  • 1. Toolchain Independent Distributed Compilation Dietmar Hauser Head of Console Technology Sproing Interactive Media GmbH
  • 3. Is it possible? Input ● 1 cpp/c source file ● Many h/hpp header files “→ Compilation Unit” (“CU”) Output ● 1 obj/o binary file ● (Misc. helper files)
  • 4. Research previous solutions IncrediBuild ● The de-facto standard ● Easy setup, works well ● Pretty, but pricey ● Limited platforms ● Coordinated load balancing distcc ● Free, but narrow focus ● Needs homogenous setup ● Two methods: ● Preprocess & distribute ● Analyse & distribute
  • 5. The Plan Send input files ● Start with .cpp/.c ● Find required .h/.inl/... ● Precompiled Header (PCH) ● Compiler executable(s) ● Calculate hash for every file Receive output files ● Main .obj binary file ● Misc .ti/.sbr/.d/... files ● Log output ● Cache inputs using hash ● Cache output by combining input hashes
  • 6. Finding all input files Preprocess Method ● Run Preprocessor locally ● Distribute Result ● Easy to do ● Less parallelisation ● Input file cache not possible ● No PCH support possible Analyse Method ● Analyse .cpp file ● Find all dependencies ● Sounds simple, but is tricky ● Slightly more data to send ● Good cache behaviour ● PCH support possible
  • 7. Precompiled Header Files ● Speeds up compilation ● Contains “global” includes ● Is included in every CU ● Proprietary format ● Often very big ● Contains extra dependency information ● May not be deterministic Source: http://www.ogre3d.org
  • 8. Fun with preprocessor directives ● Directory search order ● <> vs. “” includes ● Multi line includes ● Case mismatches ● Conditional includes ● PP constants in includes ● PP macros in includes ● Conservative approach ● Find all possible dependencies ● Reasonable overhead ● Cache dependencies locally ● Still a world of pain ● Trial & Error
  • 9. Putting it all together ● Collect all input files ● Send them if needed (zipped) ● Build directory structure in Temp ● Cache & Compile ● Collect all output files ● Cache & Send back (zipped) ● ??? ● Profit!
  • 10. It kinda works... Little problems: ● PCH files don't work ● Long & deeply nested file names ● Absolute paths ● Some compilers need registry ● Issues with parallel jobs ● And some more... Big problem: ● Debug info stores absolute paths to source files!
  • 11. Sandboxie to the rescue ● Virtual file system ● Recreate original paths ● No concurrency issues ● Simple clean up ● Virtual registry ● Not free (~€10-25 per user) ● Does not solve all problems ● But it's good enough!
  • 12. Miscellaneous titbits ● “Screen Saver” mode ● Automatic server updates ● Output file cache (~ccache) ● Data compression woes ● 100 MBit/s vs. 1 Gbit/s ● Local compilation server ● Parallel local compilation ● Parallel linking experiment
  • 13. So, is it worth the hassle? ● Measuring this is tricky ● Real projects ● In a live environment ● 34 servers, ~17 available ● Maximum speed up: ~17 ● Uncached: 0.6 – 6.68 ● Cached: 1.06 – 13.13
  • 15. Schlag den Raab 2 (1 Project)
  • 17. Conclusions It's possible distribute compilation with any compiler Speed up is highly dependent on the environment and use case Speed up is almost always positive, often greatly
  • 18. What's next? ● Get other developers involved? ● Leverage an external cloud? ● Distribute other processes? (Asset conversion,...) ● Find a better solution for PCH? ● Improve or unify front end with LLVM & Clang? ● Distributed linking?
  • 19. Thank you for your attention! dietmar.hauser@sproing.com @Rattenhirn http://www.sproing.com http://fb.me/sproing Questions?

Editor's Notes

  1.   PC Xbox 360 Wii PS3 3DS WiiU iOS Non-distributive 170 161 348 466 533 1400 665 Clean Rebuild 281 (0,6) 223 (0,72) 230 (1,51) 362 (1,29) 224 (2,38) 392 (3,57) 260 (2,56) Rebuild 160 (1,06) 115 (1,4) 125 (2,78) 215 (2,17) 122 (4,37) 245 (5,71) 141 (4,72)
  2.   PC PS3 3DS Non-distributive 81 287 394 Clean Rebuild 47 (1,72) 59 (4,86) 59 (6,68) Rebuild 18 (4,5) 22 (13,05) 30 (13,13)
  3.   XBox Non-distributive 18320 Clean Rebuild 6667 (2,75) Rebuild 4903 (3,74)