SlideShare a Scribd company logo
1 of 19
Toolchain Independent
Distributed Compilation
Dietmar Hauser
Head of Console Technology
Sproing Interactive Media GmbH
http://xkcd.com/303/
Is it possible?
Input
●
1 cpp/c source file
●
Many h/hpp header files
“→ Compilation Unit” (“CU”)
Output
●
1 obj/o binary file
●
(Misc. helper files)
Research previous solutions
IncrediBuild
●
The de-facto standard
●
Easy setup, works well
●
Pretty, but pricey
●
Limited platforms
●
Coordinated load balancing
distcc
●
Free, but narrow focus
●
Needs homogenous setup
●
Two methods:
●
Preprocess & distribute
●
Analyse & distribute
The Plan
Send input files
●
Start with .cpp/.c
●
Find required .h/.inl/...
●
Precompiled Header (PCH)
●
Compiler executable(s)
●
Calculate hash for every file
Receive output files
●
Main .obj binary file
●
Misc .ti/.sbr/.d/... files
●
Log output
●
Cache inputs using hash
●
Cache output by
combining input hashes
Finding all input files
Preprocess Method
●
Run Preprocessor locally
●
Distribute Result
●
Easy to do
●
Less parallelisation
●
Input file cache not possible
●
No PCH support possible
Analyse Method
●
Analyse .cpp file
●
Find all dependencies
●
Sounds simple, but is tricky
●
Slightly more data to send
●
Good cache behaviour
●
PCH support possible
Precompiled Header Files
●
Speeds up compilation
●
Contains “global” includes
●
Is included in every CU
●
Proprietary format
●
Often very big
●
Contains extra dependency
information
●
May not be deterministic
Source: http://www.ogre3d.org
Fun with preprocessor directives
●
Directory search order
●
<> vs. “” includes
●
Multi line includes
●
Case mismatches
●
Conditional includes
●
PP constants in includes
●
PP macros in includes
●
Conservative approach
●
Find all possible dependencies
●
Reasonable overhead
●
Cache dependencies locally
●
Still a world of pain
●
Trial & Error
Putting it all together
●
Collect all input files
●
Send them if needed (zipped)
●
Build directory structure in Temp
●
Cache & Compile
●
Collect all output files
●
Cache & Send back (zipped)
●
???
●
Profit!
It kinda works...
Little problems:
●
PCH files don't work
●
Long & deeply nested file
names
●
Absolute paths
●
Some compilers need registry
●
Issues with parallel jobs
●
And some more...
Big problem:
●
Debug info stores absolute
paths to source files!
Sandboxie to the rescue
●
Virtual file system
●
Recreate original paths
●
No concurrency issues
●
Simple clean up
●
Virtual registry
●
Not free (~€10-25 per user)
●
Does not solve all problems
●
But it's good enough!
Miscellaneous titbits
●
“Screen Saver” mode
●
Automatic server updates
●
Output file cache (~ccache)
●
Data compression woes
●
100 MBit/s vs. 1 Gbit/s
●
Local compilation server
●
Parallel local compilation
●
Parallel linking experiment
So, is it worth the hassle?
●
Measuring this is tricky
●
Real projects
●
In a live environment
●
34 servers, ~17 available
●
Maximum speed up: ~17
●
Uncached: 0.6 – 6.68
●
Cached: 1.06 – 13.13
Sproing's Codebase (21 Projects)
Schlag den Raab 2 (1 Project)
3rd
Party Codebase (64 Projects)
Conclusions
It's possible distribute compilation with any compiler
Speed up is highly dependent on the environment and use case
Speed up is almost always positive, often greatly
What's next?
●
Get other developers involved?
●
Leverage an external cloud?
●
Distribute other processes? (Asset conversion,...)
●
Find a better solution for PCH?
●
Improve or unify front end with LLVM & Clang?
●
Distributed linking?
Thank you for your attention!
dietmar.hauser@sproing.com
@Rattenhirn
http://www.sproing.com
http://fb.me/sproing
Questions?

More Related Content

What's hot

Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open sourceThomas Alrin
 
Introduction to protocol buffer
Introduction to protocol bufferIntroduction to protocol buffer
Introduction to protocol bufferTim (文昌)
 
[POSS 2019] OVirt and Ceph: Perfect Combination.?
[POSS 2019] OVirt and  Ceph: Perfect Combination.?[POSS 2019] OVirt and  Ceph: Perfect Combination.?
[POSS 2019] OVirt and Ceph: Perfect Combination.?Worteks
 
Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016Gluster.org
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/awaitC4Media
 
Object Compaction in Cloud for High Yield
Object Compaction in Cloud for High YieldObject Compaction in Cloud for High Yield
Object Compaction in Cloud for High YieldScyllaDB
 
Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0Ceph Community
 
Fluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUFluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUN Masahiro
 
Redis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRedis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRed Hat Developers
 
Sdc 2012-challenges
Sdc 2012-challengesSdc 2012-challenges
Sdc 2012-challengesGluster.org
 
.NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov).NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov)ITCamp
 
Fluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconFluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconN Masahiro
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?ScyllaDB
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
 
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017Codemotion
 
Whoops! I Rewrote It in Rust
Whoops! I Rewrote It in RustWhoops! I Rewrote It in Rust
Whoops! I Rewrote It in RustScyllaDB
 

What's hot (20)

Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open source
 
Introduction to protocol buffer
Introduction to protocol bufferIntroduction to protocol buffer
Introduction to protocol buffer
 
[POSS 2019] OVirt and Ceph: Perfect Combination.?
[POSS 2019] OVirt and  Ceph: Perfect Combination.?[POSS 2019] OVirt and  Ceph: Perfect Combination.?
[POSS 2019] OVirt and Ceph: Perfect Combination.?
 
Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016Gluster d thread_synchronization_using_urcu_lca2016
Gluster d thread_synchronization_using_urcu_lca2016
 
Rust Primer
Rust PrimerRust Primer
Rust Primer
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/await
 
Object Compaction in Cloud for High Yield
Object Compaction in Cloud for High YieldObject Compaction in Cloud for High Yield
Object Compaction in Cloud for High Yield
 
Rust programming-language
Rust programming-languageRust programming-language
Rust programming-language
 
Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0Performance optimization for all flash based on aarch64 v2.0
Performance optimization for all flash based on aarch64 v2.0
 
Fluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EUFluentd Project Intro at Kubecon 2019 EU
Fluentd Project Intro at Kubecon 2019 EU
 
Redis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRedis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech Talk
 
Sdc 2012-challenges
Sdc 2012-challengesSdc 2012-challenges
Sdc 2012-challenges
 
.NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov).NET Memory Primer (Martin Kulov)
.NET Memory Primer (Martin Kulov)
 
Fluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at KubeconFluentd and Distributed Logging at Kubecon
Fluentd and Distributed Logging at Kubecon
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uring
 
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
Asynchronous IO in Rust - Enrico Risa - Codemotion Rome 2017
 
Fluentd 101
Fluentd 101Fluentd 101
Fluentd 101
 
Kraken mesoscon 2018
Kraken mesoscon 2018Kraken mesoscon 2018
Kraken mesoscon 2018
 
Whoops! I Rewrote It in Rust
Whoops! I Rewrote It in RustWhoops! I Rewrote It in Rust
Whoops! I Rewrote It in Rust
 

Viewers also liked (11)

Surat peryataan
Surat peryataanSurat peryataan
Surat peryataan
 
Marmorarias manual de_referencia
Marmorarias manual de_referenciaMarmorarias manual de_referencia
Marmorarias manual de_referencia
 
Dara puspita
Dara puspitaDara puspita
Dara puspita
 
Marketing your Startup....things you have to do
Marketing your Startup....things you have to doMarketing your Startup....things you have to do
Marketing your Startup....things you have to do
 
รายได้ 4 ทางจาก 4life
รายได้ 4 ทางจาก 4lifeรายได้ 4 ทางจาก 4life
รายได้ 4 ทางจาก 4life
 
Diseño material-educativo-digital
Diseño material-educativo-digitalDiseño material-educativo-digital
Diseño material-educativo-digital
 
Cytokines and obesity
Cytokines and obesityCytokines and obesity
Cytokines and obesity
 
Pitch Deck-Format-NinzaBiz.com | investment Deck for Fund Raising
Pitch Deck-Format-NinzaBiz.com | investment Deck for Fund RaisingPitch Deck-Format-NinzaBiz.com | investment Deck for Fund Raising
Pitch Deck-Format-NinzaBiz.com | investment Deck for Fund Raising
 
persentasi melalui power point
persentasi melalui power pointpersentasi melalui power point
persentasi melalui power point
 
Liderazgo centrado en Principios (parte 01)
Liderazgo centrado en Principios (parte 01)Liderazgo centrado en Principios (parte 01)
Liderazgo centrado en Principios (parte 01)
 
Microsoft azure edited
Microsoft azure   editedMicrosoft azure   edited
Microsoft azure edited
 

Similar to Toolchain Independent Distributed Compilation

Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmKernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmAnne Nicolas
 
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)어형 이
 
Cache in Chromium: Disk Cache
Cache in Chromium: Disk CacheCache in Chromium: Disk Cache
Cache in Chromium: Disk CacheChang W. Doh
 
Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...VMware Tanzu
 
Manage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachManage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachinovex GmbH
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan BlueDatabricks
 
Continuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritageContinuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritageRan Levy
 
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelUnikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelThe Linux Foundation
 
It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)Jarek Potiuk
 
Real-world Experiences in Scala
Real-world Experiences in ScalaReal-world Experiences in Scala
Real-world Experiences in ScalaAmir Karimi
 
Php Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The CookbookPhp Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The CookbookVladimir Reznichenko
 
Gears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesGears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesPerforce
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmersmrsabo
 
LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013dotCloud
 
LXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software DeliveryLXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software DeliveryDocker, Inc.
 
Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)Ange Albertini
 
Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013dotCloud
 

Similar to Toolchain Independent Distributed Compilation (20)

Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmKernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
 
Mongodb meetup
Mongodb meetupMongodb meetup
Mongodb meetup
 
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
How to debug the pod which is hard to debug (디버그 하기 어려운 POD 디버그 하기)
 
Cache in Chromium: Disk Cache
Cache in Chromium: Disk CacheCache in Chromium: Disk Cache
Cache in Chromium: Disk Cache
 
Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...Customize and Secure the Runtime and Dependencies of Your Procedural Language...
Customize and Secure the Runtime and Dependencies of Your Procedural Language...
 
Manage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachManage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approach
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan Blue
 
Continuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritageContinuous Deployment Applied at MyHeritage
Continuous Deployment Applied at MyHeritage
 
Hg for bioinformatics, second part
Hg for bioinformatics, second partHg for bioinformatics, second part
Hg for bioinformatics, second part
 
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernelUnikernel User Summit 2015: Getting started in unikernels using the rump kernel
Unikernel User Summit 2015: Getting started in unikernels using the rump kernel
 
It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)It's a Breeze to develop Apache Airflow (Apache Con Berlin)
It's a Breeze to develop Apache Airflow (Apache Con Berlin)
 
Real-world Experiences in Scala
Real-world Experiences in ScalaReal-world Experiences in Scala
Real-world Experiences in Scala
 
Php Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The CookbookPhp Inspections (EA Extended): The Cookbook
Php Inspections (EA Extended): The Cookbook
 
Gears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development ChallengesGears of Perforce: AAA Game Development Challenges
Gears of Perforce: AAA Game Development Challenges
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmers
 
Go in Production
Go in ProductionGo in Production
Go in Production
 
LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013
 
LXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software DeliveryLXC Docker and the Future of Software Delivery
LXC Docker and the Future of Software Delivery
 
Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)Binary art - Byte-ing the PE that fails you (extended offline version)
Binary art - Byte-ing the PE that fails you (extended offline version)
 
Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013
 

More from Dietmar Hauser

The Case Against Human Readability
The Case Against Human ReadabilityThe Case Against Human Readability
The Case Against Human ReadabilityDietmar Hauser
 
More Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code CompletionMore Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code CompletionDietmar Hauser
 
Going Rogue - 8 Months On My Own
Going Rogue - 8 Months On My OwnGoing Rogue - 8 Months On My Own
Going Rogue - 8 Months On My OwnDietmar Hauser
 
The Rocky Road to KISS Rock City
The Rocky Road to KISS Rock CityThe Rocky Road to KISS Rock City
The Rocky Road to KISS Rock CityDietmar Hauser
 
A Half Life in Game Development
A Half Life in Game DevelopmentA Half Life in Game Development
A Half Life in Game DevelopmentDietmar Hauser
 
The Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for RaiosThe Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for RaiosDietmar Hauser
 
The Difficulty of Going Mobile
The Difficulty of Going MobileThe Difficulty of Going Mobile
The Difficulty of Going MobileDietmar Hauser
 
Handling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development TeamHandling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development TeamDietmar Hauser
 

More from Dietmar Hauser (12)

The Case Against Human Readability
The Case Against Human ReadabilityThe Case Against Human Readability
The Case Against Human Readability
 
More Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code CompletionMore Intuitive Programming Through Better Code Completion
More Intuitive Programming Through Better Code Completion
 
Data Compression 2020
Data Compression 2020Data Compression 2020
Data Compression 2020
 
The Abstraction Trap
The Abstraction TrapThe Abstraction Trap
The Abstraction Trap
 
The Settlers Returns
The Settlers ReturnsThe Settlers Returns
The Settlers Returns
 
Going Rogue - 8 Months On My Own
Going Rogue - 8 Months On My OwnGoing Rogue - 8 Months On My Own
Going Rogue - 8 Months On My Own
 
The Rocky Road to KISS Rock City
The Rocky Road to KISS Rock CityThe Rocky Road to KISS Rock City
The Rocky Road to KISS Rock City
 
A Half Life in Game Development
A Half Life in Game DevelopmentA Half Life in Game Development
A Half Life in Game Development
 
Devil Dentist
Devil DentistDevil Dentist
Devil Dentist
 
The Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for RaiosThe Unusual Rendering Pipeline of Sigils - Battle for Raios
The Unusual Rendering Pipeline of Sigils - Battle for Raios
 
The Difficulty of Going Mobile
The Difficulty of Going MobileThe Difficulty of Going Mobile
The Difficulty of Going Mobile
 
Handling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development TeamHandling Many Platforms with a Small Development Team
Handling Many Platforms with a Small Development Team
 

Recently uploaded

Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...OnePlan Solutions
 
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxReal-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxRTS corp
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxRTS corp
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...Akihiro Suda
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingOpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingShane Coughlan
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Cizo Technology Services
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Rob Geurden
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Developmentvyaparkranti
 
Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Anthony Dahanne
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonApplitools
 

Recently uploaded (20)

Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
 
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxReal-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingOpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
 
Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
 

Toolchain Independent Distributed Compilation

  • 1. Toolchain Independent Distributed Compilation Dietmar Hauser Head of Console Technology Sproing Interactive Media GmbH
  • 3. Is it possible? Input ● 1 cpp/c source file ● Many h/hpp header files “→ Compilation Unit” (“CU”) Output ● 1 obj/o binary file ● (Misc. helper files)
  • 4. Research previous solutions IncrediBuild ● The de-facto standard ● Easy setup, works well ● Pretty, but pricey ● Limited platforms ● Coordinated load balancing distcc ● Free, but narrow focus ● Needs homogenous setup ● Two methods: ● Preprocess & distribute ● Analyse & distribute
  • 5. The Plan Send input files ● Start with .cpp/.c ● Find required .h/.inl/... ● Precompiled Header (PCH) ● Compiler executable(s) ● Calculate hash for every file Receive output files ● Main .obj binary file ● Misc .ti/.sbr/.d/... files ● Log output ● Cache inputs using hash ● Cache output by combining input hashes
  • 6. Finding all input files Preprocess Method ● Run Preprocessor locally ● Distribute Result ● Easy to do ● Less parallelisation ● Input file cache not possible ● No PCH support possible Analyse Method ● Analyse .cpp file ● Find all dependencies ● Sounds simple, but is tricky ● Slightly more data to send ● Good cache behaviour ● PCH support possible
  • 7. Precompiled Header Files ● Speeds up compilation ● Contains “global” includes ● Is included in every CU ● Proprietary format ● Often very big ● Contains extra dependency information ● May not be deterministic Source: http://www.ogre3d.org
  • 8. Fun with preprocessor directives ● Directory search order ● <> vs. “” includes ● Multi line includes ● Case mismatches ● Conditional includes ● PP constants in includes ● PP macros in includes ● Conservative approach ● Find all possible dependencies ● Reasonable overhead ● Cache dependencies locally ● Still a world of pain ● Trial & Error
  • 9. Putting it all together ● Collect all input files ● Send them if needed (zipped) ● Build directory structure in Temp ● Cache & Compile ● Collect all output files ● Cache & Send back (zipped) ● ??? ● Profit!
  • 10. It kinda works... Little problems: ● PCH files don't work ● Long & deeply nested file names ● Absolute paths ● Some compilers need registry ● Issues with parallel jobs ● And some more... Big problem: ● Debug info stores absolute paths to source files!
  • 11. Sandboxie to the rescue ● Virtual file system ● Recreate original paths ● No concurrency issues ● Simple clean up ● Virtual registry ● Not free (~€10-25 per user) ● Does not solve all problems ● But it's good enough!
  • 12. Miscellaneous titbits ● “Screen Saver” mode ● Automatic server updates ● Output file cache (~ccache) ● Data compression woes ● 100 MBit/s vs. 1 Gbit/s ● Local compilation server ● Parallel local compilation ● Parallel linking experiment
  • 13. So, is it worth the hassle? ● Measuring this is tricky ● Real projects ● In a live environment ● 34 servers, ~17 available ● Maximum speed up: ~17 ● Uncached: 0.6 – 6.68 ● Cached: 1.06 – 13.13
  • 15. Schlag den Raab 2 (1 Project)
  • 17. Conclusions It's possible distribute compilation with any compiler Speed up is highly dependent on the environment and use case Speed up is almost always positive, often greatly
  • 18. What's next? ● Get other developers involved? ● Leverage an external cloud? ● Distribute other processes? (Asset conversion,...) ● Find a better solution for PCH? ● Improve or unify front end with LLVM & Clang? ● Distributed linking?
  • 19. Thank you for your attention! dietmar.hauser@sproing.com @Rattenhirn http://www.sproing.com http://fb.me/sproing Questions?

Editor's Notes

  1.   PC Xbox 360 Wii PS3 3DS WiiU iOS Non-distributive 170 161 348 466 533 1400 665 Clean Rebuild 281 (0,6) 223 (0,72) 230 (1,51) 362 (1,29) 224 (2,38) 392 (3,57) 260 (2,56) Rebuild 160 (1,06) 115 (1,4) 125 (2,78) 215 (2,17) 122 (4,37) 245 (5,71) 141 (4,72)
  2.   PC PS3 3DS Non-distributive 81 287 394 Clean Rebuild 47 (1,72) 59 (4,86) 59 (6,68) Rebuild 18 (4,5) 22 (13,05) 30 (13,13)
  3.   XBox Non-distributive 18320 Clean Rebuild 6667 (2,75) Rebuild 4903 (3,74)