Download to read offline
![Copyright © 2016 Criteo
Pre-submit tests
cbs checkout changed-repo --with-clients
cherry-pick change
cbs build
cbs test [--with-clients]
M
O
A
B](https://image.slidesharecdn.com/cdevelopmentworkflowcriteo-160609093916/85/C-development-workflow-criteo-16-320.jpg)
Uploaded byIbrahim Abubakari
C# development workflow @ criteo
This document discusses Criteo's C# development workflow, which has evolved over time. Originally, code was split across many repositories with each team responsible for a few. This led to slow change propagation and dependency issues. The new workflow aims for early integration using trunk-based development with all commits on the main branch. A "MOAB" job continuously builds all C# code from latest commits. Developers can check out snapshots from this job or all sources. Pre-submit tests are run before merging changes. Trunk-based development enables continuous delivery but requires strong test coverage and avoids large changes.
More Related Content
Similar to C# development workflow @ criteo
C# development workflow @ criteo
- 1. Copyright © 2016Criteo C# development workflow @ Criteo Patrick Bruneton Lead of DevTools 2016-06-08
- 2. Copyright © 2016Criteo C# at Criteo • 1000 .NET projects (C# and SQL server) • 100 Git repositories • 200 developers touching the C# codebase • Complex dependency graph Creating an efficient development workflow has been a challenge.
- 3. Copyright © 2016Criteo Early days: open source model • Code splited into many C# Nugets • Each team responsible of a few Nugets (/ repositories) Team A Repo A A 1.0 build Repo B build B 1.1 1.2 Team BMerge Requests 1.1
- 4. Copyright © 2016Criteo Issue #1: change propagation A B C • Changes in C are long to propagate • High “Commit to Prod” lag • Many versions of each component to maintain • High integration cost • Late feedback 1.1 1.3
- 5. Copyright © 2016Criteo Issue #2: Nuget Dependency Hell A B D C 1.0 1.0 1.1 1.2
- 6. Copyright © 2016Criteo New workflow Goal: integrate early (Continous Delivery) A B D C HEAD HEAD HEAD HEAD (for internal components)
- 7. Copyright © 2016Criteo Trunk Based Development All commits in master branch No feature branches, no internal versioning Requires a good test coverage Benefits: • Very early integration of commits • Allows true Continuous Delivery Drawbacks: • A commit can break all builds • Harder to do large changes (you’re not allowed to break anything)
- 8. Copyright © 2016Criteo Trunk Based Development with C# at Criteo Challenges • Multiple git repositories • Lots of projects • Use cases • Build on the CI servers • Build on the dev machines • Pre-merge tests
- 9. Copyright © 2016Criteo Trunk Based Development with C# at Criteo Jenkins CBS+ + +
- 10. Copyright © 2016Criteo Build in Jenkins: the MOAB A job that continuously builds all C# code from latest commits Lib1 Lib2 App MOAB #11 MOAB #10
- 11. Copyright © 2016Criteo Build in Jenkins: the MOAB $ cbs checkout $ cbs build $ cbs test $ cbs export Full build: - Uses Microsoft.Build API - Manages the build queue itself - 24 builds in parallel - CreateHardLinksFor* = true A full build every 20 minutes
- 12. Copyright © 2016Criteo Build in Jenkins: the MOAB $ cbs checkout $ cbs build $ cbs test $ cbs export Full build: Runs Unit tests in //
- 13. Copyright © 2016Criteo Build in Jenkins: the MOAB $ cbs checkout $ cbs build $ cbs test $ cbs export Full build: List of SHA1 used Dependency graph Deployable packages Assemblies (dll + pdb) Filer
- 14. Copyright © 2016Criteo Build on the Dev machine: using all sources $ cbs checkout app --with-dependencies $ ls app/app.csproj lib1/lib1.csproj lib2/lib2.csproj default.sln $ cbs build App Lib1 Lib2
- 15. Copyright © 2016Criteo Build on the Dev machine: using MOAB snapshots $ cbs checkout app $ ls app/app.csproj default.sln bin/moab/lib1.dll bin/moab/lib2.dll <Reference Include="Lib1"> <Reference Include="Lib2"> <Import Project="cbs.targets"> App Lib1 Lib2
- 16. Copyright © 2016Criteo Pre-submit tests cbs checkout changed-repo --with-clients cherry-pick change cbs build cbs test [--with-clients] M O A B
- 17. Copyright © 2016Criteo Conclusion • Trunk Based Development is powerful • It’s a pre-requisite for Continuous Delivery, which allows agility • TBD can be implemented with C#, but we needed some tooling
- 18. Copyright © 2016Criteo Questions ?
Editor's Notes
- #4 We splitted our codebase in many components packaged as Nugets. Each team was responsible of a few Nugets. Each Nuget had a version, and new versions were periodically uploaded to a nuget server. This setup worked for some time, it allowed teams to work independently from each other. On this example, team B can work on next version without worrying about their clients. And Team A is safe, because it can control exactly when it wants to integrate changes from other teams. You need only your repo to work, you don’t care about the repos of your dependencies. So it worked, but we quickly faced major issues.
- #5 A breaking change in C can take a long time to be deployed in prod. C does is development and publishes its new major version. B has to upgrade its dependencies, but they may have no reason to do so. If it works fine with the version they have, why would they upgrade ? Upgrading a dependency is scary, you get many changes at one. It can be costly if breaking changes have been introduced. So the guys from team C may have to convince their clients to upgrade their dependencies. So if you have a complex dependency graph, we a lot of levels of depenency, it can become very hard to propagate changes quickly. Also, since a Nuget can be used by several components, at different versions, a team has to maintain several versions of it Nugets. This a waste of time. You cannot be sure your commit is OK until it’s deployed in production, and it happens weeks after, it’s not efficient.
- #6 There is a second issue with the Nuget system. It’s the Nuget Dependency Hell. It’s quite simple, it happends when you have diamon dependencies. If A depends on B and C. B and C depends on D, but with different versions. Then you have problems. B and C must be synchronized in terms of dependency which is hard to enforce.
- #7 Spending too much time in integration. Bad for agility and scaling. We decided that each component would always depend on the latest commits of its dependencies. There are still components but there are simple assemblies, not versionned nugets. All development is made is master branch. When you do a change in D, then next time A is built, this change will be integrated in A immediately. It solves our problem of late integration. You do a commit in the code base, it’s immediately visible to all. Baiscally, it’s like if we had a single repository with a giant solution.
- #8 Dev are done in a single shared trunk. It’s the model that is used by companies likes Google or Facebook. There is no feature branches, all changes are done in master. Every commit produces a production ready build. It allows to do true Continuous Delivery. If you have feature branches or bumping of internal versions, then you cannot do continous delivery, because of the late integration. A commit in any repository can break the build of everybody, so we need proper tooling to avoid this situation. This is the opposite of the previous model, where when you do a commit in your component, nodody is impacted, until they actually upgrade their dependencies. Also, since you cannot break anything, it’s harder to do breaking changes, because each commit must keep the codebase in a stable state. TBD cannot be done properly if no proper test coverage: you need to be sure at any time that your code is working fine,
- #9 Since there is no more Nugets and versions, it means everybody has to build using the sources. As a dev, since there are many repos, it’s hard to know where all the sources of your dependencies are. We have actually diferent use cases: On the continous integration servers, we want to build from trunk On the dev machines, we have to build from also, but we would like some optimization to avoid cloning or building the sources we don’t need We need strong tests before merge, to be able to be sure that a commit is not going to build the trunk. And for this we need to answer the questions like: what projects that depend on a given one. This is necessary, so that when you do a change on a project, you are sure you’re not going to break project that depend on you.
- #10 Our setup is based on Gerrit for Code Review and git hosting, Jenkins for running the build jobs, MSBuild for actual build, and a custom tool called CBS (Criteo Build System) We had to add some tooling on top of MSBuild and Visual Studio, because we needed something to manage the multiple repos. Also the list of projects that we need to be built depends on the situation, we cannot have static msbuild or visual studio solutions. Instead with CBS, with can manage a dynamic list of projects to build.
- #11 We have a job that we call the MOAB (stands for mother of all builds), that runs as fast as possible. When a run finished, a new one is launched. The goal of this job of simply to get all latest commits from C# repositories, and build the master branch. A build number is assigned at each build.
- #12 cbs checkout that clones and fetches all latest sources. The we build using cbs build. Cbs is command line tool built in C#. ‘Cbs build’ computes the dependency graph between projects and build them in the right order. Cbs calls MSBuild for building each project, but it manages the build queue itself. It gives us more control than generating a build file for example.
- #13 Then we run test uwing NUnit. we have a runner in CBS that allows us to run tests of different assemblies in parallel.
- #14 And finally we publish artifacts of the build: We publish a file, where we store the sha1 that was used for each repo. It makes hotfix possible. We also store a dependency graph of all projects. This graph allows cbs to answer the question: what is the clients of my projects ? We store all the deployable artifacts, ready to be deployable in test environments and production We also store individual assemblies, the DLL files, that will be useful for the developers.
- #15 Cbs checkout --with-dependencies gets the source code of the project I want to work on, and all the source code of its dependencies It generates a solution to build the project and the dependencies. It can also be built usng cbs build, To get the latest commits of your dependencies, just run git pull in all repos (or cbs refresh sources)
- #16 It’s also possible to build using only the source code of the project you want to work on. You can ask cbs to not get the source code of your dependencies. It will get the assemblies of the latest MOAB build on Jenkins instead. It’s laike using snaphot in Maven. To get the latest commits of your dependencies, run ‘cbs refresh moab’, which will download the latest built assemblies. In our projects we don’t have any version or hintpath, when referencing an internal component. A special targets file, cbs.targets, tells msbuild where if has to find the dependencies, depending on the fact we have the source code or not.
- #17 Pre-submit tests are executed when a commit is proposed for review. It verifies that the commit is not going to break any source code that is depending on it. We use cbs that knows the dependency graph of all projecs to compute the project list impacted by the change. If pre-submit tests fail, then the commit is not merged in the trunk.