Authors: Adrian Perreau de Pinninck, Manu Cupcic
Conference: CAS 2014 (Barcelona)
Have you ever wondered how large software companies with an engineering culture make sure they are able to deliver software over and over to production? How do you coordinate 100+ software engineers so that there are no bottlenecks and quality is not compromised?
In this talk you will see how a Continuous Delivery system was implemented at Criteo, the fastest growing IT company in EMEA 2012. Before starting the project there were 160+ code repositories with dependency hell. They were being built independently and releases to production were error prone and painful. You will see the technical architecture behind a successful implementation of a Continuous Delivery system. The system was made up of a Gerrit code review tool connected to a Jenkins build pipeline, building 160 repositories with over 7M lines of code.
We will explore different architectural choices such as branching system, hot fixes, sandbox and pre-production environments, and how these were developed and used by the large R&D department.
26. Conway’s Vengeance
“Organizations which design
systems are constrained to
produce designs which are copies
of the communication structures of
these organizations.”
Teams dealt with unclear
interfaces, animosity grew
stronger.
28. SOX compliance (an example)
Required updating the SDK (lowest level repository)
Program Manager full time for over one month
Some TLAs were 5 SDK versions behind
29. Release problem
Code Freeze during peak season:
Development continued
When code unfroze:
One month worth of code to integrate
Took 5 months to release again
30. It wasn’t scaling!
Team had almost doubled
TLAs could take over a
year to release next time
We could lose our ability
to release forever
31. So why did we fail ?
ae64ca3
4567a81
47ac581 f478ff3
57d21a9
9aad4cb
71da3b5
developers owners of application
36. TBD is a real pain...
“Branch by Abstraction" is a technique for making a large-scale
change to a software system in gradual way that
allows you to release the system regularly while the
change is still in-progress.
Martin Fowler
40. Killing the NuGet lag
● Move every git repository to the new code review tool
● Revert to the last version in production
● Build master continuously and show progress
● Deploy in preproduction then production
44. The MOAB
...
...
...
Building and publishing project 70783/70784 : XXX ... [OK]
Building and publishing project 70784/70784 : XXX.UTest... [OK]
70784 projects successfully built (100.0 %).
10:09:12 cbs assembly-set update --moabId 13998
assemblyset.json written.
10:09:20 cbs export
Build 6a306a43dd147cd6a6fcacbf40e20b25f3f69845 exported.
10:10:23 cbs assembly-set upload
Assemblyset pushed to git@git.corp.criteo.com:qa/moab.git
10:10:31 cbs tag HEAD build/current