Have you ever wondered how large software companies with an engineering culture make sure they are able to deliver software over and over to production? How do you coordinate 100+ software engineers so that there are no bottlenecks and quality is not compromised? In this talk you will see how a Continuous Delivery system was implemented at Criteo, the fastest growing IT company in EMEA 2012. Before starting the project there were 160+ code repositories with dependency hell. They were being built independently and releases to production were error prone and painful. You will see the technical architecture behind a successful implementation of a Continuous Delivery system. The system was made up of a Gerrit code review tool connected to a Jenkins build pipeline, building 160 repositories with over 7M lines of code. We will explore different architectural choices such as branching system, hot fixes, sandbox and pre-production environments, and how these were developed and used by the large R&D department. Authors: Adrian Perreau de Pinninck, Manu Cupcic

1. Delivery at Scale

2. Manu Cupcic and Adrian Perreau

3. Scale
● > 38 PB on our HDFS
● > 1 billion ad impressions per day
● > 5000 servers world wide

4. cost of change
number of devs
The problem

5. Once upon a time

6. Single C# Repository

7. Build on local machines

8. Build on local machines

9. Build on local machines

10. Continuous Integration??

11. Running applications

12. No Testing Culture

13. Weekly Merges
20 engineers
50 engineers

14. Weekly Merges: Disaster!
140 engineers

16. First attempt
at fixing

17. Bring in the horse power

18. Internal Open Source Model

19. Splitting the Repository

20. Hmmmm… NuGets!

21. Problems

22. Repositories everywhere
2012 => 33 repositories
2013 => 160+ repositories

23. Spaghetti and NuGets
criteo.frontend
criteo.display
criteo.cookies
criteo.display
common.utils
criteo.serve
criteo.bidding
criteo.rendering criteo.objects criteo.urls
criteo.sql
criteo.data
criteo.banners
criteo.memcache
flash
criteo.biere

24. Repository Ownership
All repos to be owned by a team
Merge requests for repos you don’t own

25. Component Teams

26. Conway’s Vengeance
“Organizations which design
systems are constrained to
produce designs which are copies
of the communication structures of
these organizations.”
Teams dealt with unclear
interfaces, animosity grew
stronger.

27. Propagating changes

28. SOX compliance (an example)
Required updating the SDK (lowest level repository)
Program Manager full time for over one month
Some TLAs were 5 SDK versions behind

29. Release problem
Code Freeze during peak season:
Development continued
When code unfroze:
One month worth of code to integrate
Took 5 months to release again

30. It wasn’t scaling!
Team had almost doubled
TLAs could take over a
year to release next time
We could lose our ability
to release forever

31. So why did we fail ?
ae64ca3
4567a81
47ac581 f478ff3
57d21a9
9aad4cb
71da3b5
developers owners of application

33. Integrate early
developers
4567a81 57d21a9 f478ff3 9aad4cb 71da3b5 47ac581

34. Trunk based development

35. Trunk based development

36. TBD is a real pain...
“Branch by Abstraction" is a technique for making a large-scale
change to a software system in gradual way that
allows you to release the system regularly while the
change is still in-progress.
Martin Fowler

37. But we love it

38. What did it take ?

39. Build infrastructure
Virtual team for 6 months.
Afterwards, team became
permanent.

40. Killing the NuGet lag
● Move every git repository to the new code review tool
● Revert to the last version in production
● Build master continuously and show progress
● Deploy in preproduction then production

41. Test plans progress

42. Our setup
in great (technical) details

43. Code reviews with gerrit
M
O
A
B

44. The MOAB
...
...
...
Building and publishing project 70783/70784 : XXX ... [OK]
Building and publishing project 70784/70784 : XXX.UTest... [OK]
70784 projects successfully built (100.0 %).
10:09:12 cbs assembly-set update --moabId 13998
assemblyset.json written.
10:09:20 cbs export
Build 6a306a43dd147cd6a6fcacbf40e20b25f3f69845 exported.
10:10:23 cbs assembly-set upload
Assemblyset pushed to git@git.corp.criteo.com:qa/moab.git
10:10:31 cbs tag HEAD build/current

45. The MOAB pipeline

46. The sandboxes
a datacenter
a sandbox

47. How we deploy
datacenter
1. scp
2. bittorrent

48. How we deploy

49. Automatic metrics checks

50. cost of change
number of devs *
Conclusion
* including 5% working full time on engineering tools

51. We’re not the only ones

52. Mandatory “we’re hiring” slide

53. Questions ?