The document discusses continuous deployment practices at Outbrain, an online content recommendation company. It emphasizes the importance of short feedback loops between code changes and user exposure through practices like deploying new code multiple times daily and testing code changes automatically before deployment. Infrastructure is codified and deployment is automated using tools like Chef to further streamline the process.

1. Continuous Deployment @rantav @outbrain

2. How much time does it take your code to meet the users?

3. How much time does it take your code to meet the users? More than a year?

4. How much time does it take your code to meet the users? More than a year? 6 - 12 Months?

5. How much time does it take your code to meet the users? More than a year? 6 - 12 Months? 1- 6 months?

6. How much time does it take your code to meet the users? More than a year? 6 - 12 Months? 1- 6 months? 2 - 4 weeks?

7. How much time does it take your code to meet the users? More than a year? 6 - 12 Months? 1- 6 months? 2 - 4 weeks? 1 - 14 Days?

8. How much time does it take your code to meet the users? More than a year? 6 - 12 Months? 1- 6 months? 2 - 4 weeks? 1 - 14 Days? 1 - 24 Hours?

9. How much time does it take your code to meet the users? More than a year? 6 - 12 Months? 1- 6 months? 2 - 4 weeks? 1 - 14 Days? 1 - 24 Hours? < 12 Minutes?

10. ~ $ svn ci -m &quot;Implement the super-sharp image scaler. #deloy:ImageServer #to:ny &quot;

11. Outbrain enables readers to discover the most interesting , relevant and timely links to stories (paid and organic)

13. Multi Billion Page Views per month Note: Outbrain is typically installed on *every* article/blog post on each of these sites, immediately under the content.

14. Who's in?

15. WHY

16. WHY For the Business For Fun

17. WHY What was so bad before ?

18. WHY What is a startup? A startup is always on a quest to find Product-Market fit => You have to iterate Fast ... before running out of money

19. WHY Feedback loop speed is important (see REPL in programming)

20. WHY What's used to be the case before: Inefficient Waits (wait for QA, wait for other features etc) Inefficient Context Switch Feature Delayed - b/c of other features Big Changes – Big Problems.

21. HOW

22. HOW

23. HOW Continuous Deployment Themes Release is a marketing decision. Deployment is an engineering decision!

24. HOW Continuous Deployment Themes Small Changes – Reduce Risks Kent Beck: You can spill a bucket but you can't spill a Hose.

25. HOW Continuous Deployment Themes Deploy fast, release often Fast turnarounds lead to happy customers ... and happy developers

26. HOW Culture Everyone have to care about everything ! build tests quality production monitoring business

27. HOW Culture No Broken Windows! Broken Windows Theory

28. HOW Culture “ What's the worse that could happen? ”

29. WORKFLOW

30. WORKFLOW Dev

31. WORKFLOW Dev

32. The INGREDIENTS

33. INGREDIENTS Trunk Stable Everyone run tests before commit No branches, really Trunk may get released any moment. If you commit now, users will see it really soon. => Test your code, really good Use feature flags (but only if you must) Forward and backward compatibility

34. INGREDIENTS Automated Tests We have about 2000 test cases They run in < 4 minutes We use TeamCity We regularly test production Desired state: No manual QA whatsoever.

35. INGREDIENTS Infrastructure Automation &quot; Infrastructure as code &quot; Using kickstart script to install OS and Chef agent. All infrastructure is deployed by Chef . All apps are deployed by Glu All scripts, and chef cookbooks, glu configs are SVNed. Very easy to deploy large number of machines.

36. INGREDIENTS Deployer

40. INGREDIENTS Servicization At outbrain we have ~ 25 services Each service is deployed at > 1 server If there's damage, it's contained It's easy to make small changes If there's an error it's easy to find it If rollback is needed it's easier Everything is either proxies (HAProxy) or queued (AMQ) Challenge: It's sometimes hard to find the right balance: number of services that need maintanance complexity performance api conformance

41. INGREDIENTS The Immune System Coding Review Testing Monitoring We use Nagios for the vitals Regularly check servers via instrumentation KeyNote Test Production (selenium,APT) Instrumentation Self-Test Performance Monitoring KPIs PVs, 3 different CTRs, clicks, revenue, RPM etc...

42. INGREDIENTS The Immune System The line of defense will always be broken => Multiple lines of defense

43. INGREDIENTS Visibilty Everyone monitor the services When there's a deployment, you see it on the graph

44. INGREDIENTS Visibilty Yammer

45. INGREDIENTS Visibilty svn changelog glu changelog glu audit log

46. INGREDIENTS Learn Fast, Adapt Fast 5 Ys

47. ARSENAL Outbrain's arsenal Architecture: Java in most parts, Spring for the most and Struts2 SOA (dah...) with REST Message queues (ActiveMQ) Service multiplicity with HAProxy Data stored in: MySQL, Cassandra, HDFS, NFS Caching: Memcached

48. ARSENAL Outbrain's arsenal Testing: JUnit TeamCity Selenium Staging environment (for high risk deployments)

49. ARSENAL Outbrain's arsenal Deployment: Kickstart Chef Glu TeamCity In-house Deployment Manager RPM + YUM

50. ARSENAL Outbrain's arsenal Monitoring: Nagios &quot;Testing production&quot; every 10 minutes. Keynote

51. ARSENAL Outbrain's arsenal Communications: Yammer

52. Fun Numbers 5-50 production changes a day!!! More then 2000 code tests running in less then 4 minutes. More then 600 production services tests runs every 10 minutes. It takes ~30 minutes from code complete to ~100 machines deployed.

53. References Why Continuous Deployment / Eric Ries Continuous Deployment at outbrain / Ran Tavory Deployment Infrastructure for Continuous Deployment / WealthFront Continuous Deployment presentation / Eishay Smith Quantum of Deployment / Etsy Chrome Release Cycle / Anthony Laforge