Continues Deployment - Tech Talk week

2,655 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,655
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
23
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Continues Deployment - Tech Talk week

  1. 1. Continuous Deployment @rantav @outbrain
  2. 2. How much time does it take your code to meet the users?
  3. 3. How much time does it take your code to meet the users? <ul><ul><li>More than a year? </li></ul></ul>
  4. 4. How much time does it take your code to meet the users? <ul><ul><li>More than a year? </li></ul></ul><ul><ul><li>6 - 12 Months? </li></ul></ul>
  5. 5. How much time does it take your code to meet the users? <ul><ul><li>More than a year? </li></ul></ul><ul><ul><li>6 - 12 Months? </li></ul></ul><ul><ul><li>1- 6 months? </li></ul></ul>
  6. 6. How much time does it take your code to meet the users? <ul><ul><li>More than a year? </li></ul></ul><ul><ul><li>6 - 12 Months? </li></ul></ul><ul><ul><li>1- 6 months? </li></ul></ul><ul><ul><li>2 - 4 weeks? </li></ul></ul>
  7. 7. How much time does it take your code to meet the users? <ul><ul><li>More than a year? </li></ul></ul><ul><ul><li>6 - 12 Months? </li></ul></ul><ul><ul><li>1- 6 months? </li></ul></ul><ul><ul><li>2 - 4 weeks? </li></ul></ul><ul><ul><li>1 - 14 Days?  </li></ul></ul>
  8. 8. How much time does it take your code to meet the users? <ul><ul><li>More than a year? </li></ul></ul><ul><ul><li>6 - 12 Months? </li></ul></ul><ul><ul><li>1- 6 months? </li></ul></ul><ul><ul><li>2 - 4 weeks? </li></ul></ul><ul><ul><li>1 - 14 Days? </li></ul></ul><ul><ul><li>1 - 24 Hours? </li></ul></ul>
  9. 9. How much time does it take your code to meet the users? <ul><ul><li>More than a year? </li></ul></ul><ul><ul><li>6 - 12 Months? </li></ul></ul><ul><ul><li>1- 6 months? </li></ul></ul><ul><ul><li>2 - 4 weeks? </li></ul></ul><ul><ul><li>1 - 14 Days? </li></ul></ul><ul><ul><li>1 - 24 Hours? </li></ul></ul><ul><ul><li>  < 12 Minutes?  </li></ul></ul>
  10. 10. ~ $ svn ci -m &quot;Implement the super-sharp image scaler. #deloy:ImageServer #to:ny &quot;
  11. 11. Outbrain enables readers to discover the most interesting , relevant and timely links to stories (paid and organic)
  12. 13. Multi Billion Page Views per month Note: Outbrain is typically installed on *every* article/blog post on each of these sites, immediately under the content.
  13. 14. Who's in?
  14. 15. WHY
  15. 16. WHY For the Business For Fun
  16. 17. WHY What was so bad before ?
  17. 18. WHY <ul><li>What is a startup? </li></ul><ul><ul><li>A startup is always on a quest to find Product-Market fit </li></ul></ul><ul><ul><li>=> You have to iterate Fast </li></ul></ul><ul><ul><li>... before running out of money </li></ul></ul>
  18. 19. WHY <ul><ul><li>Feedback loop speed is important (see REPL in programming) </li></ul></ul>
  19. 20. WHY <ul><li>What's used to be the case before: </li></ul><ul><ul><li>Inefficient Waits (wait for QA, wait for other features etc) </li></ul></ul><ul><ul><li>Inefficient Context Switch </li></ul></ul><ul><ul><li>Feature Delayed - b/c of other features </li></ul></ul><ul><ul><li>Big Changes – Big Problems. </li></ul></ul>
  20. 21. HOW
  21. 22. HOW
  22. 23. HOW Continuous Deployment Themes <ul><ul><li>Release is a marketing decision. </li></ul></ul><ul><ul><li>Deployment is an engineering decision! </li></ul></ul>
  23. 24. HOW Continuous Deployment Themes <ul><ul><li>Small Changes – Reduce Risks </li></ul></ul><ul><ul><li>Kent Beck: You can spill a bucket but you can't spill a Hose.  </li></ul></ul>
  24. 25. HOW Continuous Deployment Themes <ul><ul><li>Deploy fast, release often </li></ul></ul><ul><ul><li>Fast turnarounds lead to happy customers </li></ul></ul><ul><ul><li>... and happy developers </li></ul></ul>
  25. 26. HOW Culture <ul><ul><li>Everyone have to care about everything ! </li></ul></ul><ul><ul><ul><li>build </li></ul></ul></ul><ul><ul><ul><li>tests </li></ul></ul></ul><ul><ul><ul><li>quality </li></ul></ul></ul><ul><ul><ul><li>production </li></ul></ul></ul><ul><ul><ul><li>monitoring </li></ul></ul></ul><ul><ul><ul><li>business </li></ul></ul></ul>
  26. 27. HOW Culture <ul><ul><li>No Broken Windows! </li></ul></ul><ul><li>Broken Windows Theory </li></ul>
  27. 28. HOW Culture “ What's the worse that could happen? ”
  28. 29. WORKFLOW
  29. 30. WORKFLOW Dev
  30. 31. WORKFLOW Dev
  31. 32. The  INGREDIENTS
  32. 33. INGREDIENTS Trunk Stable <ul><ul><li>Everyone run tests before commit </li></ul></ul><ul><ul><li>No branches, really </li></ul></ul><ul><ul><li>Trunk may get released any moment. If you commit now, users will see it really soon. </li></ul></ul><ul><li>     => Test your code, really good </li></ul><ul><ul><li>Use feature flags (but only if you must) </li></ul></ul><ul><ul><li>Forward and backward compatibility </li></ul></ul>
  33. 34. INGREDIENTS Automated Tests <ul><ul><li>We have about 2000 test cases </li></ul></ul><ul><ul><li>They run in < 4 minutes </li></ul></ul><ul><ul><li>We use TeamCity </li></ul></ul><ul><ul><li>We regularly test production </li></ul></ul><ul><li>Desired state: No manual QA whatsoever. </li></ul>
  34. 35. INGREDIENTS Infrastructure Automation <ul><li>&quot; Infrastructure as code &quot; </li></ul><ul><ul><li>Using kickstart script to install OS and Chef agent. </li></ul></ul><ul><ul><li>All infrastructure is deployed by Chef . </li></ul></ul><ul><ul><li>All apps are deployed by Glu </li></ul></ul><ul><ul><li>All scripts, and chef cookbooks, glu configs are SVNed. </li></ul></ul><ul><ul><li>Very easy to deploy large number of machines. </li></ul></ul><ul><li>  </li></ul>
  35. 36. INGREDIENTS Deployer
  36. 40. INGREDIENTS Servicization <ul><ul><li>At outbrain we have ~ 25 services </li></ul></ul><ul><ul><li>Each service is deployed at > 1 server </li></ul></ul><ul><ul><li>If there's damage, it's contained </li></ul></ul><ul><ul><li>It's easy to make small changes </li></ul></ul><ul><ul><li>If there's an error it's easy to find it </li></ul></ul><ul><ul><li>If rollback is needed it's easier </li></ul></ul><ul><ul><li>Everything is either proxies (HAProxy) or queued (AMQ) </li></ul></ul><ul><ul><li>Challenge: It's sometimes hard to find the right balance: </li></ul></ul><ul><ul><ul><li>number of services that need maintanance </li></ul></ul></ul><ul><ul><ul><li>complexity </li></ul></ul></ul><ul><ul><ul><li>performance </li></ul></ul></ul><ul><ul><ul><li>api conformance </li></ul></ul></ul>
  37. 41. INGREDIENTS The Immune System <ul><ul><li>Coding </li></ul></ul><ul><ul><ul><li>Review </li></ul></ul></ul><ul><ul><ul><li>Testing </li></ul></ul></ul><ul><ul><li>Monitoring </li></ul></ul><ul><ul><ul><li>We use Nagios for the vitals </li></ul></ul></ul><ul><ul><ul><li>Regularly check servers via instrumentation </li></ul></ul></ul><ul><ul><ul><li>KeyNote </li></ul></ul></ul><ul><ul><ul><li>Test Production (selenium,APT) </li></ul></ul></ul><ul><ul><li>Instrumentation </li></ul></ul><ul><ul><ul><li>Self-Test </li></ul></ul></ul><ul><ul><ul><li>Performance  </li></ul></ul></ul><ul><ul><li>Monitoring KPIs </li></ul></ul><ul><ul><ul><li>PVs,  </li></ul></ul></ul><ul><ul><ul><li>3 different CTRs, </li></ul></ul></ul><ul><ul><ul><li>clicks, revenue, RPM </li></ul></ul></ul><ul><ul><ul><li>etc... </li></ul></ul></ul>
  38. 42. INGREDIENTS The Immune System <ul><ul><li>The line of defense will always be broken </li></ul></ul><ul><li>=> Multiple lines of defense </li></ul>
  39. 43. INGREDIENTS Visibilty <ul><ul><li>Everyone monitor the services  </li></ul></ul><ul><ul><li>When there's a deployment, you see it on the graph </li></ul></ul>
  40. 44. INGREDIENTS Visibilty <ul><ul><li>Yammer </li></ul></ul>
  41. 45. INGREDIENTS Visibilty <ul><ul><li>svn changelog </li></ul></ul><ul><ul><li>glu changelog </li></ul></ul><ul><ul><li>glu audit log </li></ul></ul>
  42. 46. INGREDIENTS Learn Fast, Adapt Fast <ul><ul><li>5 Ys </li></ul></ul>
  43. 47. ARSENAL Outbrain's arsenal <ul><li>Architecture: </li></ul><ul><ul><li>Java in most parts, Spring for the most and Struts2 </li></ul></ul><ul><ul><li>SOA (dah...) with REST </li></ul></ul><ul><ul><li>Message queues (ActiveMQ) </li></ul></ul><ul><ul><li>Service multiplicity with HAProxy </li></ul></ul><ul><ul><li>Data stored in: </li></ul></ul><ul><ul><ul><li>MySQL, Cassandra, HDFS, NFS </li></ul></ul></ul><ul><ul><li>  Caching: Memcached </li></ul></ul>
  44. 48. ARSENAL Outbrain's arsenal <ul><li>Testing: </li></ul><ul><ul><li>JUnit </li></ul></ul><ul><ul><li>TeamCity </li></ul></ul><ul><ul><li>Selenium </li></ul></ul><ul><ul><li>Staging environment (for high risk deployments) </li></ul></ul>
  45. 49. ARSENAL Outbrain's arsenal <ul><li>Deployment: </li></ul><ul><ul><li>Kickstart </li></ul></ul><ul><ul><li>Chef </li></ul></ul><ul><ul><li>Glu </li></ul></ul><ul><ul><li>TeamCity </li></ul></ul><ul><ul><li>In-house Deployment Manager </li></ul></ul><ul><ul><li>RPM + YUM </li></ul></ul>
  46. 50. ARSENAL Outbrain's arsenal <ul><li>Monitoring: </li></ul><ul><ul><li>Nagios </li></ul></ul><ul><ul><li>&quot;Testing production&quot; every 10 minutes.  </li></ul></ul><ul><ul><li>Keynote </li></ul></ul>
  47. 51. ARSENAL Outbrain's arsenal <ul><li>Communications: </li></ul><ul><ul><li>Yammer </li></ul></ul>
  48. 52. Fun Numbers <ul><ul><li>5-50 production changes a day!!! </li></ul></ul><ul><ul><li>More then 2000 code tests running in less then 4 minutes. </li></ul></ul><ul><ul><li>More then 600 production services tests runs every 10 minutes. </li></ul></ul><ul><ul><li>It takes ~30 minutes from code complete to ~100 machines deployed. </li></ul></ul>
  49. 53. References <ul><ul><li>Why Continuous Deployment   / Eric Ries </li></ul></ul><ul><ul><li>Continuous Deployment at outbrain   / Ran Tavory   </li></ul></ul><ul><ul><li>Deployment Infrastructure for Continuous Deployment / WealthFront </li></ul></ul><ul><ul><li>Continuous Deployment presentation / Eishay Smith </li></ul></ul><ul><ul><li>Quantum of Deployment   / Etsy </li></ul></ul><ul><ul><li>Chrome Release Cycle   / Anthony Laforge </li></ul></ul>

×