Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Delivery in Practice (extended)

1,162 views

Published on

Extended version of a previously uploaded presentation:

10 practical field-proven tips for building a continuously delivered service, based on Kenshoo's experience with its RTB service - a critical, high throughput, highly available component, serving millions of requests per minute in under 50 milliseconds.

From coding practices to test automation, from monitoring tools to feature A/B testing - the entire development chain should be focused around removing blockers and manual steps between your code and your clients, without ever settling for quality. Join to see what makes our clients and developers happy and effective.

Published in: Software
  • Be the first to comment

Continuous Delivery in Practice (extended)

  1. 1. Continuous Delivery In Practice Lessons from Kenshoo’s RTB project
  2. 2. Who, What, Where Tzach Zohar: ● System Architect ● tzach.zohar@kenshoo.com Kenshoo: ● Founded 2006 ● Online Marketing Technology ● >500 employees ● 12 World Wide locations
  3. 3. Agenda ● Continuous Delivery: What? Why? ● RTB Project ● How: 10 Field Tested Tips ● The Process ● Appendices
  4. 4. Continuous Delivery: Definition(s) “Continuous Delivery (CD) is a design practice …blah blah blah… Techniques such as automated testing, continuous integration …blah blah blah... resulting in the ability to rapidly, reliably and repeatedly push out enhancements ...blah blah blah.” - Wikipedia
  5. 5. Continuous Delivery: Definition(s) TL;DR
  6. 6. Continuous Delivery: Definition(s) “Continuous delivery is a set of principles and practices to reduce the cost, time, and risk of delivering incremental changes to users.” - Jez Humble
  7. 7. Continuous Delivery: Definition(s) “Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time” - Martin Fowler
  8. 8. Continuous Delivery: Why bother? “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software” First principle of the Agile Manifesto
  9. 9. Continuous Delivery: Why bother? Better suited product Responsiveness Less waste Higher quality Simplicity Recommended Further Reading on ThoughtWorks
  10. 10. Background: RTB Project ● ~1.5 years ● ~3 developers, 1 PM, 0.5 Ops (no QA) ● ~Dozens of paying clients ● ~50 servers (AWS) ● ~1.5M requests per minute ● ~7ms average response time ● ~99.9% availability
  11. 11. Background: RTB Project Frontend Cluster Highly available, high throughput ~20 node cluster Backend Single node, internal APIs FBX Facebook RTB API Reporting Cluster Elastic Map Reduce (EMR) on-demand 16-node cluster Cassandra Cluster Highly available, high throughput ~24 node cluster S3 Raw traffic logs
  12. 12. Background: RTB Project ~5-10 deployments / week
  13. 13. How?
  14. 14. 1. The Obvious ● Single branch (details later) ● Full, Fast, Reliable coverage ● Full deployment automation ● Fast feedback ● ABCD - Always Be Continuously Deploying
  15. 15. ● Unit: complete functional coverage ● Integration: with external systems - thin! ● Behavioral: we use Cucumber ● Staging: verify actual server upgrade 2. Four-Layer Test Suite
  16. 16. 2. Four-Layer Test Suite
  17. 17. Staging: verify compatibility of new build with other components’ production builds 2. Four-Layer Test Suite
  18. 18. 3. Keep Builds Stable Do not overlook a test that “sometimes fails”, trusting build status is crucial
  19. 19. 3. Keep Builds Stable ● Random data tests ● Asynchronous tests ● Integration tests Be suspicious of:
  20. 20. 4. Master Is Always Shippable On every commit? Not Quite We follow the “GitHub Flow”: Local Master Local Feature Branch Master Feature Branch 1. pull 3. push 2. checkout 4. Merge
  21. 21. 4. Master Is Always Shippable “Merge” == Build and Deploy credit: tal.salmona@kenshoo.com
  22. 22. 5. Rigorous Code Reviews ● Because “merge” means “deploy”! ● Insist on proper coverage ● Insist on code cleanliness ● Insist on consistent design ● Insist!
  23. 23. 5. Rigorous Code Reviews https://github.com/tzachz/github-comment-counter
  24. 24. 6. Real-Time Feedback Detect issues immediately and visually
  25. 25. 7. Keep Upgrade in Mind (1) Use the “Parallel Change” pattern when changing cross-node APIs / Data 1. Write: old Read: both 2. Write: new Read: both 3. Write: new Read: new deploy deploy
  26. 26. 8. Keep Upgrade in Mind (2) Verify backward compatibility in tests
  27. 27. 9. A/B Testing Apply new features to a limited user-group Measure business results per-group (Not by branching)
  28. 28. 9. A/B Testing Splitting into groups correctly is important
  29. 29. 9. A/B Testing It’s easy to mess up (neglecting biases, wrong grouping, wrong comparison methods) This excellent talk by LivePerson’s Shlomo Lahav helped us a lot
  30. 30. 10. Own It Constantly check builds Constantly collect feedback Constantly check monitors Answer the phone at 3am
  31. 31. 10. Own It
  32. 32. That’s It.
  33. 33. The Process ● Greenfield? That’s easy: ○ Start with deployment and build ○ Deploy a Hello World application ○ Every new feature is test-covered
  34. 34. The Process (RTB) 1. Increase Unit+Integration Coverage Create naive deployment Automation Create monitoring Manual Staging tests 2. Automated staging Downtime eradicated Manual (but often) deployment trigger 3. Autopilot - deploy upon commit ~ 9 Months ~ 3 Months
  35. 35. Appendix A. Partial Tool List Testing: JUnit, Cucumber, Nose Build / CI: Jenkins, Gradle, JaCoCo Code Review: GitHub Provisioning: Puppet Deployment: Fabric, boto Monitoring: Metrics, Graphite
  36. 36. Appendix B. Are You Ready? Unit Coverage > 90%? Good Staging Tests? Informative Monitors? Builds Are Kept Green? No API Breaking Changes? Rigorous Code Reviews? Support Has Your Phone Number? Do You Own it? Not Ready No Yes credit: tal.salmona@kenshoo.com
  37. 37. Thanks. Questions?

×