Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous delivery in practice (public)

915 views

Published on

10 practical field-proven tips for building a continuously delivered service, based on Kenshoo's experience with its RTB service - a critical, high throughput, highly available component, serving millions of requests per minute in under 50 milliseconds.

From coding practices to test automation, from monitoring tools to feature A/B testing - the entire development chain should be focused around removing blockers and manual steps between your code and your clients, without ever settling for quality. Join to see what makes our clients and developers happy and effective.

Published in: Software, Technology
  • Be the first to comment

  • Be the first to like this

Continuous delivery in practice (public)

  1. 1. Continuous Delivery In Practice Lessons from Kenshoo’s RTB project
  2. 2. Who, What, Where Tzach Zohar: ● System Architect ● tzach.zohar@kenshoo.com Kenshoo: ● Founded 2006 ● Online Marketing Technology ● ~450 employees ● 12 World Wide locations
  3. 3. Continuous Delivery: Why bother? Faster development, delivery and feedback makes our clients and us happier. The “us” alone is worth it.
  4. 4. Continuous Delivery: Why bother? Better suited product Responsiveness Less waste Higher quality
  5. 5. Background: RTB Project ● ~1.5 years ● ~4 developers ● ~Dozens of paying clients ● ~50 servers (AWS) ● ~1.5M requests per minute ● ~7ms average response time ● ~99.9% availability ● ~5-10 deployments / week
  6. 6. How?
  7. 7. 1. The Obvious ● Single branch ● Full, Fast, Reliable coverage ● Full deployment automation ● Fast feedback ● ABCD - Always Be Continuously Deploying
  8. 8. ● Unit: complete functional coverage ● Integration: with external systems - thin! ● Behavioral: we use Cucumber ● Staging: verify actual server upgrade 2. Four-Layer Test Suite
  9. 9. 2. Four-Layer Test Suite
  10. 10. Staging: verify compatibility of new build with other components’ production builds 2. Four-Layer Test Suite
  11. 11. 3. Keep Builds Stable Do not overlook a test that “sometimes fails”, trusting build status is crucial
  12. 12. 3. Keep Builds Stable Be Suspicious of: ● Random data tests ● Asynchronous tests ● Integration tests
  13. 13. 4. Master Is Always Shippable On every commit? Not Quite: Use GitHub Pull Requests “Merge” == Build and Deploy credit: tal.salmona@kenshoo.com
  14. 14. 5. Rigorous Code Reviews ● Remember “merge” means “deploy”! ● Insist on proper coverage ● Insist on code cleanliness ● Insist on consistent design ● Insist!
  15. 15. 5. Rigorous Code Reviews https://github.com/tzachz/github-comment-counter
  16. 16. 6. Real-Time Feedback Detect issues immediately and visually
  17. 17. 7. Keep Upgrade in Mind (1) Use the “Parallel Change” pattern when changing cross-node APIs / Data 1. Write: old Read: both 2. Write: new Read: both 3. Write: new Read: new deploy deploy
  18. 18. 8. Keep Upgrade in Mind (2) Verify backward compatibility in tests
  19. 19. 9. A/B Testing Apply new features to a limited user-group Measure business results per-group (Not by branching)
  20. 20. 10. Own It Constantly check builds Constantly collect feedback Constantly check monitors Answer the phone at 3am
  21. 21. 10. Own It
  22. 22. That’s It.
  23. 23. Appendix A. Partial Tool List Testing: JUnit, Cucumber, Nose Build / CI: Jenkins, Gradle, JaCoCo Code Review: GitHub Provisioning: Puppet Deployment: Fabric, boto Monitoring: Metrics, Graphite
  24. 24. Appendix B. Are You Ready? Unit Coverage > 90%? Good Staging Tests? Informative Monitors? Builds Are Kept Green? No API Breaking Changes? Rigorous Code Reviews? Support Has Your Phone Number? Do You Own it? Not Ready No Yes credit: tal.salmona@kenshoo.com
  25. 25. Thanks. Questions?

×