The document discusses continuous delivery (CD) principles and practices within Kenshoo's real-time bidding (RTB) project, emphasizing its importance in delivering incremental changes to users. It outlines a practical approach to CD, including ten field-tested tips such as full deployment automation, rigorous code reviews, and maintaining stable builds. The document also highlights the technical and organizational setup of the RTB project, which involved a small team and a robust infrastructure to achieve high availability and responsiveness.

1. Continuous Delivery
In Practice
Lessons from Kenshoo’s RTB project

2. Who, What, Where
Tzach Zohar:
● System Architect
● tzach.zohar@kenshoo.com
Kenshoo:
● Founded 2006
● Online Marketing Technology
● >500 employees
● 12 World Wide locations

3. Agenda
● Continuous Delivery: What? Why?
● RTB Project
● How: 10 Field Tested Tips
● The Process
● Appendices

4. Continuous Delivery:
Definition(s)
“Continuous Delivery (CD) is a design
practice …blah blah blah… Techniques such as
automated testing, continuous integration …blah
blah blah... resulting in the ability to rapidly, reliably and
repeatedly push out enhancements ...blah blah blah.”
- Wikipedia

5. Continuous Delivery:
Definition(s)
TL;DR

6. Continuous Delivery:
Definition(s)
“Continuous delivery is a set of principles
and practices to reduce the cost, time, and
risk of delivering incremental changes to
users.”
- Jez Humble

7. Continuous Delivery:
Definition(s)
“Continuous Delivery is a software
development discipline where you build
software in such a way that the software can
be released to production at any time”
- Martin Fowler

8. Continuous Delivery:
Why bother?
“Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software”
First principle of the Agile Manifesto

9. Continuous Delivery:
Why bother?
Better suited product
Responsiveness
Less waste
Higher quality
Simplicity
Recommended Further Reading on ThoughtWorks

10. Background: RTB Project
● ~1.5 years
● ~3 developers, 1 PM, 0.5 Ops (no QA)
● ~Dozens of paying clients
● ~50 servers (AWS)
● ~1.5M requests per minute
● ~7ms average response time
● ~99.9% availability

11. Background: RTB Project
Frontend Cluster
Highly available, high throughput ~20 node cluster
Backend
Single node,
internal APIs
FBX
Facebook RTB API
Reporting Cluster
Elastic Map Reduce (EMR)
on-demand 16-node cluster
Cassandra Cluster
Highly available, high throughput ~24 node cluster
S3
Raw traffic logs

12. Background: RTB Project
~5-10 deployments / week

13. How?

14. 1.
The Obvious
● Single branch (details later)
● Full, Fast, Reliable coverage
● Full deployment automation
● Fast feedback
● ABCD - Always Be Continuously
Deploying

15. ● Unit: complete functional coverage
● Integration: with external systems - thin!
● Behavioral: we use Cucumber
● Staging: verify actual server upgrade
2.
Four-Layer Test Suite

16. 2.
Four-Layer Test Suite

17. Staging: verify compatibility of new build
with other components’ production builds
2.
Four-Layer Test Suite

18. 3.
Keep Builds Stable
Do not overlook a test that “sometimes
fails”, trusting build status is crucial

19. 3.
Keep Builds Stable
● Random data tests
● Asynchronous tests
● Integration tests
Be suspicious of:

20. 4.
Master Is Always Shippable
On every commit? Not Quite
We follow the “GitHub Flow”:
Local Master
Local Feature Branch
Master
Feature Branch
1. pull
3. push
2.
checkout
4. Merge

21. 4.
Master Is Always Shippable
“Merge” == Build and Deploy
credit: tal.salmona@kenshoo.com

22. 5.
Rigorous Code Reviews
● Because “merge” means “deploy”!
● Insist on proper coverage
● Insist on code cleanliness
● Insist on consistent design
● Insist!

23. 5.
Rigorous Code Reviews
https://github.com/tzachz/github-comment-counter

24. 6.
Real-Time Feedback
Detect issues immediately and visually

25. 7.
Keep Upgrade in Mind (1)
Use the “Parallel Change” pattern when
changing cross-node APIs / Data
1.
Write: old
Read: both
2.
Write: new
Read: both
3.
Write: new
Read: new
deploy deploy

26. 8.
Keep Upgrade in Mind (2)
Verify backward compatibility in tests

27. 9.
A/B Testing
Apply new features to a limited user-group
Measure business results per-group
(Not by branching)

28. 9.
A/B Testing
Splitting into groups correctly is important

29. 9.
A/B Testing
It’s easy to mess up (neglecting biases,
wrong grouping, wrong comparison
methods)
This excellent talk by LivePerson’s Shlomo
Lahav helped us a lot

30. 10.
Own It
Constantly check builds
Constantly collect feedback
Constantly check monitors
Answer the phone at 3am

31. 10.
Own It

32. That’s It.

33. The Process
● Greenfield? That’s easy:
○ Start with deployment and build
○ Deploy a Hello World application
○ Every new feature is test-covered

34. The Process (RTB)
1.
Increase Unit+Integration Coverage
Create naive deployment Automation
Create monitoring
Manual Staging tests
2.
Automated staging
Downtime eradicated
Manual (but often) deployment trigger
3.
Autopilot - deploy upon commit
~ 9 Months
~ 3 Months

35. Appendix A.
Partial Tool List
Testing: JUnit, Cucumber, Nose
Build / CI: Jenkins, Gradle, JaCoCo
Code Review: GitHub
Provisioning: Puppet
Deployment: Fabric, boto
Monitoring: Metrics, Graphite

36. Appendix B.
Are You Ready?
Unit
Coverage
> 90%?
Good
Staging
Tests?
Informative
Monitors?
Builds Are
Kept
Green?
No API
Breaking
Changes?
Rigorous
Code
Reviews?
Support
Has Your
Phone
Number?
Do You
Own it?
Not
Ready
No Yes
credit: tal.salmona@kenshoo.com

37. Thanks.
Questions?