Continuous delivery practices were implemented for an RTB project at Kenshoo to allow for faster development and delivery. This included having a single code branch, comprehensive testing at the unit, integration, behavioral, and staging levels, keeping builds stable, ensuring the master branch is always deployable, conducting rigorous code reviews, using real-time feedback, managing upgrades carefully, and A/B testing. It is important to constantly monitor builds and collect feedback to own the process.

1. Continuous Delivery
In Practice
Lessons from Kenshoo’s RTB project

2. Who, What, Where
Tzach Zohar:
● System Architect
● tzach.zohar@kenshoo.com
Kenshoo:
● Founded 2006
● Online Marketing Technology
● ~450 employees
● 12 World Wide locations

3. Continuous Delivery:
Why bother?
Faster development, delivery and feedback
makes our clients and us happier.
The “us” alone is worth it.

4. Continuous Delivery:
Why bother?
Better suited product
Responsiveness
Less waste
Higher quality

5. Background: RTB Project
● ~1.5 years
● ~4 developers
● ~Dozens of paying clients
● ~50 servers (AWS)
● ~1.5M requests per minute
● ~7ms average response time
● ~99.9% availability
● ~5-10 deployments / week

6. How?

7. 1.
The Obvious
● Single branch
● Full, Fast, Reliable coverage
● Full deployment automation
● Fast feedback
● ABCD - Always Be Continuously
Deploying

8. ● Unit: complete functional coverage
● Integration: with external systems - thin!
● Behavioral: we use Cucumber
● Staging: verify actual server upgrade
2.
Four-Layer Test Suite

9. 2.
Four-Layer Test Suite

10. Staging: verify compatibility of new build
with other components’ production builds
2.
Four-Layer Test Suite

11. 3.
Keep Builds Stable
Do not overlook a test that “sometimes
fails”, trusting build status is crucial

12. 3.
Keep Builds Stable
Be Suspicious of:
● Random data tests
● Asynchronous tests
● Integration tests

13. 4.
Master Is Always
Shippable
On every commit? Not Quite:
Use GitHub Pull Requests
“Merge” == Build and Deploy
credit: tal.salmona@kenshoo.com

14. 5.
Rigorous Code Reviews
● Remember “merge” means “deploy”!
● Insist on proper coverage
● Insist on code cleanliness
● Insist on consistent design
● Insist!

15. 5.
Rigorous Code Reviews
https://github.com/tzachz/github-comment-counter

16. 6.
Real-Time Feedback
Detect issues immediately and visually

17. 7.
Keep Upgrade in Mind (1)
Use the “Parallel Change” pattern when
changing cross-node APIs / Data
1.
Write: old
Read: both
2.
Write: new
Read: both
3.
Write: new
Read: new
deploy deploy

18. 8.
Keep Upgrade in Mind (2)
Verify backward compatibility in tests

19. 9.
A/B Testing
Apply new features to a limited user-group
Measure business results per-group
(Not by branching)

20. 10.
Own It
Constantly check builds
Constantly collect feedback
Constantly check monitors
Answer the phone at 3am

21. 10.
Own It

22. That’s It.

23. Appendix A.
Partial Tool List
Testing: JUnit, Cucumber, Nose
Build / CI: Jenkins, Gradle, JaCoCo
Code Review: GitHub
Provisioning: Puppet
Deployment: Fabric, boto
Monitoring: Metrics, Graphite

24. Appendix B.
Are You Ready?
Unit
Coverage
> 90%?
Good
Staging
Tests?
Informative
Monitors?
Builds Are
Kept
Green?
No API
Breaking
Changes?
Rigorous
Code
Reviews?
Support
Has Your
Phone
Number?
Do You
Own it?
Not
Ready
No Yes
credit: tal.salmona@kenshoo.com

25. Thanks.
Questions?