What's TBD, some facts and how TBD with feature toggle can increase release frequency, lower release risk, decouple release from code deployment. DevOpsDays, Taipei, 2017

1. 百倍速交付
~ 談Trunk-Based Development
Bryan Liu 劉栢峰 2017

2. Some Definitions

3. Long-Lived (Feature) Branch
• Creates lots of overhead
Git Flow
GitHub Flow

4. Short-Lived Feature Branch
• Branch’s length of life: less than 2 days
• Shared for code-review and CI only
• Easily becomes long-lived branch
https://trunkbaseddevelopment.com/short-lived-feature-branches/

5. Trunk-Based Development (TBD)
• Commit in trunk at least once a day
• Trunk is always in a releasable state
• Hide unfinished code with feature toggle
• Refactor with branch by abstraction
http://www.alwaysagileconsulting.com/articles/organisation-pattern-trunk-based-development/

6. Facts

7. Always the bridesmaid* from Paul Hammant

8. TBD
Only search few of these kings…
Also other TBD giants, ex: Google, Netflix
were not included in this startups research.
Never mind become A Category King
2000 ~ 2015 yr
600k companies
34 of them
76 % market cap
Winner-takes-all
Category King

9. ‘‘Somewhere in the middle of this two-month process you’re going to run
across a bridge and burn it behind you.’’
‘‘LinkedIn’s big switch to continuous deployment has been linked to very
concrete and visible financial success…’’
~ Kevin Scott, the senior VP of engineering
http://www.wired.com/2013/04/linkedin-software-revolution/

10. https://paulhammant.com/2013/03/13/facebook-tbd-take-2/
Facebook’s
Trunk-Based Development
branching models
v.s.
release frequency

11. TBD & Feature Toggle

12. if ( ToggleManager.isEnabled(“FastCheckout”, user) ) {
useFastCheckout()
} else {
useOriginCheckout()
}
https://martinfowler.com/articles/feature-toggles.html
What’s Toggle

13. Business Toggle:
• A/B Testing
Low Risk Release and Zero-Downtime
• Canary Release, Dark Launch, Blue-Green deployment
• DB/Schema Migration
Decouple Feature Rollout from Code Deployment
• Re-planning
• Microservices
Some Toggle Use Cases

14. Canary Release / Blue-Green Deployment & A / B Testing
http://featureflags.io/
• Phased rollout
• Monitor performance
• Check user feedback

15. • Features in live but NOT visible to user
• Warm up cache, load DB tables with real
production traffic
• Soak test, verify performance
Dark Launch

16. DB / Schema Change
• Forwards & backwards compatible
• Stage roll-out with verification online
• No downtime needed

17. Q:
What are the requirements
to achieve previous scenarios ?
A:
• Feature toggles &
• Old and new code co-exists in the same delivery
“Get good with build-flags and
toggles” ~ Paul Hammant
“ This is also the major difference
between short-lived branch and TBD!”

18. Inefficiencies ...

19. Decouple Release from Code Deployment
• TBD reduce merge pain
• Merge hell is error prone and risky
• Inefficiency on spending time in bug fix - merge - test cycle.

20. As in Microservices:
Any attempt trying to set release pace is not realistic!!
Feature flags
+ Microservices
= True decoupling

21. Re-planning
• It happens
• Leads to weeks of un-merge for teams
Image from: https://devops.paulhammant.com/

22. Inefficiencies in SW development:
• It slows down throughput
• Can’t keep talent

23. Spotify Release Train
https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/

24. Frequently commits to mainline:
• More communication with your team
• More frequent builds and tests (CI)
• More frequent deployment (CD)
• More experiments and faster feedback

25. Implementation

26. Pre-Commit Testing & Code Review
• Pre-commit (CI) test (option but best)
• Continuous (code) review
• Code ownership
Figure from https://trunkbaseddevelopment.com/continuous-integration/

27. Why Continuous Review (Large Diffs Hurts)
• Large diffs won’t get reviewed
• Reverting large diffs is hard
• Rebasing/merging large diffs is error-prone
Code review statistics: http://blogs.atlassian.com/2011/07/creating_optimal_reviews/

28. Pull Request Code Review
git commit -m "fixed issue with fan"

29. https://zachholman.com/posts/deploying-software

30. Code Review Page
• Code statistics
• CI test results
• Review discussion &
comments

33. Hello, about me …
DevOps Evangelist
QA Architect
Bryan Liu 劉栢峰

34. • trunkbaseddevelopment.com
• trunk-based development, Paul Hammant
• Facebook's Trunk-Based Development, Paul Hammant
• Branch by Abstraction, Martin Fowler
• Feature Toggles, martinfowler.com
• Category King, J. Van Grove and Play Bigger
• The Death of Continuous Integration, Steve Smith
• And many links inside each slide deck ~
Reference