How we implement a release process for Spotify clients that allows for releasing predictably with quality every two weeks.

1. Shipping for
mobile at Spotify
How we implement a release process for Spotify clients that allows for releasing
predictably with quality every two weeks.
Henrik Österdahl Djurfelter, henka@spotify.com (PO, Android Client
infrastructure, release publishing and test infrastructure)

2. About me
Background in JVM technology
Various positions at Spotify, including development
Currently leading Android client infrastructure squad

3. About Spotify clients
Clients exist for Android, iOS, Mac OS X, Windows
Key characteristics: Platform-native UI logic, most views backend-controlled,
shares some C++ code between clients
Clients exist for embedded and other smaller platforms

4. Releases Android Feb 2015-May 2016
Started enforcing
cancellation policy

5. Release awareness

6. Agenda
Predictability trumps feature set
Client releases
Following up on quality
Development tooling

7. Predictability
trumps
feature set

8. Predictability trumps feature set
Very important insight below:
Our belief: Need to choose two of predictable delivery date,
feature set, quality
We chose predictable delivery date and quality

9. Client
releases

10. Client releases - topics
Bi-weekly release trains: Our way of releasing predictably on time
Release types: Early feedback means early fixes, less surprises
Release blocking bugs: Prioritizing between quality and feature set

11. Bi-weekly release trains
Spotify clients are cut on a fixed schedule: FC every two weeks
Schedule determines feature set: Complete features get enabled
Release branch accepts fixes for blockers, only
Goal is enabling true Continuous Delivery: Allow releasing master

12. Release types
Nightly: Distributed to employees everyday
Beta: Distributed to select users after FC, before live release
Production / Live: THE release, to all users

13. Release blocking bugs
Release process defines three categories of bugs
P1: Blocking dev and/or testing, fix now
P2: Blocking release, fix now but after P1
Other: Non-release blocking, fix at your convenience
Bug reports can come from anywhere
Manual testing, automated testing, employees in general

14. Release blocking bugs (cont’d.)
Fixes for P1 or P2 bugs go to the release branch
Because they are release blocking, should have associated regression tests
Fixes for lower prio bugs go to the master branch
Want to keep risk on release branch as small as possible
Don’t game the system!

15. Pro Tip 1: Prioritizing work
1. Deal with live incidents
2. Deal with blocking bugs on the release branch
3. Deal with blocking bugs on master branch
4. Other development work and lower prio bugs
Optimize for product success rather than individual
development team success!

16. Pro Tip 2: Know your bug situation
Monitor bugs in your features
Monitor tests for your features
Triage and plan daily

17. Following
up on
quality

18. Crash monitoring
We monitor crashes for all three release stages
Top crashes are likely to show up in bigger numbers as
we hit each release stage → report and fix early!
Report and fix top crashes for every release, to make sure
to improve over time

19. Development
tooling

20. Development tooling
GitHub Enterprise and TeamCity
GitHub Enterprise hosts source code and pull requests
Merging allowed only after CI test runs report green and manual reviewer accepts
Investigate red results, don’t blindly retrigger. Accept no flakiness!
Build system performance
No queues
PR builds are quick, <10 min for Android

21. Questions?