1.
@mockitoguy@LinkedInEng
You deserve great tools!
Commit-to-production pipeline at LinkedIn
Agenda:
1. Continuous delivery in the Open Source - 5min
2. linkedin.com - 3 releases per day, 3 hours time from
commit to production - 15min
3. Commit to Production pipeline - 20min
4. Q&A - 5min
Szczepan Faber
● Originally from Poland, in US since 2015
● Creator of Mockito framework in 2007
● Core engineer of Gradle in 2011-2015
● Tech Lead at LinkedIn Development Tools
team since 2015
● @mockitoguy on Twitter
● Author on LinkedIn

2.
@mockitoguy@LinkedInEng
1.
Continuous Delivery in the Open Source, lessons
learned from Mockito

3.
@mockitoguy@LinkedInEng
● Project started in 2007
● Easy manual releases
2008 - Mockito 1.0

4.
@mockitoguy@LinkedInEng
Mockito popularity

5.
@mockitoguy@LinkedInEng
2014: Mockito & Continuous Delivery
● Every merged pull request lands new version in Maven Central

6.
@mockitoguy@LinkedInEng
The bright side of CD in OSS
● Productivity – zero release overhead
● Happy users – get features faster
● Faster debugging – quickly identify
bad version (MTTR)
● Sustainability – release & stay alive
● No waste - no unreleased code
● Quality – self-enforced craftsmanship
of every change
● Thriving, engaged community –
contributions are released quickly

7.
@mockitoguy@LinkedInEng
Feedback from community
● Quality anxiety
● What version to use?
● Dependency Management cost

8.
@mockitoguy@LinkedInEng
http://shipkit.org
● Mockito uses Shipkit
● Toolkit for shipping it

9.
@mockitoguy@LinkedInEng
And also:
○ Rapid innovation
○ Competitive advantage
● Productivity
● Happy users
● Faster debugging
● Sustainability
● No unreleased code
● Quality
Continuous Delivery in the enterprise

10.
@mockitoguy@LinkedInEng
2.
linkedin.com - 3 releases per day, 3 hour time from
commit to production

11.
@mockitoguy@LinkedInEng
Years before 2015
● monthly releases
● feature rush
● branch hell
● manual verification

12.
@mockitoguy@LinkedInEng
2015 at LinkedIn: 3x3
● 3 hours max time from commit to production
● 3 releases per day

13.
@mockitoguy@LinkedInEng
Continuous delivery is hard!
(what do we do if something is hard?)
● Flaky test is worse than no test
● Production-grade tests and infrastructure
● Improving Open Source test frameworks
● Need for speed – going parallel and
distributed
● Master branch always green

14.
@mockitoguy@LinkedInEng
3x3 & engineering culture @LI
● Engineering culture shift
● It’s not about technology (but it plays a role)
● Read more on LinkedIn eng blog

15.
@mockitoguy@LinkedInEng
3.
Commit to Production Pipeline

16.
@mockitoguy@LinkedInEng
Several years ago at LinkedIn
● Monolithic codebase
● We tried, but it didn’t scale
● Decision to break free from Mono

17.
@mockitoguy@LinkedInEng
Monorepo vs. Multirepo
● It’s not about how you keep the code
but how you ship it
● Monorepo vs. Multirepo
● Monolith vs. Federation

18.
@mockitoguy@LinkedInEng
Monolith vs. Federation
Monolith - central government
● All software components are developed, tested and
published at the same time
● No versioning between internal components
● Unified versions of external dependencies
Federation - autonomy within ecosystem
● Every software component owns its release cadence
● Individual versioning of software components
● Freedom to choose dependency version

19.
@mockitoguy@LinkedInEng
Federated software components
● A codebase at LinkedIn is
called “Multiproduct” (MP)
● Commit to Production in MP
is standardized
● Easy contributing to
codebases (GitHub analogy)

20.
@mockitoguy@LinkedInEng
Commit to Production in Multiproduct
● Code change
● Code review
(strong code ownership)
● Pre/post push validation
● Depender testing
● New version
Library:
● Update consumers
● Deprecate/end-of-life
previous versions
Application:
● Staging
● Canary
● Ramp-up features
● Remove experiment

21.
@mockitoguy@LinkedInEng
THANK YOU! Talks at JavaOne ‘17
You deserve great tools!!!
● Now: Commit-to-production pipeline at LinkedIn
○ This talk. Hope you liked it :)
● Tomorrow: Play on Gradle - Midtier behind linkedin.com
○ Lessons learned from driving disruptive change in large org
○ Replacing the heart of Play Framework with Gradle at scale
● Tomorrow: Mockito 2 Cleaner Tests and Enhanced Productivity
○ Do you want to write cleaner tests faster?

22.
@mockitoguy@LinkedInEng
Thank you!!! Questions?
Commit-to-production pipeline at LinkedIn
● Productivity
● Happy users
● Faster debugging
● Sustainability
● No unreleased code
● Quality
● Rapid innovation http://shipkit.org
Toolkit for shipping
it used by Mockito
3x3 Continuous delivery at LinkedIn:
● 3 hours commit to production
● 3 releases per day
● Standardized workflow
● Commit-to-production pipeline
Szczepan Faber
● Originally from Poland, in US since
2015
● Creator of Mockito framework in 2007
● Core engineer of Gradle in
2011-2015
● Tech Lead at LinkedIn Development
Tools team since 2015
● @mockitoguy on Twitter
● Author on LinkedIn
● Tomorrow: Play on Gradle - Midtier behind linkedin.com
● Tomorrow: Mockito 2 Cleaner Tests and Enhanced Productivity