The document compares different large-scale open source development models including Linux kernel, Apache Software Foundation, Debian, OpenStack, and Docker. It discusses factors like growth rate, culture, governance structure, team scaling approach, release processes, and tools. The Linux kernel uses a decentralized model with subsystem maintainers. Apache Foundation uses a flat trust model across many projects. Debian emphasizes individual maintainers. OpenStack uses centralized review and Docker relies on maintainers across multiple repos with automated testing. There is no single best model and projects must consider goals, community, and challenges around consensus, ownership and scaling.

1. LARGE SCALE OPEN SOURCE
DEVELOPMENT MODELS
A COMPARATIVE ANALYSIS
By Joe Gordon

3. WHY
Saw OpenStack grow from around 60 developers to over 2,100
developers
Unusual development model
But how do other projects solve the same problems?

4. WHY IS PICKING THE RIGHT
DEVELOPMENT MODEL
IMPORTANT?

5. ACCELERATING GROWTH
Time till 100 contributors
Linux: 1991 - 2 years
Linux 2.0 had 190 contributors in 1996 in credits
OpenStack: 2010 - 1 year
Docker: 2013 - several months
over 300 contributors in its first year
200 contributors per month
Linux: 1991 - June 2004 (13 years)
Debian: 1993 - March 2007 (14 years)
OpenStack: 2010 - October 2012 (2 years)

6. ACCELERATING GROWTH
Linux, Debian, Docker, OpenStack (clockwise from top left)
source: openhub

7. OPEN SOURCE IS BIG BUSINESS
Open source is the new standard bodies
Balancing corporate interests
Linux foundation Gold and Platinum Members

8. PICKING THE RIGHT DEVELOPMENT MODEL
CONWAY'S LAW
organizations which design systems ... are constrained
to produce designs which are copies of the
communication structures of these organizations

9. DEVELOPMENT MODELS
PROJECTS COVERED
Linux Kernel
Apache Software Foundation
Debian
OpenStack
Docker

10. LINUX KERNEL

11. LINUX KERNEL
Time based release
100-150 subsystem maintainers
Chain of trust
Decentralized review process
Rolling development model

12. CULTURE
Chain of trust
About the individual
Value frankness over politeness
Corporate friendly
No single company controls
Not much automated pre commit testing
Failing testing is very bad for author

13. APACHE SOFTWARE
FOUNDATION

14. APACHE SOFTWARE
FOUNDATION
ASF is more of a governance umbrella and culture
Separate projects
4,400+ committers
150+ top level projects
flat (ish) trust model
'Review then commit' vs. 'commit then review'
I n o r d e r t o r e d u c e f r i c t i o n a n d a l l o w f o r d i v e r s i t y t o e m e r g e , r a t h e r t h a n f o r c i n g
a m o n o c u l t u r e f r o m t h e t o p . . . e a c h p r o j e c t i s d e l e g a t e d a u t h o r i t y o v e r d e v e l o p m e n t
o f i t s s o f t w a r e , a n d i s g i v e n a g r e a t d e a l o f l a t i t u d e i n d e s i g n i n g i t s o w n t e c h n i c a l
c h a r t e r a n d i t s o w n g o v e r n i n g r u l e s .

15. CULTURE
Lazy consensus
Focus is on the team
All decisions are team based
Focus is on contributors not companies
No monoculture
Within the ASF we worry about any community which
centers around a few individuals who are working
virtually uncontested.

16. DEBIAN

17. DEBIAN
When its ready, not time based. Notoriously slow
Package Maintainers: No review, trust maintainers more

18. CULTURE
Rotating leadership (elections)
Do-ocracy: An individual Developer may make any technical or
nontechnical decision with regard to their own work
Open development
Independent not 'profit-driven': no imposed decisions by who has money,
infrastructure, people
no benevolent dictator, no oligarchy
It is all about the individual (although individual's can form groups)
Territorial

19. OPENSTACK

20. OPENSTACK
continuous delivery + stable releases
No rolling development
Break down into services and build teams around each service
5,000 commits per month from 500 contributors
Strong centralized review process (two core reviews per patch)
Automated testing to reduce reviewer burden
Having trouble with scaling the team responsible for a single repository
Can't get past 15 or so members on a core team

22. CULTURE
Group over individual
Egalitarian
Elections
Welcoming to new contributors
Corporate friendly
Not controlled by single company
Lazy consensus
Decentralized design
Uniform tooling/process across projects

23. CULTURE
OPENSTACK'S 4 OPENS
Open Source, not open core
Open Design
Open Development
Open Community
Lazy consensus
technical governance is a meritocracy
put everything in the public

24. FACTORS LIMITING GROWTH
Cross project issues
Team size
Single vision

25. DOCKER
'Github' development model

26. DOCKER
Release every 2 months from stable branch (master isn't frozen)
Scaling
split into multiple repos
Maintainers / subsystem maintainers
35+ maintainers in Docker
1 ) T h e y s h a r e r e s p o n s i b i l i t y i n t h e p r o j e c t ' s s u c c e s s .
2 ) T h e y h a v e m a d e a l o n g - t e r m , r e c u r r i n g t i m e i n v e s t m e n t t o i m p r o v e t h e p r o j e c t .
3 ) T h e y s p e n d t h a t t i m e d o i n g w h a t e v e r n e e d s t o b e d o n e , n o t n e c e s s a r i l y w h a t
i s t h e m o s t i n t e r e s t i n g o r f u n . "
T h i s " c e l l u l a r d i v i s i o n " i s t h e p r i m a r y m e c h a n i s m f o r s c a l i n g m a i n t e n a n c e o f t h e
p r o j e c t a s i t g r o w s .

27. BDFL
I d e a l l y , t h e B D F L r o l e i s l i k e t h e Q u e e n o f E n g l a n d : a w e s o m e c r o w n , b u t n o t
a n a c t u a l o p e r a t i o n a l r o l e d a y - t o - d a y . T h e r e a l j o b o f a B D F L i s t o N E V E R G O A W A Y .
. . . t h e B D F L w i l l a l w a y s b e t h e r e , p r e s e r v i n g t h e p h i l o s o p h y a n d p r i n c i p l e s o f t h e
p r o j e c t , a n d k e e p i n g u l t i m a t e a u t h o r i t y o v e r i t s f a t e . T h i s g i v e s u s g r e a t
f l e x i b i l i t y i n e x p e r i m e n t i n g w i t h v a r i o u s g o v e r n a n c e m o d e l s , k n o w i n g t h a t w e c a n
a l w a y s p r e s s t h e " r e s e t " b u t t o n w i t h o u t f e a r o f f r a g m e n t a t i o n o r d e a d l o c k . S e e t h e
U S c o n g r e s s f o r a c o u n t e r - e x a m p l e .
B D F L d a i l y r o u t i n e :
* I s t h e p r o j e c t g o v e r n a n c e s t u c k i n a d e a d l o c k o r i r r e v e r s i b l y f r a g m e n t e d ?
* I f y e s : r e f a c t o r t h e p r o j e c t g o v e r n a n c e
* A r e t h e r e i s s u e s o r c o n f l i c t s e s c a l a t e d b y c o r e ?
* I f y e s : r e s o l v e t h e m
* G o b a c k t o p o l i s h i n g t h a t c r o w n .

28. CULTURE
Embraces the BDFL
Open source
Open design
Docker the company dominates
Most of the maintainers are docker employees
Automated testing
Be Nice and Encourage diversity and participation

29. THINGS TO CONSIDER FOR
YOUR OWN PROJECT
There is no one size fits all solution

30. WHAT DOES OPEN EVEN MEAN?
Open Source
Open Core
Open Design
Open Development
Open Community

31. TOOLING CONSIDERATIONS
Bug tracking
Review process
Testing
Barrier to entry for new contributors
DCO vs. CLA
Developer Certificate of Origin
Contributor Licensing Agreement
Overall workflow

32. DEVELOPMENT MODEL COMPONENTS
Communication
Team scaling model
chain of trust model
flat trust model
maintainers
Release cadence
Stabilization periods
Rolling development
CI/CD
Number of artifacts
Decision making process -- Consensus model
Project culture

33. PROBLEMS SPECIFIC TO OPEN SOURCE
BDFL
Vision
Managing competing interests
Corporate
Consensus model
Trust and Ownership
Team vs individual
First come first serve?
How do you fire someone?

34. THANK YOU
QUESTIONS?
Slides can be found at jogo.github.io/development-models
Powered by reveal.js