How we made jenkins community

950 views

Published on

My main track session in FOSDEM 2013.

Published in: Technology
  • Be the first to comment

How we made jenkins community

  1. 1. ©2010 CloudBees, Inc. All Rights Reserved ©2010 CloudBees, Inc. All Rights Reserved How we made the Jenkins community Kohsuke Kawaguchi Creator of Jenkins / Architect, CloudBees, Inc.
  2. 2. ©2010 CloudBees, Inc. All Rights Reserved 2©2010 CloudBees, Inc. All Rights Reserved What is Jenkins? • Java web application • For builds, tests, and broader automation – Called “Continuous Integration server” • MIT-licensed, 8 years old jenkins-ci.org
  3. 3. ©2010 CloudBees, Inc. All Rights Reserved 3©2010 CloudBees, Inc. All Rights Reserved Where we are today • 53,000+ installations around the world • 600+ plugins by 500 contributors • #1 CI server in the market
  4. 4. ©2010 CloudBees, Inc. All Rights Reserved 4©2010 CloudBees, Inc. All Rights Reserved How did we come this far? • All the usual things – Software that works – Being at the right place at the right time • But I‟d like to think there‟s more to it – And it‟s something I think others can replicate
  5. 5. ©2010 CloudBees, Inc. All Rights Reserved 5©2010 CloudBees, Inc. All Rights Reserved Communication is inherently painful • Ask if the project likes the change I‟m thinking • Work on the change • Submit the patch and ask for approval • Defend the patch and revise it • Docs, tests, IP assignment • Done
  6. 6. ©2010 CloudBees, Inc. All Rights Reserved 6©2010 CloudBees, Inc. All Rights Reserved Programmers are picky bunch • All sorts of things to quarrel about – Should „{‟ be on the same line as “if”? – Different taste for design trade-off – Different anticipation for future expansions • Hardly bearable even when you know them personally
  7. 7. ©2010 CloudBees, Inc. All Rights Reserved 7©2010 CloudBees, Inc. All Rights Reserved More so in FLOSS • We don‟t know each other • No relationship outside coding • No common goals • No command structure • Time zone / language barrier
  8. 8. ©2010 CloudBees, Inc. All Rights Reserved 8©2010 CloudBees, Inc. All Rights Reserved Communication is root of all evil • Less, the better • Get rid of the need for it – Don‟t stand in the way of people and their itches – Innovations happen everywhere. Don‟t kill them before it materializes
  9. 9. ©2010 CloudBees, Inc. All Rights Reserved 9©2010 CloudBees, Inc. All Rights Reserved The Question
  10. 10. ©2010 CloudBees, Inc. All Rights Reserved 10©2010 CloudBees, Inc. All Rights Reserved It starts with modularity • It means: – Large software, divided into small silos – Module interaction controlled by contracts – Code/build/test at module level • Modularity enables extensibility
  11. 11. ©2010 CloudBees, Inc. All Rights Reserved 11©2010 CloudBees, Inc. All Rights Reserved Extensibility! • Open up to the rest of the world • The only thing that really matters – It enables all kinds of social/technical tricks • It‟s not free, but it‟s worth the effort
  12. 12. ©2010 CloudBees, Inc. All Rights Reserved 12©2010 CloudBees, Inc. All Rights Reserved #0: Programmability / Scriptability • Every software should be a library – Let other programs talk to your program – There‟s always a layer above you • “App” people tend to forget this – ImageMagick vs Gimp
  13. 13. ©2010 CloudBees, Inc. All Rights Reserved 13©2010 CloudBees, Inc. All Rights Reserved #1: Code loading at runtime • Define extension points – Contracts to be followed by extensions • Discovering implementations – Java, etc. requires some work • Provide isolation
  14. 14. ©2010 CloudBees, Inc. All Rights Reserved 14©2010 CloudBees, Inc. All Rights Reserved #2: Treat extensions as 1st class citizen • If core can do something that extensions can‟t, that‟s a bug • If users can tell apart extensions from core, you aren‟t doing enough
  15. 15. ©2010 CloudBees, Inc. All Rights Reserved 15©2010 CloudBees, Inc. All Rights Reserved #3: Data model extensibility • Extensions need to be able to store stuff – By adding stuff to core domain objects – By adding more domain objects – For Jenkins, XML really helped • Schema versioning becomes non-linear
  16. 16. ©2010 CloudBees, Inc. All Rights Reserved 16©2010 CloudBees, Inc. All Rights Reserved #4: Embrace heterogeneousness • Good contract enables multiple impls – Take something, slightly change something, and you have a new extension impl – You see a list, I see extensibility • And inheritance • This is surprisingly tricky in some context – In webapp, for example
  17. 17. ©2010 CloudBees, Inc. All Rights Reserved 17©2010 CloudBees, Inc. All Rights Reserved #5: Turtles all the way down • Extensibility needs to be recursive – Extensions should define its own extension points • In Jenkins, we provide an idiom • Partly culture • Partly refactoring
  18. 18. ©2010 CloudBees, Inc. All Rights Reserved 18©2010 CloudBees, Inc. All Rights Reserved Symptom of fake modularity/extensibility • Division of labor is not modularity • Modules are tailor-fit to their neighbors • Contracts aren‟t general enough • A fixed set of modules become the product Mosaic I - M.C. Escher
  19. 19. ©2010 CloudBees, Inc. All Rights Reserved 19©2010 CloudBees, Inc. All Rights Reserved Extensibility in Jenkins • Core – Creates structure – Includes Modules (plugins in disguise) • 600+ Plugins – Actually provide useful CI functionalities
  20. 20. ©2010 CloudBees, Inc. All Rights Reserved 20©2010 CloudBees, Inc. All Rights Reserved The Question
  21. 21. ©2010 CloudBees, Inc. All Rights Reserved 21©2010 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/jvuokko/4566270758/
  22. 22. ©2010 CloudBees, Inc. All Rights Reserved 22©2010 CloudBees, Inc. All Rights Reserved Extensible silo is a good thing • It forces contracts at boundaries – Principles of “continuous X”: do it often enough and it becomes painless • Small code is easier to hack – Looking at 5 files feel easier than 500 files – Faster to build, easier to maintain • Special use case does not clutter general- purpose code/UI – Example: Exclusion in Jenkins Subversion plugin • Niche features are welcome
  23. 23. ©2010 CloudBees, Inc. All Rights Reserved 23©2010 CloudBees, Inc. All Rights Reserved Extensible silo requires less communication • Everyone gets his own sandbox to hack – No need to discuss designs/features/schedules upfront • No more conflicting development model – Conference-driven development vs test- driven development • No need to see someone else‟s ugly code – No need to feel ashamed of it
  24. 24. ©2010 CloudBees, Inc. All Rights Reserved 24©2010 CloudBees, Inc. All Rights Reserved Extensible silo fosters innovations • Every whacky idea gets a shot – No need to sell ideas – No need to turn down ideas • Communicate in code, not in English • Some of them will turn out great – No need to drag failed experiments
  25. 25. ©2010 CloudBees, Inc. All Rights Reserved 25©2010 CloudBees, Inc. All Rights Reserved Extensibility highlights higher-order problems • With enough silos, core devs become silo devs • What core does directly benefits silos – Simplified build mechanism – IDE support – Error checker for common mistakes • It accelerates dev scalability
  26. 26. ©2010 CloudBees, Inc. All Rights Reserved 26©2010 CloudBees, Inc. All Rights Reserved Lower barrier to entry, enabled by silo • In Jenkins, everyone gets to be a committer – Just ask, and it‟ll be given – We even have an IRC bot that does this • This works, because… – People aren‟t collaborating – Any damage done is compartmentalized – VCS enables rollback so long as it‟s not released • After all, unless you are Linus Torvalds, “too many committers” is not your problem
  27. 27. ©2010 CloudBees, Inc. All Rights Reserved 27©2010 CloudBees, Inc. All Rights Reserved Lower barrier to entry: observations • People are cautious when they have power – Very few incidents • Works well when people move on – Abandoned plugins get new maintainers • It encourages small contributions • You can never predict who becomes significant contributors
  28. 28. ©2010 CloudBees, Inc. All Rights Reserved 28©2010 CloudBees, Inc. All Rights Reserved But silo alone doesn‟t create a community Image: NASA/JPL-Caltech
  29. 29. ©2010 CloudBees, Inc. All Rights Reserved 29©2010 CloudBees, Inc. All Rights Reserved You need a center of gravity • Something that: – Drag everything together – Pulls in people • Shared source tree repo doesn‟t have to be “it” Image: NASA/JPL-University of Arizona
  30. 30. ©2010 CloudBees, Inc. All Rights Reserved 30©2010 CloudBees, Inc. All Rights Reserved Center of Gravity in Jenkins: IRC • Closest thing to the water cooler in OSS – It is a place people can come, unlike Twitter – You can be present & absent at the same time • It builds human relationships
  31. 31. ©2010 CloudBees, Inc. All Rights Reserved 31©2010 CloudBees, Inc. All Rights Reserved Center of Gravity in Jenkins: outreach • Connecting people – Meet-ups, Hackathon, Events • Giving the project a voice – Twitter, Blog Mad props to @agentdero
  32. 32. ©2010 CloudBees, Inc. All Rights Reserved 32©2010 CloudBees, Inc. All Rights Reserved But above all: Update Center • UI built into Jenkins to let users install plugins • To be listed, your plugins need to be co- located in GitHub org • Commit access covers all the repos • Everyone can add others to the project
  33. 33. ©2010 CloudBees, Inc. All Rights Reserved 33©2010 CloudBees, Inc. All Rights Reserved Observations • Creates an incentive for devs to come • When people move on, – Keeping plugin alive is easier • A structure helps new contributors – Right thing is now the easiest thing • Providing pin-point technical help is easier
  34. 34. ©2010 CloudBees, Inc. All Rights Reserved 34©2010 CloudBees, Inc. All Rights Reserved Center of Gravity = Self-reinforcing • As we acquire more plugins… – We attract more users – UC becomes more attractive to devs • Machinery gets oiled – As a result of solving higher-order problem
  35. 35. ©2010 CloudBees, Inc. All Rights Reserved 35©2010 CloudBees, Inc. All Rights Reserved Emerging Challenges • Subversion → Git changed dynamics – Repository no longer projects enough gravity • GitHub and pull requests – It‟s great that contributions are easier – But it also creates “throw the stuff over the wall” mentality – It lets people contribute without joining, reducing the inflow
  36. 36. ©2010 CloudBees, Inc. All Rights Reserved 36©2010 CloudBees, Inc. All Rights Reserved Conclusions • Scale development with extensibility • Reduce communication with extensibility • Let people innovate with extensibility • I think I made Steve proud Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility!

×