8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 1/29
Au...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 2/29
We...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 3/29
Me...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 4/29
20...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 5/29
20...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 6/29
20...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 7/29
20...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 8/29
20...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 9/29
20...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 10/29
P...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 11/29
W...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 12/29
A...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 13/29
W...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 14/29
A...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 15/29
G...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 16/29
C...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 17/29
B...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 18/29
H...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 19/29
H...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 20/29
W...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 21/29
A...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 22/29
W...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 23/29
W...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 24/29
H...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 25/29
S...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 26/29
P...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 27/29
T...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 28/29
W...
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis
localhost:9090/onepage 29/29
T...
Upcoming SlideShare
Loading in …5
×

Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

5,518 views

Published on

Attend this talk and learn how Puppet Labs handles community contributions and the FOSS tools we’ve published to automate much of the process. Puppet Labs handles thousands of contributions from hundreds of contributors and we’ve integrated Github, TravisCI and Trello to manage all of it. Come see how we do it and what we’ve built!

Jeff McCune
Software Developer, Puppet Labs
Recovering systems engineer turned software engineer, responsible for helping the Puppet community contribute their code to our open source projects.

Published in: Technology, Spiritual
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,518
On SlideShare
0
From Embeds
0
Number of Embeds
245
Actions
Shares
0
Downloads
55
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

  1. 1. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 1/29 Automating Community Contributions Jeff McCune Twitter: @0xEFF Email: jeff@puppetlabs.com http://puppetlabs.com
  2. 2. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 2/29 We'll Cover: Process Challenging 2012 process Improved 2013 process Automation & Tooling Github (patches) Travis (tests) Trello (work items) Puppet Webhooks (automation) Questions Please ask questions during or after the talk, there will be time.
  3. 3. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 3/29 Merge more patches with automation Dedicated people work well Context switching is a challenge Automation Reduce context switching Single pane of glass
  4. 4. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 4/29 2012 Process Platform Team Bug Fixes New Features Security Incidents Community Pull Requests Two types of work Planned - Bugs and Features Unplanned - Pull Requests and Security Pull requests often came last in the priority list
  5. 5. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 5/29 2012 Process Outcome Large and growing backlog No response for months Sprints to "catch up" Public complaints from key community members Stressed out team
  6. 6. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 6/29 2012 Results
  7. 7. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 7/29 2013 Process Community Team Community Pull Requests Goal: 2 PM next business day response One type of work, not two Reduced context switching Platform Team Bug Fixes New Features Security Incidents Reduced context switching
  8. 8. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 8/29 2013 Process Community Team responsibilities: Community Pull Requests Platform Team responsibilities: Bug Fixes New Features Security Incidents One type of work: Unplanned - Pull Requests and Security Pull requests are consistently prioritized.
  9. 9. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 9/29 2013 Results We're merging ~ 15 community patches per week
  10. 10. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 10/29 Process Conclusion Dedicated People Minimize context switching Decreased backlog, faster response time. Backlog down to 25 from 80 New pull requests responded to within two days instead of months. Automating the process With a solid process in place, we focused on automating the tedious aspects.
  11. 11. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 11/29 Work in Flight Trello
  12. 12. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 12/29 Automation and Tooling Github (patches) Travis (tests) Trello (work items) Three separate tools, lots of switching back and forth. Puppet Webhooks puts it all in once place.
  13. 13. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 13/29 Why more Tooling? Too many sources of information are tedious and error prone to review. Issue Tracker (Should I work on this now?) Jenkins and Travis (Does this patch break stuff?) Github (What are people saying about this patch?) CLA (Has the contributor signed the license agreement?) Trello (What should we be working on?) All of this information is important. It's also spread out on four or more different sites. It was often overlooked. A solution is to move all of the information into one place.
  14. 14. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 14/29 All information in Trello Our goal is to get all of the information we need to review a pull request into Trello. New pull request creates a card. Code comments Contributor bio CLA Information select(contributor) Completed this week Puppet, Facter, Hiera, stdlib on one board
  15. 15. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 15/29 Gepetto Bot Gepetto Bot is a service account with a profile on Github, Trello, and Heroku. The puppet-webhooks Heroku app logs into these services as Gepetto Bot.
  16. 16. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 16/29 Consolidated Information Gepetto Bot gathers contributor bio using the Github API:
  17. 17. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 17/29 Board Columns Cards move left to right. Gepetto Bot creates them in the Response Needed column. Response Needed Accepted Doing Waiting on Contributor Waiting > 1 week Going through CI Finished this Iteration
  18. 18. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 18/29 How it works Webhooks Github executes a callback when events happen. The callback is a very simple HTTP POST with a JSON body containing data about the event. Think push notifications for the web.
  19. 19. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 19/29 How it works The Puppet Webhooks code running in Heroku receives the HTTP POST and: 1. Github posts JSON to our Heroku app 2. Persist the JSON to a Delayed Job queue 3. Start a Delayed Job worker process using workless gem 4. Data is posted to Trello in a matter of seconds
  20. 20. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 20/29 Why use Delayed Job? Heroku provides: Small PostgreSQL databases for free 750 hours of process time for free 31 x 24 = 744 hours for the web process 6 "extra" hours for free Automatic retries with back-off The workless gem allows us to only run DJ worker processes when there's actually work to be done. Note: While this is all free of charge, a credit card is required to use the Heroku API with workless.
  21. 21. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 21/29 Automatic Process Management The web process runs persistently. bundle exec thin start ­p $PORT ­e $RACK_ENV The worker process runs only when there's work to be done. bundle exec rake jobs:worksilent Workless automatically adjusts the worker dyno:
  22. 22. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 22/29 What is automated? New Pull Request => New Trello Card Code Comment => Trello Comment Collect Contributor Bio Card due Date Weekly Summary of completed cards Unimplemented event handlers Pull Request is closed New code is pushed Travis CI tests complete
  23. 23. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 23/29 Weekly summary email Sent to puppet-dev mailing list weekly Automated summary of completed Trello cards Scans card comments and extracts summary:matches
  24. 24. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 24/29 How the weekly summary works Github Gist as a persistent data store! 1. App reads a Liquid template from a raw github URL 2. Collect all completed Trello cards 3. Fill in the template with the card data 4. Post the resulting Markdown back to the Gist No persistent state in the app or the database. Liquid template is easily updated.
  25. 25. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 25/29 Summary Template Header: ## Puppet Dev Community Summary  * Finished Card Total: **{{ cards.size }}** {% for section in sections %} * {{ section[0] }}: {{ section[1].size }} {% endfor %} Sections: {% for section in sections %}## {{ section[0] }} Cards: {% for card in section[1] %}#### [{{card.title}}]({{card.url}}) {{card.message }}
  26. 26. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 26/29 Post back to Github heroku run bundle exec rake jobs:summary
  27. 27. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 27/29 Tooling Wrap Up Automate the Tedious Stuff Try the Puppet Webhooks App github.com/puppetlabs/puppet-webhooks
  28. 28. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 28/29 What we covered Process Context Switches = (ಥ_ಥ) Dedicated People = (◕‿◕) Automation & Tooling Github, Travis, Trello Puppet Webhooks automates all three Key Message Merge more patches with automation
  29. 29. 8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 29/29 Thank You! Questions? Jeff McCune 0xEFF jeff@puppetlabs.com

×