Lessons Learned in the OSUOSL Puppet Migration

849 views
751 views

Published on

The OSU Open Source Lab has used CFengine 2 over the past ten years and the code has exploded towards over 14,000 lines of code. We had set out to migrate to Puppet over four years ago and still haven't completed the process. We have made several mistakes over that time and this session will discuss some of the do's and don'ts of what we learned along the way. We are finally down a path where we can make the transition and want to share our experiences. This session will also be helpful for those in the middle of a major puppet upgrade.

Lance Albertson
Director, OSU Open Soruce Lab
Lance Albertson is the Director for the Oregon State University Open Source Lab (OSUOSL) and has been involved with the Gentoo Linux project as a developer and package maintainer since 2003. Since joining the OSUOSL in 2007, Lance has managed all of the hosting activities that the OSL provides for nearly 160 high-profile open source projects. He was recently promoted to Director in early 2013 after being the Lead Systems Administration and Architect since 2007. In his free time he helps organize Beaver BarCamp and plays trumpet in a local jazz group The Infallible Collective. He holds a B.A. in Agriculture Technology Management from Kansas State University, where he minored in Agronomy and Computer Science.

Published in: Technology, Spiritual
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
849
On SlideShare
0
From Embeds
0
Number of Embeds
170
Actions
Shares
0
Downloads
14
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Lessons Learned in the OSUOSL Puppet Migration

  1. 1. Lessons Learned in the OSUOSL Puppet Migration Lance Albertson Director lance@osuosl.org @ramereth
  2. 2. Session Summary ● OSL team environment overview ● Brief OSL systems architecture history ● CFEngine environment & stats ● Initial Puppet environments ● Git repo all the things! ● Modules strategy ● Deployment/migration strategy ● Future Plans Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  3. 3. OSL Team Environment ● 2 FTE Sysadmins, 1 FTE developer ● 6-10 student sysadmins ● 4-6 student devs ● Student productivity / turnover ● Multiple on-going projects ● Incoming tickets keeps us busy ● Domain knowledge is mostly on me* * I'm bad at documentation Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  4. 4. OSL Team Environment Systems weekly team meeting Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  5. 5. OSL Systems History ● Started out on Debian ('03-'05) ● Switched to Gentoo Hardened ('05-'12) ○ Gentoo devs on staff (mostly me) ○ Wanted the grsec/PaX features ● Started deploying CentOS 5 ('08+) ● All new deployments CentOS 6 ('12+) ○ EOL schedule worked best for us ● CFEngine 2 for config management ● Some mixed CFEngine/puppet env. Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  6. 6. CFengine Environment ● Manages all package installs & upgrades ● 180 "Services" ● 174 Package classes ● 113 cf. files ● 19,500 lines of raw cf files ● 14,700 lines actual code ● 1490 lines in cf.classes alone ● 24,300 commits (9,100 are mine alone) Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  7. 7. CFengine Environment ● "Do all the things in cfengine" ● A lot of hacked logic because its CF 2 ● Not very dynamic for our needs ● No way to delegate access to projects ● Upgrade path to CF 3 would be a nightmare ● Works very well for our Gentoo hosts Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  8. 8. Reason for choosing Puppet ● Liked its goals and approach overall ● Proximity to Puppet Labs (PDX) ● Lots of sharable modules and code ● Excellent community ● Very sysadmin focused ● Lots of progress in its feature set ○ Razor, Puppet Enterprise, Foreman, etc Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  9. 9. Puppet Migration Strategy ● Avoid mixed cfengine / puppet environments at all cost ○ Either all cfengine or all puppet ● Convert all CentOS hosts first ○ Easy transition ● Rebuild, Retire, rethink Architecture ○ Same basic architecture since 2005 Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  10. 10. The Beginnings ● Planning in early 2010 ● Summer of 2010 initial implementation ● Student project ● Used code from example42 ● Single git repo (#1) for everything ● Gentoo Puppet issues ○ No concept of use flags, keywords, etc ○ Package dependencies are hell Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  11. 11. Git repo all the things! - 2010 ● Fall 2010 - Git repo #2 created ● Split into repos based on projects ○ Try to solve the delegation problem ○ Manifests were done in an ugly way ○ Didn't use submodules but a simple script to keep things in sync - i.e. non-standard ● Repo is in production still today ○ Drupal Project has their own module repo ○ They use it to manage their services primarily ○ We run CFengine along-side puppet on Drupal Project machines - YUCK! Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  12. 12. Git repo all the things! - 2011 ● Summer 2011 Repo #3 created ● 2-3 students started it again ● Partnered with PSU CAT team ● Repo for every module, and submodule everything ● Nightmare management of the super-repo ● Very confusing to new students ● Deployed it on all the student workstations to test ○ Now its instance #2 that was running in production at the time Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  13. 13. Git repo all the things! - 2011 ● Summer 2011 ● Added basic puppet syntax checking commit hook ● Implemented puppet-sync ○ https://github.com/pdxcat/puppet-sync ○ A script to synchronize you manifests from a GIT repository to your Puppet master. ● This helped but the submodules produced so much rage face Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  14. 14. Git repo all the things! - 2012 ● Summer 2012 ○ I take a crack at fixing the repo mess ● Repo #4 is created with a slightly saner approach ● Still using submodules, but much less ● Refactored a bunch of code ● Did take a look at mr for repo management ● Never went into production, kind of forgotten Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  15. 15. “puppet-simple” repo - 2012 ● Fall 2012 - "Lets get this right for real" ● Very simplified multi-repo layout (K.I.S.S.) ● Single repo ● Directory layout: ○ dist/ - internal modules ○ libs/modules - public or internal->public modules ○ site/ - site specific modules ■ site/os - OS specific module ■ site/role - Role specific module ● Submodules for libs/* Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  16. 16. Repo Recap ● #1 Single git repo - deprecated ● #2 Multi-repos w/o submodules - prod (Drupal) ● #3 Multi-repos w/ submodules - deprecated ● #4 Multi-repos w/ submodules (Lance refactor) ○ never used ● #5 puppet-simple - prod (workstations) Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  17. 17. Lessons learned from the repos ● K.I.S.S. from the start ● Submodules produces a lot of rage face ● Pick something and stick with it the best you can ● Flexibility is nice, but don't need it in the beginning ● Try to use community modules when possible Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  18. 18. Chef Experiment - 2013 ● Wait, I thought this was a Puppet migration talk? ● Decided to take another look at Chef ● Testing on a small set of new machine environments ● Has its own set of issues and problems ○ Fixed some initial workflow problems we had with puppet ● Jury is still out ● “Yay” for three config management envs! </sarcasm> Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  19. 19. Current status of migration ● Workstations on new "simple" repo ● Building base modules (80% finished) ○ Importing some from the other repos, refactoring, cleaning up ○ Using community modules (mysql, concat, etc) ● Importing OSL site specific magic ○ Converting CFengine-isms into Puppet ○ Try and undo really hacked up code and processes ● Testing and more testing Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  20. 20. Testing Environment ● Vagrant and more vagrant ○ Build standard cfengine-ized basebox ○ Run new modules to see changes ● Future Plans ○ RSpec testing on modules ○ Jenkins CI environment of some kind ● Projects' access ○ Give them an environment to test major changes ● Needs a lot of work Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  21. 21. Current Migration Strategy ● Finish base puppet modules ○ Build other modules as needed ● Deploy application service management ○ Services not currently managed by CFengine ○ Will run in parallel with CFengine ● Convert all current CentOS 5/6 hosts one-by-one ● Gentoo -> CentOS Migration ○ Rebuilt as CentOS 6 under puppet ○ Retired, re-architected, etc Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  22. 22. Future Plans ● Publish OSL modules ● Delegation to projects ○ Allow projects to check out their puppet config ○ Integrated testing, merging, etc ○ Less work on our staff long term ● Use other tools ○ Deploy Foreman (or something similar like PE) ○ Try out Puppet Enterprise ○ Look into PuppetDB, MCollective, Hiera, etc Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  23. 23. Conclusion Takeaways ● Dedicate someone on the conversion ○ Maintain consistency, less context switching ● Use K.I.S.S. principle as much as possible ● Target specific hosts as examples for conversion ○ Do test deploys with vagrant ● Use "brick and mortar" philosophy ○ Try and keep site specific code out of the modules the best you can Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  24. 24. Questions? Lance Albertson lance@osuosl.org @ramereth http://osuosl.org http://lancealbertson.com Follow OSUOSL @osuosl | fb.com/OSUOSL G+ "Open Source Lab" This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. Copyright 2013

×