Lessons Learned inthe OSUOSL Puppet     Migration     Lance Albertson        Director    lance@osuosl.org       @ramereth
Session Summary●   OSL team environment overview●   Brief OSL systems architecture history●   CFEngine environment & stats...
OSL Team Environment●   2 FTE Sysadmins, 1 FTE developer●   6-10 student sysadmins●   4-6 student devs●   Student producti...
OSL Team Environment     Left to Right: Rudy (basic), Daniel (irdan) and Sean (chekka)         Lessons Learned in the OSUO...
OSL Systems History● Started out on Debian (03-05)● Switched to Gentoo Hardened (05-12)  ○ Gentoo devs on staff (mostly me...
CFengine Environment●   Manages all package installs & upgrades●   180 "Services"●   174 Package classes●   110 cf. files●...
CFengine Environment● "Do all the things in cfengine"● A lot of hacked logic because its CF 2● Not very dynamic for our ne...
Reason for choosing Puppet●   Liked its goals and approach overall●   Proximity to Puppet Labs (PDX)●   Lots of sharable m...
Puppet Migration Strategy● Avoid mixed cfengine / puppet  environments at all cost  ○   Either all cfengine or all puppet●...
The Beginnings●   Planning in early 2010●   Summer of 2010 initial implementation●   Student project●   Used code from exa...
Git repo all the things! - 2010● Fall 2010 - Git repo #2 created● Split into repos based on projects  ○ Try to solve the d...
Git repo all the things! - 2011● Summer 2011 Repo #3 created● 2-3 students started it again● Partnered with PDXCAT team● R...
Git repo all the things! - 2011● Summer 2011● Added basic puppet syntax checking commit  hook● Implemented puppet-sync  ○ ...
Git repo all the things! - 2012● Summer 2012  ○ I take a crack at fixing the repo mess● Repo #4 is created with a slightly...
Git repo all the things! - 2013● Fall 2013 - "Lets get this right for real"● Very simplified multi-repo layout (K.I.S.S.)●...
Lessons learned from the repos● K.I.S.S. from the start● Submodules produces a lot of rage face● Pick something and stick ...
Current status of migration● Workstations on new "simple" repo● Building base modules (80% finished)  ○ Importing some fro...
Testing Environment● Vagrant and more vagrant  ○ Build standard cfengine-ized basebox  ○ Run new modules to see changes● F...
Current Migration Strategy● Finish base puppet modules  ○ Build other modules as needed● Deploy application service manage...
Future Plans● Publish OSL modules● Delegation to projects  ○ Allow projects to check out their puppet config  ○ Integrated...
Conclusion Takeaways● Dedicate someone on the conversion  ○ Maintain consistency, less context switching● Use K.I.S.S. pri...
Questions?                  Lance Albertson                 lance@osuosl.org                     @ramereth                ...
Upcoming SlideShare
Loading in …5
×

PuppetCampLA 2013: Lessons Learned in the OSUOSL Puppet Migration

448 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
448
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

PuppetCampLA 2013: Lessons Learned in the OSUOSL Puppet Migration

  1. 1. Lessons Learned inthe OSUOSL Puppet Migration Lance Albertson Director lance@osuosl.org @ramereth
  2. 2. Session Summary● OSL team environment overview● Brief OSL systems architecture history● CFEngine environment & stats● Initial Puppet environments● Git repo all the things!● Modules strategy● Deployment/migration strategy● Future Plans Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  3. 3. OSL Team Environment● 2 FTE Sysadmins, 1 FTE developer● 6-10 student sysadmins● 4-6 student devs● Student productivity / turnover● Multiple on-going projects● Incoming tickets keeps us busy● Domain knowledge is mostly on me* * Im bad at documentation Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  4. 4. OSL Team Environment Left to Right: Rudy (basic), Daniel (irdan) and Sean (chekka) Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  5. 5. OSL Systems History● Started out on Debian (03-05)● Switched to Gentoo Hardened (05-12) ○ Gentoo devs on staff (mostly me) ○ Wanted the grsec/PaX features● Started deploying CentOS 5 (08+)● All new deployments CentOS 6 (12+) ○ EOL schedule worked best for us● CFEngine 2 for config management● Some mixed CFEngine/puppet env. Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  6. 6. CFengine Environment● Manages all package installs & upgrades● 180 "Services"● 174 Package classes● 110 cf. files● 19,200 lines of raw cf files● 14,700 lines actual code● 1440 lines in cf.classes alone● 23,000 commits (8,800 are mine alone) Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  7. 7. CFengine Environment● "Do all the things in cfengine"● A lot of hacked logic because its CF 2● Not very dynamic for our needs● No way to delegate access to projects● Upgrade path to CF 3 would be a nightmare● But excellent-ish support for Gentoo :-) Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  8. 8. Reason for choosing Puppet● Liked its goals and approach overall● Proximity to Puppet Labs (PDX)● Lots of sharable modules and code● Excellent community● Lots of progress in its feature set● Horrible Gentoo support :-( ○ But its improving a little! Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  9. 9. Puppet Migration Strategy● Avoid mixed cfengine / puppet environments at all cost ○ Either all cfengine or all puppet● Convert all CentOS hosts first ○ Easy transition● Rebuild, Retire, Rearchitect Gentoo hosts ○ Same basic architecture since 2005 Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  10. 10. The Beginnings● Planning in early 2010● Summer of 2010 initial implementation● Student project● Used code from example42● Single git repo (#1) for everything● Gentoo Puppet issues ○ No concept of use flags, keywords, etc ○ Package dependencies are hell Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  11. 11. Git repo all the things! - 2010● Fall 2010 - Git repo #2 created● Split into repos based on projects ○ Try to solve the delegation problem ○ Manifests were done in an ugly way ○ Didnt use submodules but a simple script to keep things in sync - i.e. non-standard● Repo is in production still today ○ Drupal Project has their own module repo ○ They use it to manage their services primarily ○ We run CFengine along-side puppet on Drupal Project machines Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  12. 12. Git repo all the things! - 2011● Summer 2011 Repo #3 created● 2-3 students started it again● Partnered with PDXCAT team● Repo for every module, and submodule everything● Nightmare management of the super-repo● Very confusing to new students● Deployed it on all the student workstations to test ○ Now its instance #2 that is running in production Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  13. 13. Git repo all the things! - 2011● Summer 2011● Added basic puppet syntax checking commit hook● Implemented puppet-sync ○ https://github.com/pdxcat/puppet-sync ○ A script to synchronize you manifests from a GIT repository to your Puppet master.● This helped but the submodules produced so much rage face Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  14. 14. Git repo all the things! - 2012● Summer 2012 ○ I take a crack at fixing the repo mess● Repo #4 is created with a slightly saner approach● Still using submodules, but much less● Refactored a bunch of code● Did take a look at mr for repo management● Never went into production, kind of forgotten Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  15. 15. Git repo all the things! - 2013● Fall 2013 - "Lets get this right for real"● Very simplified multi-repo layout (K.I.S.S.)● Single repo● Directory layout: ○ dist/ - internal modules ○ libs/modules - public or internal->public modules ○ site/ - site specific modules ■ site/os - OS specific module ■ site/role - Role specific module● Submodules for libs/* Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  16. 16. Lessons learned from the repos● K.I.S.S. from the start● Submodules produces a lot of rage face● Pick something and stick with it the best you can● Flexibility is nice, but dont need it in the beginning● Try to use community modules when possible Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  17. 17. Current status of migration● Workstations on new "simple" repo● Building base modules (80% finished) ○ Importing some from the other repos, refactoring, cleaning up ○ Using community modules (mysql, concat, etc)● Importing OSL site specific magic ○ Converting CFengine-isms into Puppet ○ Try and undo really hacked up code and processes● Testing and more testing Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  18. 18. Testing Environment● Vagrant and more vagrant ○ Build standard cfengine-ized basebox ○ Run new modules to see changes● Future Plans ○ RSpec testing on modules (eventually) ○ Jenkins CI environment of some kind● Projects access ○ Give them an environment to test major changes● Needs a lot of work Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  19. 19. Current Migration Strategy● Finish base puppet modules ○ Build other modules as needed● Deploy application service management ○ Services not currently managed by CFengine ○ Will run in parallel with CFengine● Convert all current CentOS 5/6 hosts one-by-one● Gentoo -> CentOS Migration ○ Rebuilt as CentOS 6 under puppet ○ Retired, re-architected, etc Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  20. 20. Future Plans● Publish OSL modules● Delegation to projects ○ Allow projects to check out their puppet config ○ Integrated testing, merging, etc ○ Less work on our staff long term● Use other tools ○ Deploy Foreman (or something similar like PE) ○ Try out Puppet Enterprise ○ Look into PuppetDB, MCollective, Hiera, etc Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  21. 21. Conclusion Takeaways● Dedicate someone on the conversion ○ Maintain consistency, less context switching● Use K.I.S.S. principle as much as possible● Target specific hosts as examples for conversion ○ Do test deploys with vagrant● Use "brick and mortar" philosophy ○ Try and keep site specific code out of the modules the best you can Lessons Learned in the OSUOSL Puppet Migration Lance Albertson | lance@osuosl.org | @ramereth
  22. 22. Questions? Lance Albertson lance@osuosl.org @ramereth http://osuosl.org http://lancealbertson.com Follow OSUOSL @osuosl | fb.com/OSUOSL G+ "Open Source Lab" This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. Copyright 2013

×