Your SlideShare is downloading. ×
0
Continuous Delivery of
Puppet Code, Lessons
Learned
Kris Buytaert
@krisbuytaert
Kris Buytaert
●
●
●

●
●
●
●

I used to be a Dev,
Then Became an Op
Chief Trolling Officer and Open Source
Consultant @inu...
Todays Goals
●

A reproducable way to deploy and upgrade
/etc/puppet

●

Automatically

●

Fast

●

Consistent

●

Continu...
What's this devops thing
anyhow ?
devops
●

Culture

●

(Lean)

●

Automation

●

Measurement

●

Sharing
Damon Edwards and John Willis
Gene Kim
devops (<)> continuous delilvery
Nirvana
An “ecosystem” that supports continuous delivery, from
infrastructure, data and configuration management to
busine...
How many times a day ?
●

10 @ Flickr

●

Deployments used to be pain

●

Nobody dared to deploy a site

●

Practice makes...
" Our job as engineers (and ops, dev-ops, QA,
support, everyone in the company actually) is to
enable the business goals. ...
For years we've tolerated humans to make
structural manual changes to the infrastructure
our critical applications are run...
I hate maturity model
So, what did we try already ?
Level -1:Hacking in production

•

Cd /etc/puppet/manifests

•

Vi site.pp
Level -1: more production hacking
•

cd /etc/puppet/manifests

•

vi site.pp

•

•

•

rsync -av * /etc/puppet

•

WFM Syn...
Level 0: I use git
•

ssh puppetmaster

•

cd /etc/puppet

•

git init

•

git add *.pp

•

git commit -m “Initial commit ...
Level 0.1: I almost understand git
•

•

Ssh puppetmaster

•

Cd /etc/puppet

•

Git init

•

Git add *.pp

•

Git commit ...
Level 0.2: But
•

Ssh puppetmaster

•

Cd /etc/puppet

•

Git init

•

Git add *.pp

•

Git commit “Initial commit of pupp...
Level 0.2:But
•

I don't always commit or push my changes

•

When I do commit I do it as root
Level 0.3:
Development happens locally
Puppetmaster runs git pull in a cronjob
=> code modified on puppet master, not push...
Level 0.3: Euh modules
vi modules/apache/manifest/init.pp
Wait,
I need to track upstream ,
How do I isolate my code ?
Git subtrees
●

Looks nice

●

Till you want to track upstream
Librarian Puppet
●

Hides complexity of submodules

●

Easy if you use Forge Modules
•

•

●

●

Does anyone ?
Do you trus...
Librarian Puppet
●

Insert ugly shell script

●
●

Even with this in place .. people can still hack on
the PuppetMaster
We all love branches
●

When they are short lived feature branches

●

Environment per branch ?
•

•

How many hosts do yo...
Is R10K faster ?
●

R10K

●

But what about Testing ?

●

You can't do CD without CI
Git Submodules
●

Basic git,

●

No extra tools required
Integrates with other (non puppet) projects too
Package all the things
Release artifacts:
●

A module

●

A set of manifests

●

A set of manifests that work with
a strict set of modules
Software Release
management is not a
solved problem
Git Submodules
●

Release management = main git project
cd modules/blah
vi manifest/init.pp
git add manifests/init.pp
git ...
Infrastructure as Code
●

Treat configuration automation as code

●

Development best practices
•

Model your infrastructu...
Continuous Integration
●

Builds

●

Nightly Builds

●

Builds with tests

●

Nightly Builds with tests

●

Frequent integ...
Jenkins
●

Open Source Continuous Integration Server

●

A zillion plugins (400)

●

Have developers build stable and depl...
Jenkins Pipeline
What's in your Pipeline ?
A pipeline
●

Checkout code

●

Syntax

●

Style

●

Code Coverage

●

Tests

●

Build

●

More Tests

●

Package
Syntax and Style
●

Initially ,
all code, all the time

●

Now,
only the changed code

●

Why not in post Commit Hooks ?
Why ops like to package
●

Packages give you features

•Consistency, security, dependencies
●

Uniquely identify where fil...
Jordan Sissel is a Hero !
#packaginlove
It's not really packaging
•

It's an immutable branch

•

It's a tracable release artefact
https://github.com/vStone/jenkinspuppet-scripts
●

Tests

●

Packages full tree in
/
etc/puppet/environm
ents/$environment...
A pipeline
●

Checkout code

●

Syntax

●

Style

●

Code Coverage

●

Tests

●

Build

●

More Tests

●

Package

●

Uplo...
Repository Management
●

Pulp
•

Pro : MirroringLove

•

Con : Mongo, Stability, .deb

●

PRM ?

●

https://github.com/Imm...
Repository Management
A pipeline
●

Checkout code

●

Upload to Repo

●

Syntax

●

Deploy on Test

●

Style

●

Code Coverage

●

Tests

●

Bui...
mc-package
Repos are SLOW
●

Createrepo is slow.

●

Pulp is slow

●

Bypass repos , upload straight to appropriate
PuppetMaster

●

...
A pipeline
●

Checkout code

●

Upload to Repo

●

Syntax

●

Deploy on Test

●

Style

●

Check Puppetruns

●

Code Cover...
Jenkins Promotion
Done ?
●

Close the feedback loop,

●

Send metric on deployment
echo "deployed.$package_name 1 `date +%s`"
> /dev/tcp/<%=...
Contact
Kris Buytaert
Kris.Buytaert@inuits.be
Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.in...
Continuous Delivery of Puppet Manifests
Continuous Delivery of Puppet Manifests
Upcoming SlideShare
Loading in...5
×

Continuous Delivery of Puppet Manifests

2,995

Published on

PuppetCamp Amsterdam 2014 Talk

Published in: Technology

Transcript of "Continuous Delivery of Puppet Manifests"

  1. 1. Continuous Delivery of Puppet Code, Lessons Learned Kris Buytaert @krisbuytaert
  2. 2. Kris Buytaert ● ● ● ● ● ● ● I used to be a Dev, Then Became an Op Chief Trolling Officer and Open Source Consultant @inuits.eu Everything is an effing DNS Problem Building Clouds since before the bookstore Some books, some papers, some blogs Evangelizing devops
  3. 3. Todays Goals ● A reproducable way to deploy and upgrade /etc/puppet ● Automatically ● Fast ● Consistent ● Continuously
  4. 4. What's this devops thing anyhow ?
  5. 5. devops ● Culture ● (Lean) ● Automation ● Measurement ● Sharing Damon Edwards and John Willis Gene Kim
  6. 6. devops (<)> continuous delilvery
  7. 7. Nirvana An “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management to business. Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its code base. Continuous Delivery , Jez Humble
  8. 8. How many times a day ? ● 10 @ Flickr ● Deployments used to be pain ● Nobody dared to deploy a site ● Practice makes perfect ● Knowing you can vs constantly doing it
  9. 9. " Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to enable the business goals. We strongly feel that in order to do that you must have the ability to deploy code quickly and safely. Even if the business goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push all the time), having a reliable and easy deployment should be non-negotiable." Etsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
  10. 10. For years we've tolerated humans to make structural manual changes to the infrastructure our critical applications are running on. Whilst at the same time demanding those critical applications to go through rigid test scenarios. Who let this happen ?
  11. 11. I hate maturity model
  12. 12. So, what did we try already ?
  13. 13. Level -1:Hacking in production • Cd /etc/puppet/manifests • Vi site.pp
  14. 14. Level -1: more production hacking • cd /etc/puppet/manifests • vi site.pp • • • rsync -av * /etc/puppet • WFM Syndrome • Does not work in teams
  15. 15. Level 0: I use git • ssh puppetmaster • cd /etc/puppet • git init • git add *.pp • git commit -m “Initial commit of puppet tree”
  16. 16. Level 0.1: I almost understand git • • Ssh puppetmaster • Cd /etc/puppet • Git init • Git add *.pp • Git commit “Initial commit of puppet tree” • Git remote add • Git push
  17. 17. Level 0.2: But • Ssh puppetmaster • Cd /etc/puppet • Git init • Git add *.pp • Git commit “Initial commit of puppet tree” • Git remote add • Git push • Vi nodes/default.pp
  18. 18. Level 0.2:But • I don't always commit or push my changes • When I do commit I do it as root
  19. 19. Level 0.3: Development happens locally Puppetmaster runs git pull in a cronjob => code modified on puppet master, not pushed => changes never make it to the platform
  20. 20. Level 0.3: Euh modules vi modules/apache/manifest/init.pp Wait, I need to track upstream , How do I isolate my code ?
  21. 21. Git subtrees ● Looks nice ● Till you want to track upstream
  22. 22. Librarian Puppet ● Hides complexity of submodules ● Easy if you use Forge Modules • • ● ● Does anyone ? Do you trust the internet to be around Librarian = Old English for “can't use submodules” And hmm... which customer uses which patched version again ?
  23. 23. Librarian Puppet ● Insert ugly shell script ● ● Even with this in place .. people can still hack on the PuppetMaster
  24. 24. We all love branches ● When they are short lived feature branches ● Environment per branch ? • • How many hosts do you connect per branch ? Limited number of branches ?
  25. 25. Is R10K faster ? ● R10K ● But what about Testing ? ● You can't do CD without CI
  26. 26. Git Submodules ● Basic git, ● No extra tools required Integrates with other (non puppet) projects too
  27. 27. Package all the things
  28. 28. Release artifacts: ● A module ● A set of manifests ● A set of manifests that work with a strict set of modules
  29. 29. Software Release management is not a solved problem
  30. 30. Git Submodules ● Release management = main git project cd modules/blah vi manifest/init.pp git add manifests/init.pp git commit -m “Fixed bug #313” cd ../../ git add modules/blah git commit -m “Fixed bug #313” git push
  31. 31. Infrastructure as Code ● Treat configuration automation as code ● Development best practices • Model your infrastructure • Version your cookbooks / manifests • Test your cookbooks/ manifests • Dev/ test /uat / prod for your infra ● Model your infrastructure ● A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )
  32. 32. Continuous Integration ● Builds ● Nightly Builds ● Builds with tests ● Nightly Builds with tests ● Frequent integration ● Continuous Integration
  33. 33. Jenkins ● Open Source Continuous Integration Server ● A zillion plugins (400) ● Have developers build stable and deployable code ● Test Infra code
  34. 34. Jenkins Pipeline
  35. 35. What's in your Pipeline ?
  36. 36. A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package
  37. 37. Syntax and Style ● Initially , all code, all the time ● Now, only the changed code ● Why not in post Commit Hooks ?
  38. 38. Why ops like to package ● Packages give you features •Consistency, security, dependencies ● Uniquely identify where files come from •Package or cfg-mgmt ● Source repo not always available •Firewall / Cloud etc .. ● Weird deployment locations , no easy access ● Little overhead when you automate
  39. 39. Jordan Sissel is a Hero !
  40. 40. #packaginlove
  41. 41. It's not really packaging • It's an immutable branch • It's a tracable release artefact
  42. 42. https://github.com/vStone/jenkinspuppet-scripts ● Tests ● Packages full tree in / etc/puppet/environm ents/$environment/
  43. 43. A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package ● Upload to Repo
  44. 44. Repository Management ● Pulp • Pro : MirroringLove • Con : Mongo, Stability, .deb ● PRM ? ● https://github.com/ImmobilienScout24/yum-repo-se ?
  45. 45. Repository Management
  46. 46. A pipeline ● Checkout code ● Upload to Repo ● Syntax ● Deploy on Test ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package
  47. 47. mc-package
  48. 48. Repos are SLOW ● Createrepo is slow. ● Pulp is slow ● Bypass repos , upload straight to appropriate PuppetMaster ● Upload to repo for rebootstrapping
  49. 49. A pipeline ● Checkout code ● Upload to Repo ● Syntax ● Deploy on Test ● Style ● Check Puppetruns ● Code Coverage ● Check Icinga ● Tests ● Promote to UAT ● Build ● More Tests ● Package
  50. 50. Jenkins Promotion
  51. 51. Done ? ● Close the feedback loop, ● Send metric on deployment echo "deployed.$package_name 1 `date +%s`" > /dev/tcp/<%= graphite_host %>/2003
  52. 52. Contact Kris Buytaert Kris.Buytaert@inuits.be Further Reading @krisbuytaert http://www.krisbuytaert.be/blog/ http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×