1. Continuous Delivery of
Puppet Code, Lessons
Learned
Kris Buytaert
@krisbuytaert

2. Kris Buytaert
●
●
●
●
●
●
●
I used to be a Dev,
Then Became an Op
Chief Trolling Officer and Open Source
Consultant @inuits.eu
Everything is an effing DNS Problem
Building Clouds since before the bookstore
Some books, some papers, some blogs
Evangelizing devops

3. Todays Goals
●
A reproducable way to deploy and upgrade
/etc/puppet
●
Automatically
●
Fast
●
Consistent
●
Continuously

4. What's this devops thing
anyhow ?

5. devops
●
Culture
●
(Lean)
●
Automation
●
Measurement
●
Sharing
Damon Edwards and John Willis
Gene Kim

6. devops (<)> continuous delilvery

7. Nirvana
An “ecosystem” that supports continuous delivery, from
infrastructure, data and configuration management to
business.
Through automation of the build, deployment, and testing
process, and improved collaboration between developers,
testers, and operations, delivery teams can get changes
released in a matter of hours — sometimes even minutes–no
matter what the size of a project or the complexity of its code
base.
Continuous Delivery , Jez Humble

9. How many times a day ?
●
10 @ Flickr
●
Deployments used to be pain
●
Nobody dared to deploy a site
●
Practice makes perfect
●
Knowing you can vs constantly doing it

10. " Our job as engineers (and ops, dev-ops, QA,
support, everyone in the company actually) is to
enable the business goals. We strongly feel that
in order to do that you must have the ability to
deploy code quickly and safely. Even if the
business goals are to deploy strongly QA’d code
once a month at 3am (it’s not for us, we push all
the time), having a reliable and easy deployment
should be non-negotiable."
Etsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

11. For years we've tolerated humans to make
structural manual changes to the infrastructure
our critical applications are running on.
Whilst at the same time demanding those critical
applications to go through rigid test scenarios.
Who let this happen ?

12. I hate maturity model

13. So, what did we try already ?

14. Level -1:Hacking in production
•
Cd /etc/puppet/manifests
•
Vi site.pp

15. Level -1: more production hacking
•
cd /etc/puppet/manifests
•
vi site.pp
•
•
•
rsync -av * /etc/puppet
•
WFM Syndrome
•
Does not work in teams

16. Level 0: I use git
•
ssh puppetmaster
•
cd /etc/puppet
•
git init
•
git add *.pp
•
git commit -m “Initial commit of puppet tree”

17. Level 0.1: I almost understand git
•
•
Ssh puppetmaster
•
Cd /etc/puppet
•
Git init
•
Git add *.pp
•
Git commit “Initial commit of puppet tree”
•
Git remote add
•
Git push

18. Level 0.2: But
•
Ssh puppetmaster
•
Cd /etc/puppet
•
Git init
•
Git add *.pp
•
Git commit “Initial commit of puppet tree”
•
Git remote add
•
Git push
•
Vi nodes/default.pp

19. Level 0.2:But
•
I don't always commit or push my changes
•
When I do commit I do it as root

20. Level 0.3:
Development happens locally
Puppetmaster runs git pull in a cronjob
=> code modified on puppet master, not pushed
=> changes never make it to the platform

21. Level 0.3: Euh modules
vi modules/apache/manifest/init.pp
Wait,
I need to track upstream ,
How do I isolate my code ?

22. Git subtrees
●
Looks nice
●
Till you want to track upstream

23. Librarian Puppet
●
Hides complexity of submodules
●
Easy if you use Forge Modules
•
•
●
●
Does anyone ?
Do you trust the internet to be around
Librarian = Old English for “can't use
submodules”
And hmm... which customer uses which patched version again ?

24. Librarian Puppet
●
Insert ugly shell script
●
●
Even with this in place .. people can still hack on
the PuppetMaster

26. We all love branches
●
When they are short lived feature branches
●
Environment per branch ?
•
•
How many hosts do you connect per
branch ?
Limited number of branches ?

27. Is R10K faster ?
●
R10K
●
But what about Testing ?
●
You can't do CD without CI

28. Git Submodules
●
Basic git,
●
No extra tools required
Integrates with other (non puppet) projects too

29. Package all the things

30. Release artifacts:
●
A module
●
A set of manifests
●
A set of manifests that work with
a strict set of modules

31. Software Release
management is not a
solved problem

32. Git Submodules
●
Release management = main git project
cd modules/blah
vi manifest/init.pp
git add manifests/init.pp
git commit -m “Fixed bug #313”
cd ../../
git add modules/blah
git commit -m “Fixed bug #313”
git push

33. Infrastructure as Code
●
Treat configuration automation as code
●
Development best practices
•
Model your infrastructure
•
Version your cookbooks / manifests
•
Test your cookbooks/ manifests
•
Dev/ test /uat / prod for your infra
●
Model your infrastructure
●
A working service = automated ( Application Code + Infrastructure
Code + Security + Monitoring )

34. Continuous Integration
●
Builds
●
Nightly Builds
●
Builds with tests
●
Nightly Builds with tests
●
Frequent integration
●
Continuous Integration

35. Jenkins
●
Open Source Continuous Integration Server
●
A zillion plugins (400)
●
Have developers build stable and deployable
code
●
Test Infra code

36. Jenkins Pipeline

37. What's in your Pipeline ?

38. A pipeline
●
Checkout code
●
Syntax
●
Style
●
Code Coverage
●
Tests
●
Build
●
More Tests
●
Package

39. Syntax and Style
●
Initially ,
all code, all the time
●
Now,
only the changed code
●
Why not in post Commit Hooks ?

40. Why ops like to package
●
Packages give you features
•Consistency, security, dependencies
●
Uniquely identify where files come from
•Package or cfg-mgmt
●
Source repo not always available
•Firewall / Cloud etc ..
●
Weird deployment locations , no easy access
●
Little overhead when you automate

41. Jordan Sissel is a Hero !

42. #packaginlove

43. It's not really packaging
•
It's an immutable branch
•
It's a tracable release artefact

44. https://github.com/vStone/jenkinspuppet-scripts
●
Tests
●
Packages full tree in
/
etc/puppet/environm
ents/$environment/

45. A pipeline
●
Checkout code
●
Syntax
●
Style
●
Code Coverage
●
Tests
●
Build
●
More Tests
●
Package
●
Upload to Repo

46. Repository Management
●
Pulp
•
Pro : MirroringLove
•
Con : Mongo, Stability, .deb
●
PRM ?
●
https://github.com/ImmobilienScout24/yum-repo-se
?

47. Repository Management

48. A pipeline
●
Checkout code
●
Upload to Repo
●
Syntax
●
Deploy on Test
●
Style
●
Code Coverage
●
Tests
●
Build
●
More Tests
●
Package

49. mc-package

50. Repos are SLOW
●
Createrepo is slow.
●
Pulp is slow
●
Bypass repos , upload straight to appropriate
PuppetMaster
●
Upload to repo for rebootstrapping

51. A pipeline
●
Checkout code
●
Upload to Repo
●
Syntax
●
Deploy on Test
●
Style
●
Check Puppetruns
●
Code Coverage
●
Check Icinga
●
Tests
●
Promote to UAT
●
Build
●
More Tests
●
Package

52. Jenkins Promotion

53. Done ?
●
Close the feedback loop,
●
Send metric on deployment
echo "deployed.$package_name 1 `date +%s`"
> /dev/tcp/<%= graphite_host %>/2003

54. Contact
Kris Buytaert
Kris.Buytaert@inuits.be
Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.be/
Inuits
Duboistraat 50
2060 Antwerpen
Belgium
891.514.231
+32 475 961221