This document provides an overview of steps to create a mature Puppet system, including setting up editors, code validation, environments, master and client configuration, and data management with Hiera. Key recommendations include using Vagrant for testing, validating code with puppet-lint, implementing environments for code promotion, separating code and data, and writing less code by leveraging defaults and Hiera lookups.
With PHP frameworks being more decoupled than ever, and with the help of a package and dependency manager, large and heavy PHP frameworks are becoming a thing of the past. Modern PHP developers now have a wealth of libraries available that specialize at specific tasks, and microservices are fast becoming a preferred way to architect applications. But many don't know how to start.
This talk will briefly introduce what microservices are, and how to use them. Then show how to build a foundation using the Zend Expressive microframework leveraging components of Zend Framework, and other libraries, to quickly create awesome things without requiring an entire framework. Resources for reference and continued learning will also be shared.
Package manages and Puppet - PuppetConf 2015ice799
This talk will begin by explaining what a package manager is and how package managers work, at a high level. Next, we'll observe the common patterns seen on the internet of compiling software in a Puppet manifest and discuss why this not ideal. This talk will conclude by showing how you can add package repositories to your infrastructure using Puppet and what settings are important for ensuring secure access to remote package repositories.
PuppetCamp London fall 2014
Martin Alfke - Can you upgrade to Puppet 4.x?
My talk at PuppetCamp London 2014 taking care on best practices and bad examples and an outlook to Puppet 4.
Create ReactJS Component & publish as npm packageAndrii Lundiak
How to prepare your (provider) ReactJS component and let your friends (consumer) to use it.
What issues you may face with Babel, Webpack, Eslint, Node, npm.
When to use “npm link” approach and “npm publish” approach.
What else to read and to try.
A book for learning puppet by real example and by building code. Chapter 1 gives you basic introduction and sets you up with a server-agent using Vagrant so that you can do hands-on.
With PHP frameworks being more decoupled than ever, and with the help of a package and dependency manager, large and heavy PHP frameworks are becoming a thing of the past. Modern PHP developers now have a wealth of libraries available that specialize at specific tasks, and microservices are fast becoming a preferred way to architect applications. But many don't know how to start.
This talk will briefly introduce what microservices are, and how to use them. Then show how to build a foundation using the Zend Expressive microframework leveraging components of Zend Framework, and other libraries, to quickly create awesome things without requiring an entire framework. Resources for reference and continued learning will also be shared.
Package manages and Puppet - PuppetConf 2015ice799
This talk will begin by explaining what a package manager is and how package managers work, at a high level. Next, we'll observe the common patterns seen on the internet of compiling software in a Puppet manifest and discuss why this not ideal. This talk will conclude by showing how you can add package repositories to your infrastructure using Puppet and what settings are important for ensuring secure access to remote package repositories.
PuppetCamp London fall 2014
Martin Alfke - Can you upgrade to Puppet 4.x?
My talk at PuppetCamp London 2014 taking care on best practices and bad examples and an outlook to Puppet 4.
Create ReactJS Component & publish as npm packageAndrii Lundiak
How to prepare your (provider) ReactJS component and let your friends (consumer) to use it.
What issues you may face with Babel, Webpack, Eslint, Node, npm.
When to use “npm link” approach and “npm publish” approach.
What else to read and to try.
A book for learning puppet by real example and by building code. Chapter 1 gives you basic introduction and sets you up with a server-agent using Vagrant so that you can do hands-on.
Ganeti Web Manager: Cluster Management Made SimpleOSCON Byrum
Looking for an easy, scalable way to manage your Ganeti-based clusters? Ganeti Web Manager provides admins an easy to deploy, Django based GUI that effectively manages private clusters & works equally well for providing customers access. With a caching system designed to scale to thousands of virtual machines without decreasing performance, Ganeti Web Manager makes cluster management truly simple.
Dennis Matotek, Technical Lead Platforms at Experian Hitwise Australia, gave an excellent presentation on setting up puppet using vagrant, puppet and testing, including a full demo of rspec-puppet and Jenkins.
Cooking Perl with Chef: Real World Tutorial with JitterbugDavid Golden
This tutorial provides a command-by-command walk-through for deploying the Jitterbug continuous integration application using the Chef configuration management tool
PECL Picks - Extensions to make your life betterZendCon
One of the biggest strengths of PHP is its "glue" power. Take any C library and with a little magic and a compiler you have a fantastic extension. These extensions hide in PECL, but few people can tell the good from the unmaintained or just plain broken. Find the best extensions for your project, learn about PECL, and find out how to become a part of the PECL developer community.
Kicking off with Zend Expressive and Doctrine ORM (ZendCon 2016)James Titcumb
You've heard of the new Zend framework, Expressive, and you've heard it's the new hotness. In this talk, I will introduce the concepts of Expressive, how to bootstrap a simple application with the framework using best practices, and how to integrate a third party tool like Doctrine ORM.
Most projects in CF now involve creating some type of consumable CFC Endpoint or API Service... do you Unit test your API, do you use Integration Tests on your API? How many ways do you test your API? Not all tests are created equal.
We build our CFCs and CF API to be consumed with CF Apps, Mobile Apps, Javascript apps and devices we haven’t even thought about yet. To be smart developers we need to be able to test our CFC endpoints, and the code that uses those endpoints.
We’ll learn how to test your API serverside with Testbox and Clientside with Jasmine.
With Testbox and Jasmine both using BDD, your test code can almost be isomorphic.
Attendees should have some exposure to CFCs as endpoints, or CF API creations, and consuming with other languages/devices, in this case, JavaScript.
Attendees will learn
How to use Testbox to test your CFCs 2 different ways
Different types and ways to test JavaScript
Overview of client/server side testing tools
Building testing into your workflow
You are one of many that are not testing your APIs thoroughly
Challenges when building high profile editorial sitesYann Malet
This talk will be a walk through the challenges encountered when building a high profile editorial sites. My goal is to present some of the common pitfalls we have encountered at Lincoln Loop and to explain how we solved:
* Legacy migration always take longer
* devops
* Multiple environment
* Easy deployment
* Responsive design impacts the backend
* Journey of an image
* Picturefill.js
* Danger of reusing published django applications
* Caching strategy
* Html fragment
* Varnish
Audience Decision maker that are going to rebuild their magazine Developer bidding for this kind of projects for the first time
Most scalability bottlenecks come from code containing locks, producing significant contention under heavy loads. We'll cover striping, copy-on-write, ring buffer, spinning, to reduce this contention or get lock free code & explain concepts like Compare-And-Swap and memory barriers.
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017Ortus Solutions, Corp
Testbox is a tool we all should be using to test our ColdFusion Applications which was created and is maintained by Ortus Solutions, the people that brought you ColdBox. We will have Gavin from Ortus in house on this day to go over some testbox examples, talk about its importance, and answer any questions you have.
SO --- if you have and high level questions for Gavin, reply to this post (or hit me up) so I can get the questions to Gavin a head of time just in case he needs to consult others at Ortus.
Gavin Pickin is a proud ColdFusion developer, starting with ColdFusion in the late 90s. Now working with Ortus Solutions, a leading force in CFML Development frameworks and tools, Gavin gets to work on a lot of great projects, for a big variety of clients. At Ortus Solutions, a big focus is on free and open source tools, on open source Fridays, Gavin spend most of his open source time working on ContentBox Content Management System.
A book for learning puppet by real example and by building code. Third chapter shows a basic use case of installing tomcat and creating a module to do the same.
Slides for a pre-conference workshop I delivered together with Johan Abildskov (@randomsort) at Git Merge 2017 in Brussels.
In the workshop we covered fun things to do with Git hooks, Git attributes and custom drivers.
In the first half, we demonstrate how you can implement a fully local continuous integration workflow using git hooks.
In the second half, we cover cool and creative ways to diff binary files and custom filters for modifying file content while commit'ing.
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios CoreNagios
Eric Mislivec's presentation on getting started with Nagios Core. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference.
Ganeti Web Manager: Cluster Management Made SimpleOSCON Byrum
Looking for an easy, scalable way to manage your Ganeti-based clusters? Ganeti Web Manager provides admins an easy to deploy, Django based GUI that effectively manages private clusters & works equally well for providing customers access. With a caching system designed to scale to thousands of virtual machines without decreasing performance, Ganeti Web Manager makes cluster management truly simple.
Dennis Matotek, Technical Lead Platforms at Experian Hitwise Australia, gave an excellent presentation on setting up puppet using vagrant, puppet and testing, including a full demo of rspec-puppet and Jenkins.
Cooking Perl with Chef: Real World Tutorial with JitterbugDavid Golden
This tutorial provides a command-by-command walk-through for deploying the Jitterbug continuous integration application using the Chef configuration management tool
PECL Picks - Extensions to make your life betterZendCon
One of the biggest strengths of PHP is its "glue" power. Take any C library and with a little magic and a compiler you have a fantastic extension. These extensions hide in PECL, but few people can tell the good from the unmaintained or just plain broken. Find the best extensions for your project, learn about PECL, and find out how to become a part of the PECL developer community.
Kicking off with Zend Expressive and Doctrine ORM (ZendCon 2016)James Titcumb
You've heard of the new Zend framework, Expressive, and you've heard it's the new hotness. In this talk, I will introduce the concepts of Expressive, how to bootstrap a simple application with the framework using best practices, and how to integrate a third party tool like Doctrine ORM.
Most projects in CF now involve creating some type of consumable CFC Endpoint or API Service... do you Unit test your API, do you use Integration Tests on your API? How many ways do you test your API? Not all tests are created equal.
We build our CFCs and CF API to be consumed with CF Apps, Mobile Apps, Javascript apps and devices we haven’t even thought about yet. To be smart developers we need to be able to test our CFC endpoints, and the code that uses those endpoints.
We’ll learn how to test your API serverside with Testbox and Clientside with Jasmine.
With Testbox and Jasmine both using BDD, your test code can almost be isomorphic.
Attendees should have some exposure to CFCs as endpoints, or CF API creations, and consuming with other languages/devices, in this case, JavaScript.
Attendees will learn
How to use Testbox to test your CFCs 2 different ways
Different types and ways to test JavaScript
Overview of client/server side testing tools
Building testing into your workflow
You are one of many that are not testing your APIs thoroughly
Challenges when building high profile editorial sitesYann Malet
This talk will be a walk through the challenges encountered when building a high profile editorial sites. My goal is to present some of the common pitfalls we have encountered at Lincoln Loop and to explain how we solved:
* Legacy migration always take longer
* devops
* Multiple environment
* Easy deployment
* Responsive design impacts the backend
* Journey of an image
* Picturefill.js
* Danger of reusing published django applications
* Caching strategy
* Html fragment
* Varnish
Audience Decision maker that are going to rebuild their magazine Developer bidding for this kind of projects for the first time
Most scalability bottlenecks come from code containing locks, producing significant contention under heavy loads. We'll cover striping, copy-on-write, ring buffer, spinning, to reduce this contention or get lock free code & explain concepts like Compare-And-Swap and memory barriers.
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017Ortus Solutions, Corp
Testbox is a tool we all should be using to test our ColdFusion Applications which was created and is maintained by Ortus Solutions, the people that brought you ColdBox. We will have Gavin from Ortus in house on this day to go over some testbox examples, talk about its importance, and answer any questions you have.
SO --- if you have and high level questions for Gavin, reply to this post (or hit me up) so I can get the questions to Gavin a head of time just in case he needs to consult others at Ortus.
Gavin Pickin is a proud ColdFusion developer, starting with ColdFusion in the late 90s. Now working with Ortus Solutions, a leading force in CFML Development frameworks and tools, Gavin gets to work on a lot of great projects, for a big variety of clients. At Ortus Solutions, a big focus is on free and open source tools, on open source Fridays, Gavin spend most of his open source time working on ContentBox Content Management System.
A book for learning puppet by real example and by building code. Third chapter shows a basic use case of installing tomcat and creating a module to do the same.
Slides for a pre-conference workshop I delivered together with Johan Abildskov (@randomsort) at Git Merge 2017 in Brussels.
In the workshop we covered fun things to do with Git hooks, Git attributes and custom drivers.
In the first half, we demonstrate how you can implement a fully local continuous integration workflow using git hooks.
In the second half, we cover cool and creative ways to diff binary files and custom filters for modifying file content while commit'ing.
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios CoreNagios
Eric Mislivec's presentation on getting started with Nagios Core. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference.
Hands on Virtualization with Ganeti (part 1) - LinuxCon 2012Lance Albertson
Ganeti is a robust cluster virtualization management software tool. It’s built on top of existing virtualization technologies such as Xen and KVM and other Open Source software. Its integration with various technologies such as DRBD and LVM results in a cheaper High Availability infrastructure and linear scaling.
This hands-on tutorial will cover a basic overview of Ganeti, the step-by-step install & setup of a single-node and multi-node Ganeti cluster, operating the cluster, and some best practices of Ganeti.
Towards Continuous Deployment with DjangoRoger Barnes
It's no secret that python is fantastic when it comes to rapid prototyping and development. When it comes to deploying a web application, the road to glory isn't as well paved and navigating the array of techniques and tools can be daunting.
This talk will address the advantages of continuous deployment, the success factors involved and the tools available, mainly focusing on experiences with Django web development.
Talk from Puppet Camp Paris 2015 by Nicolas Brousse and Julien Fabre, presenting a Continuous Delivery workflow used by the Operations Teams that allowed them to do over 10,000 puppet changes deployment in 2014.
Puppet Camp Silicon Valley 2015: How TubeMogul reached 10,000 Puppet Deployme...Nicolas Brousse
TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they over come challenges.
ContainerCon - Test Driven InfrastructureYury Tsarev
Great external coverage of this presentation can be found at https://www.cedric-meury.ch/2016/10/test-driven-infrastructure-with-puppet-docker-test-kitchen-and-serverspec-yury-tsarev-gooddata/
Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Creating a mature puppet system
1. Creating
a Mature
Puppet System
github.com/rkhatibi/puppetcampla2013
2. Thanks Scale 11x
& PuppetLabs
● Attended 3x (or was it 4?)
● Discovered PuppetLabs at 8x
● I like Puppet (more sleep, better work)
3. Hi, My name is:
Ramin
● Sysadmin for seventeen years
● Currently at SnappyTV
● Yahoo!, Netzero
● Half dozen startups
● Not a ninja or rockstar
4. SnappyTV
● Cloud based video editing
● Immediate publishing
● Real time social media data
consumption and analysis
5. Mature?
● Operable (by more than 1 person)
● Consistent (updates and fresh installs)
● Flexible (hold on while I refactor, again)
● Enjoyable (the opposite of frustrating)
6. Getting There
● Process (remember less, do more)
● Technique (sneaky tricks)
● Documentation (words, boring words)
● Experimentation (aka failure)
7. Are you ready to
write code?
● Software development
● Your environment is important
● Take a few hours to set it up
9. Syntax
Highlighting
● git commit -m 'missing comma'
● git commit -m 'missing quote'
● git commit -m 'I hate my life!!!'
● Defense in depth
10. Code
Snippets
● Add code easily
● Reminders for resource types
● Config to your usage
11. It's
puppet-lint
● It's opinionated, use it anyway
● Chokes on complex quoting
● .puppet-lint.rc
● When in doubt, do what it says
12. Code Style is
Important
● Try to decide on one early
● Use the Puppet Style Guide
● aka puppet-lint
● Consistency is always good
13. Validate Your
Code
● puppet parser validate some.pp
● Doesn't catch everything
● Or work on templates
14. All Together
With VIM
● vim + pathogen
● syntastic, tabular
● vim-puppet, puppet-lint
● mv-vim-puppet
15. Exists for Other
Editors Too
● Emacs (for terrible people)
● Sublime (didn't look mature)
● Eclipse (very nice)
● Anything else?
16. Vagrant
for VMs
● Your experimentation system
● spin VMs up, test, destroy
● Cost of failure is very loooow
17. Testing
in Puppet
● Test from a fresh install
● Easy to miss dependencies
Nothing worse than discovering
ordering problems in prod
18. Puppet
Environments
● Use them, use them, use them
● stage and production
● directories on the master
● --env stage from client
19. Promote Code
to Each Env
● Standard development practice
● devel -> stage -> prod
● There are some caveats
20. Environment
Caveats
● Providers and facts leak
● Best to have an env per Puppet
master instance in adv usage
● This may change (I hope)
21. Setup Simple
Environments
● puppet.stage cname puppet02
● puppet cname puppet01
● Push to one, then the other
22. Clients to Env
● Just add to puppet.conf
● Ideally part of template
● production env is default
[agent]
<% if @fqdn =~ /(.*)stage(.*)/ -%>
environment = stage
<% end -%>
23. Watch Paths
● Might need to rearrange your
repo or push process
● Where will auth.conf live?
./puppet/production/modules/
./puppet/stage/modules/
./puppet/auth.conf
./puppet/hiera.yaml
24. Pushing
Your Code
● puppet_push stage
● puppet_push prod
● rsync, fabric, capistrano, etc
● Steal from your Developers or
reuse your normal process
25. Sync your
plugins
● --pluginsync from cli
● pluginsync=true under [main]
● Default in 3.x
26. Puppet
Master
● Is it ready for production traffic?
● Apache/Passenger is common
● Upgrade to Passenger 3.0.x
● debs/rpms available
27. Tuning
Passenger
● MaxPoolSize = CPU Cores x 2
● MinInstances CPU Cores
● RAM may limit this
● Each Puppet/Rack = 200MB(ish)
28. More Tuning
Passenger
● Use vhost or passenger.conf
● vhost if sharing machine
● PassengerPreStart <url>
● multi Ruby instances in 4.x
29. Apache
Tuning
● mpm-worker > prefork
● should "just work"
● more threads if > 8 cores
● nginx/passenger also an option
30. Other App
Servers
● Little personal experience
● Not worth it in my opinion
● Use what you know best
31. Isolate
the Master
● Easier to manage
● Quite easy to do
● Less likely to make mistakes
32. Like These
Problems
● certname = hostname (no! no!)
● rm -rf /var/lib/puppet/ssl
● puppet:puppet vs root:root
33. Split Your
Modules Too
● include puppet
● include puppetmaster
● shared nothing (almost)
34. Create Dir
Structure
● mkdir -p ./puppet/{etc,rack,var}
● ./puppet/pm.conf
● All in one tree
35. For Masters,
pm.conf
● no complicated concat
● config.ru is the entry point
● ARGV << "--config=pm.conf"
● Also takes other arguments
37. Simple to
Backup
● sudo tar -czvf p.tgz ./puppet/
● that's it
● ignore reports
● always backup certs
38. Can Re-Use
Locally
● rvm, ruby, gem install puppet
● mini puppet environment
● test new setups without affecting
the rest of the server
39. Master
Monitoring
● https:8140
● At least one Rack process
● logwatch
● ask for a catalog
40. Client
Monitoring
● Daemon running (or not)
● last_run_summary.yaml
● Easy to parse
● simple check in my github
41. You Can't
Escape Crons
● delete those reports
● couple of days is fine
● prune nodes in Dashboard
● PuppetDB (not sure yet)
42. Mysql Tuning
● Default my.cnf is useless
● Do at least the following
● Also prune tasks (rake -T)
innodb_buffer_pool_size = 512M
innodb_file_per_table = 1
key_buffer = 32M
43. Certs, not that
complicated
● Master cert
● Client cert
● Application cert
● /etc/hosts is not a solution.
49. Puppet Apply
● good for development
● testing without a puppet master
● aka masterless Puppet
$ puppet apply -l ./test.log manifest.pp
$ puppet apply --modulepath=~/puppet/modules -e "include ntp"
$ puppet apply --catalog catalog.json
50. Your multi-tool
puppet-stdlib
● Does a bit of everything
● validate, replace, convert
● Should be a talk in its own right
51. puppet-stdlib
validation
● One simple example
● Or I'll never finish this talk
● Really
if $order != '' and !is_integer($order) {
fail('Only integers are allowed in the apt::pin order
param')
}
52. Towards a Better
Module
● No god modules
● Each module is a discrete chunk
of functionality
● Apply functionality as needed
53. Code
vs Data
● Data and code separation
● wordpress => db
● puppet => hiera
● Manipulate data, not code
54. Why
Separate?
● Your system will change
● versions, vhosts, aliases
● change code as little as possible
● portability and shared code
55. Write
Less Code
● Default values in your modules
still useful (if Debian do..)
● if { if { if { if { gah!
● Write once, feed data
56. Hiera, as in
hierarchical
● yaml by default, json available
● redis, mongo, mysql, others
● your hierarchy will take a few
tries to get right
57. Hiera, How
does it work?
● Data position matters, it's
hierarchy
● start at the top
● work your way down
● First match or collect
58. Hiera, the
Mistakes
● hiera_array is not for arrays
● hiera_hash is not for hashes
● Just hiera('some_var')
64. Templates
● <% I'm ruby, I execute code %>
● <%= I'll print the output %>
server_id = <%= @ipaddress.split('.').inject(0) {|total,value| (total << 8 ) + value.to_i} %>
expire_logs_days = <%= scope.lookupvar('mysql::data::expire_logs_days') %>
<% if (scope.lookupvar('mysql::data::slaves')).include? @clientcert then -%>
read_only = 1
<% end -%>
65. puppet-concat
● Download from the Forge
● remember pluginsync = true
● Useful for daemons that don't
support configdirs
● sshd, rsync, haproxy (sorta)
67. Augeas
● Single line replacement
● usage is less common
● install the cli tools in devel
● make sure you have installed a
recent version
68. Augeas is
best for...
● Files you can't fully control
● Files you don't want to control
● your last resort
● grub.conf, sysctl.conf
69. Simple
Documentation
● Start by reminding yourself
cat /etc/ntp.conf
# PUPPETHEADER: This file is owned by Puppet.
ls -a /etc/apache2/sites-enabled/
.00_puppet_will_delete_files
.01_that_are_not_directly_managed
.02_by_puppet_you_have_been_warned