Designing Puppet: Roles/Profiles Pattern

Designing Puppet:
Roles / Proﬁles Design Pattern

Puppet Camp Stockholm, Feb 2013

Thursday, 7 February 13

Hello
• Craig Dunn
• Puppet user since 2008 as an IT contractor
• Started with 0.24
• Joined Puppet Labs in June 2012
• @crayﬁshX
• Freenode IRC: crayﬁshx

Agenda
• How people typically design Puppet
• Real-world case study
• Thinking about components
• Designing Puppet for your users
• Node classiﬁcation
• Data separation

Background

• Originally a blog post written in May 2012
• Advocated by many Puppet Labs Engineers
• Based on a real world solution
• Several community members have adopted
with success


Designing Puppet

• You write awesome modules
• You classify them to your node


Designing Puppet

Node Classiﬁcation

Modules


Down the road...
• Your infrastructure grows
• Business requirements will change
• Your Puppet code feels bulky and high
maintenence
• There will always be edge cases eventually
• You decide it needs refactoring

Danger Signs
• Resources being declared in two modules
• You don’t know where your
implementation “ﬁts”
• Lot’s of logic at a node level
• Repetition and duplication
• The if statement is your go-to-guy

Write good modules

• Should manage only it’s own resources
• Should be granular
• Should be portable


Thinking beyond the
module....

• Puppet is a code base
• How do I design an effective framework
• Gluing everything together


Node-level logic

• Risks duplication and repetition
• No guarantee of consistency
• TMI!


Node-level logic
node basil {
class { ‘apache’:
version => ‘latest’,
}
class { ‘motd’: }
class { ‘ssh’: }
class { ‘users’:
default_shell => ‘/bin/false’,
}
Class[‘ssh’] -> Class[‘users’]
}


Node-level logic

• What happens when I have 1000 nodes
• Or 10,000 nodes!!
• That’s a lot of code!
• So where should implement this?


Designing Puppet
• Provide business logic to classiﬁcation
• Provide an abstraction layer for
implementation of components
• Make code adaptable to complex
requirements
• Reduce node-level logic
• Reduce functionality overlap

What is the worse
thing that is
going to happen to
your Puppet code?


Business requirements


Business logic does not
align with technology


Case study

• Real world problem
• Solved through design


Case study
“We have 3
applications we
need to deploy using
Puppet”


The business view

Application X

Application Y Application Z


Go forth and
Puppetize!


Go forth and
Puppetize!
And we jumped right in...


Things got painful


Problems
• These applications aren’t that different
• They seem to share a whole bunch of
similarities
• Implementation differed on different
environments and locations
• Writing 3 separate modules creates
conﬂicts and duplication


Our code was hacky


We are trying to code
business logic.


Stop thinking about
what it looks like

• Break everything down into components
• Granularity is the key
• Think about what it actually is


What we realised

• Each application stack is a collection of a
subset of the same Java apps implemented
in different ways


The technical reality

Application X

ApplicationApplication Z
Y


We only have one
application

Implemented many
different ways

So we had an idea!

• Reduce each Java sub application into
granular Puppet modules
• Create a code layer responsible for
implementation
• Let’s call them proﬁles


class profiles::x {
include tomcat
include mysql
include componenta
include componentb
componentb::resource { ‘name’:
ensure => present,
}
}

class profiles::y {
include tomcat
include mysql
include componenta
include componentc
include componentd
}

class profiles::z {
include tomcat
include mysql
include componenta
include componentb
include componentd
include dependancy
Class[‘dependancy’] -> Class[‘componentd’]
}


Use inheritance for abstraction within proﬁles
class profiles::application {
include tomcat
include mysql
include componenta
}

class profiles::application::x inherits profiles::application {
include componentb
componentb::resource { ‘name’:
ensure => present,
}
}

class profiles::application::y inherits profiles::application {
include componentc
include componentd
}

class profiles::application::z inherits profiles::application {
include componentb
include componentd
include dependancy
Class[‘dependancy’] -> Class[‘componentd’]
}


Proﬁles and
Components

Resources

Proﬁles and
Components

Components: Resource modelling

Resources

Proﬁles and
Components

Proﬁles : Implementation


Resources

In reality it was worse


• 2 different deployment types made up of
over 15 server types each


• 10+ locations


• 10+ locations
• 4 environment types


• 10+ locations
• Every installation was an edge case!


• 10+ locations
• Every installation was an edge case!
• My slides weren’t big enough.


Lessons learned

• Granularity is good
• Don’t assume business logic will directly
translate to technology
• Abstraction is awesome.... but that’s
nothing new....


Abstraction is a core
principle of coding
• Functions are abstracted by methods
• Methods abstracted by classes and modules
• They are abstracted with libraries
• Puppet is code!


Puppet is all about
abstraction
• Data is abstracted by Hiera
• Providers are abstracted by types
• Resources are abstracted by classes
• Classes are abstracted by modules


Puppet is all about
abstraction
• Data is abstracted by Hiera
• Providers are abstracted by types
• Resources are abstracted by classes
• Classes are abstracted by modules
• Modules are abstracted by proﬁles

Focussing on
Abstraction
• We’ve turned business logic into a
technology stack
• Can we translate that back into business
logic?
• Why would we even want to do that?


UAT Cluster node
Our example conﬁguration model:

include security
include users
include ntp
include ssh::server
include customapp
include tomcat::server

class { ‘jenkins’:
require => Class[‘tomcat::server’],
}

include mysql
database { ‘apptest’:
ensure => present,
}


Think about the users
Meet John, Susan and Bill.


John is a Sysadmin

• Wants to ensure all servers have kernel
hardening, NTP and SSH Server installed
• Wants to manage what packages, services,
ﬁles and other resources
• Is responsible for maintaining all the
components of a UAT cluster server


Susan is an application
specialist

• Cares that a UAT Cluster node requires
MySQL Server, Tomcat Server and Jenkins
server installed.


Bill is an IT manager

• Bill cares that the server is a UAT Cluster
node


What do they care
about?

• John cares about modelling all resources
• Susan cares about the technology stack
• Bill cares about the business logic


In Puppet

• Resource modelling is done in component
modules
• The technology stack is deﬁned in proﬁles
• Where do we represent the business logic
for Bill?


Introducing Roles

• Represent business logic, not technology
• Define a set of technology stacks (profiles)
that make up the logical role
• Allow the business to manage how the
infrastructure looks without defining what it
is


A node can only have
one role
• A role can include as many profiles as
required to define itself
• If a node requires two roles, it has by
definition become a new role


A node can only have
one role
• A role can include as many profiles as
required to define itself
• If a node requires two roles, it has by
definition become a new role
• Something couldn’t be a lion and a
kangaroo at the same time!


It would be a Lingaroo


Roles

• One-to-one to nodes
• One-to-many to proﬁles
• Only implement proﬁles


Example role
class role::uat_server {
include profiles::base
include profiles::customapp
include profiles::test_tools
}


Classification

• Node classification simply assigns roles to
nodes
• Roles expose profiles


Classiﬁcation

node ‘craig.puppetlabs.vm’ {
include roles::uat_server
}


Classiﬁcation


The Stack

Resources

The Stack


Resources

The Stack



Resources

The Stack
Roles : Business Logic



Resources

Terminology

• Proﬁles and Roles are Puppet modules
• Components are Puppet modules
responsible for modelling resources
• Everything is a module


Naming conventions
• Components should be named after what
they manage (apache, ssh, mysql)
• Proﬁles should be named after the logical
stack they implement (database, bastion,
email)
• Roles should be named in business logic
convention (uat_server, web_cluster,
application, archive)


Hiera Overview
Let’s talk about data!


Managing infrastructure

Dev



Dev

QA



Dev

QA

Production



Dev

QA DC1

Production



Dev

QA DC1 DC2 DC3

Production


Managing data in
Puppet is hard.


Without Hiera?
if ( $::environment == ‘dev’ ) {
$ntpserver = ‘192.168.2.1’
} else {
if ( $::fqdn == ‘host4.mycorp.com’) {
$ntpserver = ‘127.0.0.1’
} else {
$ntpserver = ‘213.21.6.4’
}
}


With Hiera?
$ntpserver = hiera(‘ntpserver’)


Hierarchical lookups

• Hiera uses facter facts to determine a
hierarchy
• Top down hierarchy for overriding
conﬁguration values based on roles,
environments, locations.... or anything else
• And do this without any coding!


Separation of data from code

• Puppet modules without hard-coded data
are easily shared and more re-usable
• Infrastructure conﬁguration can be
managed without needing to edit Puppet
code


Pluggable Backends
• Source data from multiple locations
• Data source is abstracted from code


Pluggable Backends
• Source data from multiple locations
• Data source is abstracted from code
• hiera-gpg • hiera-redis
• hiera-http • hiera-json
• hiera-mysql • hiera-zookeeper


Data Separation

• Use Hiera to abstract your data from your
code
• Components and proﬁles can source data
from Hiera


Proﬁles and Hiera

• Use Hiera to model your data
• Use proﬁles to model your implementation


The Stack



Resources


The Stack

Hiera:
Data

Resources


Classiﬁcation

• Assigning classes to a node
• You can classify within Puppet code
(site.pp)
• You can use an External Node Classiﬁer
(ENC)


Leveraging an ENC
• You can classify your nodes however you
want
• Puppet Dashboard
• Enterprise Console
• Foreman
• Site.pp
• Custom script

Leveraging an ENC

• An ENC should classify a node to it’s role
• Nothing else


The Stack
Roles : Business Logic Classiﬁer

Hiera:
Data

Resources


Key beneﬁts
• Reduced node-level logic to a role.
• Gain the ability to be ﬂexible with
implementation
• Business logic improves managability by
non-Puppet users
• Edge cases are now easy to solve


Enough Preaching!


This is not the way to
design Puppet... It’s a
way.


Can I implement this
design without roles?


Can I implement this
design without roles?

• Yes.
• You lose the layer of abstraction that
exposes business logic


Can my roles be
deﬁned in my ENC?


Can my roles be
deﬁned in my ENC?

• Yes.
• Keeping it in code makes it versionable


Can’t I just use Hiera
to deﬁne proﬁles?


Can’t I just use Hiera
to define profiles?
• Technically yes.
• You lose the flexibility to implement code
logic in profiles and it may become
restrictive
• You could possibly use: https://github.com/
ripienaar/hiera-puppet-nodes


The fundamental
concepts....


The fundamental
concepts....

• Abstraction, abstraction, abstraction


The fundamental
concepts....

• Decoupling business logic, implementation
and resource modelling.


The fundamental
concepts....

• Separating data and code


The fundamental
concepts....

• Separating data and code
• Reducing node-level complexity

Other Resources

• Adrien Thebos’ excellent blog post http://
sysadvent.blogspot.co.uk/2012/12/day-13-conﬁguration-management-as-
legos.html

• My original blog post
2012/05/239/
http://www.craigdunn.org/

• Module Structure Redux by R.I.Pienaar http://
www.devco.net/archives/2012/12/13/simple-puppet-module-structure-
redux.php


Thank you. Questions?

• Follow me at @crayﬁshX
• Bug me on Freenode: crayﬁshx
Enjoy the rest of Puppet Camp!
In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beer
and helped me on my journey of discovery learning how to implement Puppet properly. R.I.P


Designing Puppet: Roles/Profiles Pattern

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Designing Puppet: Roles/Profiles Pattern

Similar to Designing Puppet: Roles/Profiles Pattern (20)

More from Puppet

More from Puppet (20)

Recently uploaded

Recently uploaded (20)

Designing Puppet: Roles/Profiles Pattern