1. Designing Puppet:
Roles / Profiles Design Pattern
Puppet Camp Stockholm, Feb 2013
Thursday, 7 February 13

2. Hello
• Craig Dunn
• Puppet user since 2008 as an IT contractor
• Started with 0.24
• Joined Puppet Labs in June 2012
• @crayfishX
• Freenode IRC: crayfishx
Thursday, 7 February 13

3. Agenda
• How people typically design Puppet
• Real-world case study
• Thinking about components
• Designing Puppet for your users
• Node classification
• Data separation
Thursday, 7 February 13

4. Background
• Originally a blog post written in May 2012
• Advocated by many Puppet Labs Engineers
• Based on a real world solution
• Several community members have adopted
with success
Thursday, 7 February 13

5. Designing Puppet
• You write awesome modules
• You classify them to your node
Thursday, 7 February 13

6. Designing Puppet
Node Classification
Modules
Thursday, 7 February 13

7. Down the road...
• Your infrastructure grows
• Business requirements will change
• Your Puppet code feels bulky and high
maintenence
• There will always be edge cases eventually
• You decide it needs refactoring
Thursday, 7 February 13

8. Danger Signs
• Resources being declared in two modules
• You don’t know where your
implementation “fits”
• Lot’s of logic at a node level
• Repetition and duplication
• The if statement is your go-to-guy
Thursday, 7 February 13

9. Write good modules
• Should manage only it’s own resources
• Should be granular
• Should be portable
Thursday, 7 February 13

10. Thinking beyond the
module....
• Puppet is a code base
• How do I design an effective framework
• Gluing everything together
Thursday, 7 February 13

11. Node-level logic
• Risks duplication and repetition
• No guarantee of consistency
• TMI!
Thursday, 7 February 13

12. Node-level logic
node basil {
class { ‘apache’:
version => ‘latest’,
}
class { ‘motd’: }
class { ‘ssh’: }
class { ‘users’:
default_shell => ‘/bin/false’,
}
Class[‘ssh’] -> Class[‘users’]
}
Thursday, 7 February 13

13. Node-level logic
• What happens when I have 1000 nodes
• Or 10,000 nodes!!
• That’s a lot of code!
• So where should implement this?
Thursday, 7 February 13

14. Designing Puppet
• Provide business logic to classification
• Provide an abstraction layer for
implementation of components
• Make code adaptable to complex
requirements
• Reduce node-level logic
• Reduce functionality overlap
Thursday, 7 February 13

15. What is the worse
thing that is
going to happen to
your Puppet code?
Thursday, 7 February 13

16. Business requirements
Thursday, 7 February 13

17. Business logic does not
align with technology
Thursday, 7 February 13

18. Case study
• Real world problem
• Solved through design
Thursday, 7 February 13

19. Case study
“We have 3
applications we
need to deploy using
Puppet”
Thursday, 7 February 13

20. The business view
Application X
Application Y Application Z
Thursday, 7 February 13

21. Go forth and
Puppetize!
Thursday, 7 February 13

22. Go forth and
Puppetize!
And we jumped right in...
Thursday, 7 February 13

23. Things got painful
Thursday, 7 February 13

24. Problems
• These applications aren’t that different
• They seem to share a whole bunch of
similarities
• Implementation differed on different
environments and locations
• Writing 3 separate modules creates
conflicts and duplication
Thursday, 7 February 13

25. Our code was hacky
Thursday, 7 February 13

26. We are trying to code
business logic.
Thursday, 7 February 13

27. Stop thinking about
what it looks like
• Break everything down into components
• Granularity is the key
• Think about what it actually is
Thursday, 7 February 13

28. What we realised
• Each application stack is a collection of a
subset of the same Java apps implemented
in different ways
Thursday, 7 February 13

29. The business view
Application X
Application Y Application Z
Thursday, 7 February 13

30. The technical reality
Application X
ApplicationApplication Z
Y
Thursday, 7 February 13

31. We only have one
application
Implemented many
different ways
Thursday, 7 February 13

32. So we had an idea!
• Reduce each Java sub application into
granular Puppet modules
• Create a code layer responsible for
implementation
• Let’s call them profiles
Thursday, 7 February 13

33. class profiles::x {
include tomcat
include mysql
include componenta
include componentb
componentb::resource { ‘name’:
ensure => present,
}
}
class profiles::y {
include tomcat
include mysql
include componenta
include componentc
include componentd
}
class profiles::z {
include tomcat
include mysql
include componenta
include componentb
include componentd
include dependancy
Class[‘dependancy’] -> Class[‘componentd’]
}
Thursday, 7 February 13

34. Use inheritance for abstraction within profiles
class profiles::application {
include tomcat
include mysql
include componenta
}
class profiles::application::x inherits profiles::application {
include componentb
componentb::resource { ‘name’:
ensure => present,
}
}
class profiles::application::y inherits profiles::application {
include componentc
include componentd
}
class profiles::application::z inherits profiles::application {
include componentb
include componentd
include dependancy
Class[‘dependancy’] -> Class[‘componentd’]
}
Thursday, 7 February 13

35. Profiles and
Components
Resources
Thursday, 7 February 13

36. Profiles and
Components
Components: Resource modelling
Resources
Thursday, 7 February 13

37. Profiles and
Components
Profiles : Implementation
Components: Resource modelling
Resources
Thursday, 7 February 13

38. In reality it was worse
Thursday, 7 February 13

39. In reality it was worse
• 2 different deployment types made up of
over 15 server types each
Thursday, 7 February 13

40. In reality it was worse
• 2 different deployment types made up of
over 15 server types each
• 10+ locations
Thursday, 7 February 13

41. In reality it was worse
• 2 different deployment types made up of
over 15 server types each
• 10+ locations
• 4 environment types
Thursday, 7 February 13

42. In reality it was worse
• 2 different deployment types made up of
over 15 server types each
• 10+ locations
• 4 environment types
• Every installation was an edge case!
Thursday, 7 February 13

43. In reality it was worse
• 2 different deployment types made up of
over 15 server types each
• 10+ locations
• 4 environment types
• Every installation was an edge case!
• My slides weren’t big enough.
Thursday, 7 February 13

44. Lessons learned
• Granularity is good
• Don’t assume business logic will directly
translate to technology
• Abstraction is awesome.... but that’s
nothing new....
Thursday, 7 February 13

45. Abstraction is a core
principle of coding
• Functions are abstracted by methods
• Methods abstracted by classes and modules
• They are abstracted with libraries
• Puppet is code!
Thursday, 7 February 13

46. Puppet is all about
abstraction
• Data is abstracted by Hiera
• Providers are abstracted by types
• Resources are abstracted by classes
• Classes are abstracted by modules
Thursday, 7 February 13

47. Puppet is all about
abstraction
• Data is abstracted by Hiera
• Providers are abstracted by types
• Resources are abstracted by classes
• Classes are abstracted by modules
• Modules are abstracted by profiles
Thursday, 7 February 13

48. Focussing on
Abstraction
• We’ve turned business logic into a
technology stack
• Can we translate that back into business
logic?
• Why would we even want to do that?
Thursday, 7 February 13

49. UAT Cluster node
Our example configuration model:
include security
include users
include ntp
include ssh::server
include customapp
include tomcat::server
class { ‘jenkins’:
require => Class[‘tomcat::server’],
}
include mysql
database { ‘apptest’:
ensure => present,
}
Thursday, 7 February 13

50. Think about the users
Meet John, Susan and Bill.
Thursday, 7 February 13

51. John is a Sysadmin
• Wants to ensure all servers have kernel
hardening, NTP and SSH Server installed
• Wants to manage what packages, services,
files and other resources
• Is responsible for maintaining all the
components of a UAT cluster server
Thursday, 7 February 13

52. Susan is an application
specialist
• Cares that a UAT Cluster node requires
MySQL Server, Tomcat Server and Jenkins
server installed.
Thursday, 7 February 13

53. Bill is an IT manager
• Bill cares that the server is a UAT Cluster
node
Thursday, 7 February 13

54. What do they care
about?
• John cares about modelling all resources
• Susan cares about the technology stack
• Bill cares about the business logic
Thursday, 7 February 13

55. In Puppet
• Resource modelling is done in component
modules
• The technology stack is defined in profiles
• Where do we represent the business logic
for Bill?
Thursday, 7 February 13

56. Introducing Roles
• Represent business logic, not technology
• Define a set of technology stacks (profiles)
that make up the logical role
• Allow the business to manage how the
infrastructure looks without defining what it
is
Thursday, 7 February 13

57. A node can only have
one role
• A role can include as many profiles as
required to define itself
• If a node requires two roles, it has by
definition become a new role
Thursday, 7 February 13

58. A node can only have
one role
• A role can include as many profiles as
required to define itself
• If a node requires two roles, it has by
definition become a new role
• Something couldn’t be a lion and a
kangaroo at the same time!
Thursday, 7 February 13

59. It would be a Lingaroo
Thursday, 7 February 13

60. Roles
• One-to-one to nodes
• One-to-many to profiles
• Only implement profiles
Thursday, 7 February 13

61. Example role
class role::uat_server {
include profiles::base
include profiles::customapp
include profiles::test_tools
}
Thursday, 7 February 13

62. Classification
• Node classification simply assigns roles to
nodes
• Roles expose profiles
Thursday, 7 February 13

63. Classification
node ‘craig.puppetlabs.vm’ {
include roles::uat_server
}
Thursday, 7 February 13

64. Classification
Thursday, 7 February 13

65. The Stack
Resources
Thursday, 7 February 13

66. The Stack
Components: Resource modelling
Resources
Thursday, 7 February 13

67. The Stack
Profiles : Implementation
Components: Resource modelling
Resources
Thursday, 7 February 13

68. The Stack
Roles : Business Logic
Profiles : Implementation
Components: Resource modelling
Resources
Thursday, 7 February 13

69. Terminology
• Profiles and Roles are Puppet modules
• Components are Puppet modules
responsible for modelling resources
• Everything is a module
Thursday, 7 February 13

70. Naming conventions
• Components should be named after what
they manage (apache, ssh, mysql)
• Profiles should be named after the logical
stack they implement (database, bastion,
email)
• Roles should be named in business logic
convention (uat_server, web_cluster,
application, archive)
Thursday, 7 February 13

71. Hiera Overview
Let’s talk about data!
Thursday, 7 February 13

72. Managing infrastructure
Dev
Thursday, 7 February 13

73. Managing infrastructure
Dev
QA
Thursday, 7 February 13

74. Managing infrastructure
Dev
QA
Production
Thursday, 7 February 13

75. Managing infrastructure
Dev
QA DC1
Production
Thursday, 7 February 13

76. Managing infrastructure
Dev
QA DC1 DC2 DC3
Production
Thursday, 7 February 13

77. Managing data in
Puppet is hard.
Thursday, 7 February 13

78. Without Hiera?
if ( $::environment == ‘dev’ ) {
$ntpserver = ‘192.168.2.1’
} else {
if ( $::fqdn == ‘host4.mycorp.com’) {
$ntpserver = ‘127.0.0.1’
} else {
$ntpserver = ‘213.21.6.4’
}
}
Thursday, 7 February 13

79. With Hiera?
$ntpserver = hiera(‘ntpserver’)
Thursday, 7 February 13

80. Hierarchical lookups
• Hiera uses facter facts to determine a
hierarchy
• Top down hierarchy for overriding
configuration values based on roles,
environments, locations.... or anything else
• And do this without any coding!
Thursday, 7 February 13

81. Separation of data from code
• Puppet modules without hard-coded data
are easily shared and more re-usable
• Infrastructure configuration can be
managed without needing to edit Puppet
code
Thursday, 7 February 13

82. Pluggable Backends
• Source data from multiple locations
• Data source is abstracted from code
Thursday, 7 February 13

83. Pluggable Backends
• Source data from multiple locations
• Data source is abstracted from code
• hiera-gpg • hiera-redis
• hiera-http • hiera-json
• hiera-mysql • hiera-zookeeper
Thursday, 7 February 13

84. Data Separation
• Use Hiera to abstract your data from your
code
• Components and profiles can source data
from Hiera
Thursday, 7 February 13

85. Profiles and Hiera
• Use Hiera to model your data
• Use profiles to model your implementation
Thursday, 7 February 13

86. The Stack
Roles : Business Logic
Profiles : Implementation
Components: Resource modelling
Resources
Thursday, 7 February 13

87. The Stack
Roles : Business Logic
Profiles : Implementation
Hiera:
Data
Components: Resource modelling
Resources
Thursday, 7 February 13

88. Classification
• Assigning classes to a node
• You can classify within Puppet code
(site.pp)
• You can use an External Node Classifier
(ENC)
Thursday, 7 February 13

89. Leveraging an ENC
• You can classify your nodes however you
want
• Puppet Dashboard
• Enterprise Console
• Foreman
• Site.pp
• Custom script
Thursday, 7 February 13

90. Leveraging an ENC
• An ENC should classify a node to it’s role
• Nothing else
Thursday, 7 February 13

91. The Stack
Roles : Business Logic
Profiles : Implementation
Hiera:
Data
Components: Resource modelling
Resources
Thursday, 7 February 13

92. The Stack
Roles : Business Logic Classifier
Profiles : Implementation
Hiera:
Data
Components: Resource modelling
Resources
Thursday, 7 February 13

93. Key benefits
• Reduced node-level logic to a role.
• Gain the ability to be flexible with
implementation
• Business logic improves managability by
non-Puppet users
• Edge cases are now easy to solve
Thursday, 7 February 13

94. Enough Preaching!
Thursday, 7 February 13

95. This is not the way to
design Puppet... It’s a
way.
Thursday, 7 February 13

96. Can I implement this
design without roles?
Thursday, 7 February 13

97. Can I implement this
design without roles?
• Yes.
• You lose the layer of abstraction that
exposes business logic
Thursday, 7 February 13

98. Can my roles be
defined in my ENC?
Thursday, 7 February 13

99. Can my roles be
defined in my ENC?
• Yes.
• Keeping it in code makes it versionable
Thursday, 7 February 13

100. Can’t I just use Hiera
to define profiles?
Thursday, 7 February 13

101. Can’t I just use Hiera
to define profiles?
• Technically yes.
• You lose the flexibility to implement code
logic in profiles and it may become
restrictive
• You could possibly use: https://github.com/
ripienaar/hiera-puppet-nodes
Thursday, 7 February 13

102. The fundamental
concepts....
Thursday, 7 February 13

103. The fundamental
concepts....
• Abstraction, abstraction, abstraction
Thursday, 7 February 13

104. The fundamental
concepts....
• Abstraction, abstraction, abstraction
• Decoupling business logic, implementation
and resource modelling.
Thursday, 7 February 13

105. The fundamental
concepts....
• Abstraction, abstraction, abstraction
• Decoupling business logic, implementation
and resource modelling.
• Separating data and code
Thursday, 7 February 13

106. The fundamental
concepts....
• Abstraction, abstraction, abstraction
• Decoupling business logic, implementation
and resource modelling.
• Separating data and code
• Reducing node-level complexity
Thursday, 7 February 13

107. Other Resources
• Adrien Thebos’ excellent blog post http://
sysadvent.blogspot.co.uk/2012/12/day-13-configuration-management-as-
legos.html
• My original blog post
2012/05/239/
http://www.craigdunn.org/
• Module Structure Redux by R.I.Pienaar http://
www.devco.net/archives/2012/12/13/simple-puppet-module-structure-
redux.php
Thursday, 7 February 13

108. Thank you. Questions?
• Follow me at @crayfishX
• Bug me on Freenode: crayfishx
Enjoy the rest of Puppet Camp!
In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beer
and helped me on my journey of discovery learning how to implement Puppet properly. R.I.P
Thursday, 7 February 13