This document discusses how Puppet, Mcollective, and related tools have advanced to help users take DevOps to the next level. It summarizes new features in Puppet 3.0 like improved speed and scalability, Hiera for hierarchical configuration data, and PuppetDB for fast, scalable storage of Puppet data. It also outlines advances in Mcollective for orchestration and tools like Geppetto and the Puppet Forge. The document argues these tools help users achieve higher levels of speed, scalability, extensibility, reliability, flexibility, agility, and community involvement.

1. Taking Devops to the Next Level!
Max Martin
Developer & Manager
@maxmartin
1

2. So
you’ve
automated
your
infrastructure…
• Configura7on
as
code
• Be9er
insight
into
the
structure
of
your
systems
• More
robust
deployments
• But
how
do
you
reach
the
next
level?
2

3. Advances
in
the
Puppet
Ecosystem
•
•
•
•
•
•
3
Puppet
3.0+
Hiera
PuppetDB
Mcollec7ve
2.0+
Geppe9o
Puppet
Forge

4. 2011
2012
2013
Puppet Enterprise 1.0
You Are Here
Puppet Enterprise 3.0
• Configuration Management
• Reporting
Puppet Enterprise 2.0
• Discovery
• Cloud Provisioning
• Orchestration
•
•
•
•
•
Enhanced Scalability & Performance
Dynamic Discovery
Progressive Deployment
Orchestration for Windows
Module Reusability
Puppet Enterprise 2.5
• Windows
• Puppet Forge Integration
• Puppet Data Library
Puppet Enterprise 2.6/2.7
PuppetDB 0.9.0
• Big Data for Puppet
• Fast, scalable Stored
Configs
• RBAC and external auth
• Certificate Management
Puppet Enterprise 2.8
• AIX
Mcollective 2.0
• Sub-collectives
• Advanced Filter Syntax
4
Puppet OSS 3.0
• Performance
• Data Bindings
Puppet OSS 3.1 and 3.2
• API docs
• Future parser
Mcollective 2.2
• Direct addressing
• Pluggable Discovery
PuppetDB 1.4
* Report Storage

5. Puppet
3.0
–
Speed
and
Scalability
5
+100%
2.7x Speed-up

6. Puppet
3.0
–
Hiera
func7ons
+
data
bindings
• Hiera:
hierarchical
key-­‐value
store
• Keeps
site-­‐specific
data
out
of
Puppet
code
• Parameter
values
are
now
automa7cally
looked
up
in
Hiera
• Hiera
func7ons
in
core
give
7ghter
integra7on
6

7. Puppet
3.0
–
Hiera
Example
node ”ntp.example.com" {
class { "ntp":
servers
=> [ '0.us.pool.ntp.org
'1.us.pool.ntp.org
'2.us.pool.ntp.org
'3.us.pool.ntp.org
autoupdate => false,
restrict => false,
enable => true,
}
}
7
iburst’,
iburst',
iburst',
iburst'],

8. Puppet
3.0
–
Hiera
Example
{
"ntp::restrict" : false,
"ntp::autoupdate" : false,
"ntp::enable" : true,
"ntp::servers" : [
"0.us.pool.ntp.org iburst",
"1.us.pool.ntp.org iburst",
"2.us.pool.ntp.org iburst",
"3.us.pool.ntp.org iburst"
]
}
8

9. Puppet
3.0
–
Hiera
Example
node "ntp.example.com" {
include ntp
}
9

10. PuppetDB
• Fast,
durable
data
storage
for
data
generated
by
Puppet,
including:
– Facts
– Catalogs
– Reports
– Exported
resources
• Replaces
exis7ng
libraries
(including
storeconfigs
and
the
old
Inventory
Service
backend),
is
much
faster
&
more
reliable
10

11. PuppetDB
-­‐
Technology
• Based
on
PostgreSQL:
reliable,
ba9le-­‐tested
• Wri9en
in
Clojure:
fast,
mul7-­‐core
• Based
on
the
JVM:
can
be
shipped
as
a
JAR
11

12. PuppetDB
-­‐
Technology
•
•
•
•
12
Thousands
of
PuppetDB
deployments
Hundreds
of
threads
per
install
Zero
deadlocks
Zero
bugs
involving
state

13. PuppetDB
–
Query
Syntax
and
APIs
• AST-­‐based
query
syntax
["and", ["=", "type", "User”],
["=", "title", ”max"]]
["and",
["=”,["fact", "operatingsystem"], "Debian"],
["<”,["fact", "uptime_seconds"], 10000]]
• API
for
defining
your
own
query
syntax
– Erik
Dalén,
Spo7fy:
h9ps://github.com/dalen/puppet-­‐puppetdbquery
• Rich
APIs
make
it
easy
to
build
your
own
tools
to
consume
Puppet
data
13

14. PuppetDB
–
Event
Inspector
14

15. PuppetDB
-­‐
Puppetboard
h9ps://github.com/nedap/puppetboard
15

16. PuppetDB
-­‐
Metrics
16

17. MCollec7ve
A
powerful
orchestra7on
engine
17

18. Mcollec7ve
-­‐
Subcollec7ves
$ mco inventory --list-collectives
* [ ==================================== ] 52 / 52
Collective
==========
za_collective
us_collective
uk_collective
de_collective
eu_collective
mcollective
Nodes
=====
2
7
19
24
45
52
Total nodes: 52
18

19. Mcollec7ve
–
Pluggable
Discovery
• Query
a
file:
$ mco rpc service restart service=httpd
--nodes=hosts.txt
• Discover
using
PuppetDB:
$ mco rpc service restart service=httpd W country=uk --dm=puppetdb
• Direct
addressing:
$ mco rpc rpcutil ping –I example.com
19

20. MCollec7ve
–
Ruby
Integra7on
c = rpcclient("service")
c.discover :nodes => File.readline("hosts.txt").map
{|i| i.chomp}
printrpc c.restart(:service => "httpd")
20

21. Geppe9o
•
•
•
•
21
IDE
for
developing
Puppet
modules
&
code
Integrates
with
git
&
SVN
Linux,
OS
X
&
Windows
Debug
your
Puppet
modules
as
you
develop
them,
and
publish
directly
to
the
Puppet
Forge

22. Puppet
Forge
•
•
•
•
•
22
Puppet
Labs’
module
repository
1,500+
community-­‐contributed
modules
Improved
search
Team
of
module
engineers
Standards
&
best
prac7ces

23. Taking
you
to
the
next
level
of…
•
•
•
•
•
•
•
23
Speed
Scalability
Extensibility
Reliability
Flexibility
Agility
Community