"Puppet at Bazaarvoice" by Dave Barcelo of Bazaarvoice at Puppet Camp Austin 2013.

1. Puppet
at
Bazaarvoice

2. Hi,
my
name
is
Dave.
• DevOps
Engineer
at
Bazaarvoice.
• Started
working
with
puppet
in
2008
while
working
at
Bioware.
First
version
was
0.24.
• At
Bioware,
puppet
configured
over
14k
nodes
that
comprised
of
web
servers,
databases
and
game
servers.
• All
5
datacenters
(in
California,
Virginia,
Ireland,
Australia
and
Texas)
housed
puppet
managed
nodes
that
all
reported
back
to
a
centralized
puppet
dashboard.
• My
contact
info:
– hTp://www.linkedin.com/in/jamesbarcelo

3. Bazaarvoice
Plug
• We
do
embedded
DevOps!
• ApplicaYon
teams
are
responsible
for
their
applicaYon’s
operaYonal
success.
No
centralized
operaYons!
• 2.0
stack
is
100%
in
Amazon!
• Conferences!
• Work
on
awesome
projects
with
spiffy
tech
like
Cassandra
or
ElasYcSearch.
• Send
me
your
resume!
Dave.barcelo@bazzarvoice.com

4. Preview
presentaYon
• Puppet
in
the
legacy
stack.
• Puppet
in
the
Data
Infrastructure
Team.
– Focus
on
client/server.
• Puppet
in
the
Data
Services
Team.
– Focus
on
masterless
puppet.

5. Puppet
in
the
Legacy
Infrastructure
• TradiYonal
puppet
use
with
client/server.
• MulYple
levels
of
inheritance
using
node
inheritance.
• Puppet
managed
instances
are
configured
according
to
DNS
naming
convenYon:
node
/my-­‐hostname/
{
…
…
}

6. • Some
issues
encountered:
– Very
hard
to
work
with.
Very
complex.
– Large
codebase.
Adds
to
complexity.
MulYple
teams
working
with
same
code
base.
– No
confidence
in
making
changes.
Side
effects
feared
ader
code
change.
A
jinga
tower
of
puppet
code.
– Too
many
pivot
points.
Many
places
to
configure.
Adds
to
complexity.
– Lots
of
code
rot.
Had
not
been
refactored.

7. Puppet
master/client
in
Data
Infrastructure
teams

8. Architecture
• Each
server
type
we
care
about
will
be
referenced
by
its
role.
We
only
care
about
roles,
not
hostnames.
• Centered
around
an
uber
IT
tools
server
that
runs
everything
ops
(including
puppet)
to
do
work
in
an
environment.
The
Mothership.
• Hiera
and
parameterized
classes
will
be
used
to
create
generic
puppet
modules
that
can
be
reused
for
different
roles.
• Development
will
be
centered
on
using
puppet
environments
on
the
Mothership
to
protect
devs
from
stepping
on
each
other.

9. Mothership
• Contains
a
cocktail
of
different
applicaYon
tools
for
doing
work
in
the
environment.
Tools
included:
– McollecYve/AcYveMQ
– Puppet
server
– Puppet
managed
operaYon
scripts.
• Motherships
configured
to
be
highly
available
in
regular
AWS
fashion(Autoscaling,
cluster
mulYple
acYveMQ,
etc).
• AdverYses
mulYple
puppet
environments
that
clients
can
switch
between
via
–environments.

10. What
is
a
Universe?

11. Anatomy
of
a
Mothership

12. Methods
of
passing
in
data
Gejng
environment
data
into
puppet
configuraYon.
• Hiera
datastore.
• Puppet
stdlib/tags.txt
• Cloud
formaYon
parameters
–
Universe,
VPC

13. Puppet
Stdlib/facts.d
•
Bootstrap
process
(EC2
user
data)
populates
/
etc/facter/facts.d/tags.txt
with
mappings.
These
mappings
become
facters.
• Example
of
data
in
tags.txt:
– Universe
value.
– Ec2
metadata.
/etc/facter/facts.d/tags.txt:
universe=dev
Tag_region=us-­‐east-­‐1

14. Hiera
datastore
• Hiera
is
used
extensively
where
different
data
needs
to
be
passed
into
puppet
according
to
context.
Different
contexts
would
include:
– Amazon
region.
– Instance
role.
– Universe.
• Example
usage:
$app_version
=
hiera(‘app_version’,
nil)

15. /etc/hiera.yaml
-­‐-­‐
:logger:
console
:hierarchy:
-­‐
%{fqdn}
-­‐
10-­‐team/%{team}/10-­‐region/%{tag_region}/10-­‐universe/%{universe}/10-­‐roles/%{role}
-­‐
10-­‐team/%{team}/10-­‐region/%{tag_region}/10-­‐universe/%{universe}/20-­‐common
-­‐
10-­‐team/%{team}/10-­‐region/%{tag_region}/20-­‐roles/%{role}
-­‐
10-­‐team/%{team}/10-­‐region/%{tag_region}/30-­‐common
-­‐
10-­‐team/%{team}/20-­‐universe/%{universe}/10-­‐roles/%{role}
-­‐
10-­‐team/%{team}/20-­‐universe/%{universe}/20-­‐common
-­‐
10-­‐team/%{team}/30-­‐roles/%{role}
-­‐
10-­‐team/%{team}/40-­‐common
-­‐
40-­‐common
-­‐
environments
:backends:
-­‐
yaml
-­‐
json
:yaml:
:datadir:
/etc/puppet_env/%{environment}/manifests/hieradata
:json:
:datadir:
/etc/puppet/env/global_hieradata

16. Puppet
Environments/Universe

17. Mothership
ExecuYon
Flow

18. Puppet
code
on
the
Mothership
• The
files
that
do
the
things:
– /etc/puppet/puppet.conf
– /etc/puppet/env/global_hieradata/
environments.json
– /etc/puppet/puppetmaster.conf
– /etc/puppet_env/{puppet_env}/…
• manifests/…
• modules/…

19. /etc/puppet/puppet.conf
#
File
managed
by
Puppet.
[main]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
usecacheonfailure
=
true
pluginsync
=
true
factpath
=
$vardir/lib/facter
preferred_serializaYon_format
=
yaml
[user]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
[agent]
runinterval
=
1800
ca_server
=
<%=
ca_srv
%>
server
=
<%=
logical_srv
%>
cerYficate_revocaYon
=
False
environment
=
<%=
environment
%>
report
=
true

20. /etc/puppet/env/global_hieradata/
environments.json
{
"environments":
[
{
"cert":
[
{
"modulepath":
"/etc/puppet_env/cert/modules"
},
{
"manifestdir":
"/etc/puppet_env/cert/manifests"
},
{
"manifest":
"/etc/puppet_env/cert/manifests/site.pp"
}
]
}
}

21. /etc/puppet/puppetmaster.conf
[main]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
usecacheonfailure
=
true
pluginsync
=
true
factpath
=
$vardir/lib/facter
preferred_serializaYon_format
=
yaml
syslogfacility
=
local1
[master]
certname=<%=
certname
%>
ca
=
True
cerYficate_revocaYon=False
dns_alt_names=<%=
logical_srv
%>
ssl_client_header
=
SSL_CLIENT_S_DN
ssl_client_verify_header
=
SSL_CLIENT_VERIFY
autosign
=
true
#
For
puppet
dashboard
reporYng.
reports
=
store,
datadog_reports
<%
if
store_configs
==
true
%>
#
Puppetdb.
storeconfigs
=
true
storeconfigs_backend
=
puppetdb
<%
end
%>
[user]
vardir
=
/var/lib/puppet
logdir
=
/var/log/puppet
rundir
=
/var/run/puppet
ssldir
=
$vardir/ssl
#
Environments
<%
environments.each
do
|env_val|
-­‐%>
<%
env_val.keys.each
do
|env_key|
-­‐%>
[<%=
env_key
-­‐%>]
<%
env_val[env_key].each
do
|env_data|
-­‐%>
<%
env_data.each_pair
do
|k,
v|
-­‐%>
<%=
k
%>
=
<%=
v
-­‐%>
<%
end
%>
<%
end
%>
<%
end
%>
<%
end
%>

22. /etc/puppet_env/{env}/manifests/
site.pp
import
'roles/*.pp'
node
default
{
class
{
'basenode_role':
}
class
{
"$tag_role":
}
}

23. /etc/puppet_env/{env}/manifests/
role/00_basenode.pp
class
basenode_role
{
class
{
‘security’:
}
class
{
‘monitoring’:
}
…..
}

24. /etc/puppet_env/{env}/manifests/
role/mothership_role.pp
class
mothership_role
{
class
{
'puppet':
master
=>
true,
ca_srv
=>
$tag_caserver,
logical_srv
=>
$tag_puppet_server,
}
}

25. Mothership
Dev
Workflow

26. Masterless
Puppet
in
Data
Services
Teams

27. Architecture
SYll
keeping
bits
of
the
Mothership
project:
• ApplicaYons/Services
scoped
in
zookeeper
by
Universe.
• Emphasis
is
put
on
making
things
simpler.
Puppet
code
will
not
be
monolithic.
Individual
applicaYon
teams
will
only
need
to
maintain
there
own
modules/manifests.
• Changes
to
modules/manifests
will
not
impact
other
teams.

28. Methods
of
passing
in
data
• The
usual
suspects:
– Puppet
stdlib/tags.txt.
– Hiera.
– Cloud
formaYon
parameters
–
Universe,
VPC
• Some
new
ones:
– EC2
data/metadata
-­‐>
facter.
– Zookeeper.
– Cloud
formaYon
parameters
-­‐
DeployTag

29. getEC2data_cache.rb
• Script
runs
out
of
/etc/facts.d
that
converts
EC2
data/metadata
into
facts.

30. Zookeeper/Ostrich
• Custom
funcYons
to
pull
data
from
zookeeper
the
same
way
applicaYons
do
discovery.

31. Masterless
ExecuYon
Flow

32. Puppet
code
in
Masterless
• No
more
Mothership.
All
work
is
done
via
puppet
apply.
– /etc/hiera.yaml
– /etc/puppet/manifests/{role}.pp
– /etc/puppet/manifests/00_common.pp
– /etc/puppet/manifests/01_users.pp
– /etc/puppet/modules/…

33. /etc/hiera.yaml
-­‐-­‐
:logger:
console
:hierarchy:
-­‐
%{fqdn}
-­‐
10-­‐universe/%{universe}/10-­‐roles/%{role}
-­‐
10-­‐universe/%{universe}/20-­‐common
-­‐
20-­‐roles/%{role}
-­‐
30-­‐common
:backends:
-­‐
yaml
:yaml:
:datadir:
/etc/puppet/manifests/hieradata

34. /etc/puppet/manifests/{role}.pp
import
'00_common'
node
default
{
#
This
class
contains
common
modules
that
should
be
used
by
all
roles.
class
{
'common':
}
class
{
'acYvemq’:
}
-­‐>
class
{
'mcollecYve':
server
=>
true,
client
=>
true,
}
}

35. /etc/puppet/manifests/
00_common.pp
import
'01_users'
#####################################
#
Common
#####################################
class
common
{
class
{
'stdlib':
}
file
{
'/opt/bazaarvoice':
ensure
=>
directory,
}
#
Authorized
keys
for
project
developers.
class
{
'user_setup':
stage
=>
setup,
}
host
{
'internal_ip':
ensure
=>
'present',
name
=>
$fqdn,
ip
=>
$ipaddress,
}
class
{
'prompt':
}
}

36. /etc/puppet/manifests/01_users.pp
class
user_setup
{
include
users
users::user
{
'dbarcelo':
groups
=>
'wheel',
sshKey
=>
'ssh-­‐dss
AAAAB3NzaC1kc3MAAACBANL1zoZdYJp/6vQ4G5iNQXjdJ7NGmK0J2eqHbztvuD0CBPyqMuEtuYKRg14tFd4iwp5EpnT4UWpv8kXF/
dkEN3b5xgN/R+1hYq7/3mnRLchMFTl0tyryLuARC9zTI003mQrXd/
W9jzXaNlCTpxh8Ihj2Ov3lvAAX65tN9nijxhCTAAAAFQCgMU0obmTLo5CRYtRwDCkj1mb2hQAAAIAiZF9axkCvMa9vwigDiAf3rNMbut1gtqtwdzux8c9T1inApKV5
sccjg5POKm+4WmWTBOtQfYR8cNot2Mn/mO+MRiKH8sYapYnU2es+KRBmhdARE+N7EqdD0WqoP7NrsNVbObHwDQBNkODuc3ZPyTQuqv/
w4poTXaS5u5M1XZbgZwAAAIEAjt4r7SN1I/m0V/
TvedTVxJvKln4wZkFxyI5CAgpsAr435kwSLM08R9Hd0/5Vy9L…YpH1aZTBaoTqmTCtnv3mp1coXoscEp5nE0llfm+4DX3YvWnR80S/OeMUe71Ucm1ORwFpST/
K4WKQoZ30TAVVsc8nYy2hyD7hyozjzsS09o=
dave.barcelo@dbarcelo-­‐mbpro'
}
users::user
{
'lwadhwani':
groups
=>
'wheel',
sshKey
=>
'ssh-­‐rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAysFCPpffw9LIOqAEFZxOOb52m2FbHhumBFc07o8sm3c4cmdLq/
bBtr5TyuQp89zVNEaTGRbw1nMpQCDno4i5ipTvCLoKkOE1PRdtyJw6PGu6VV/0U1ghK+1xmveM2jDX/otj5hjnQiRm1+Fx/
orYwNBkywDlDHZQCGxalWaFgXVyReCRUqq0jBwj3EKJfsQgoxuSrh7F6GjsQ6DUOsA3wBfewS25hPmhulEqvga4/P58BMHemL9d4Ugu98Vg7fgaur/b1adX
+LzbmE6C2T4Gn1kzAOEct6bFgLPRj3n5/EaspdOsZ/Nnik0LUvIwZNHgDCLgkS0D8aMIsiUrB4OqSw==
luveen@Pantalaimon'
}
}

37. /etc/puppet/modules/…
• Do
stuff!
• Code
is
sYll
generic
but
it
does
not
have
to
be.

38. Masterless
Dev
Workflow