A metadata ocean in
Puppet and Chef
FOSDEM’14

Marc Cluet

DevOps & Automation Advisory
Rackspace

http://www.flickr.com/photos/elias_daniel/7470265444/
Who am I?!
Marc Cluet (@lynxman)
More than 16 years of experience as a
SysAdmin
Founding member of Juju and MAAS
while at Canonical
Leading a DevOps Engineering Team at
Rackspace
Likes DevOps, programming and long
walks on the beach

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
What is Rackspace?!
Founded in 1998 in San Antonio, TX by
three guys that wanted to create a hosting
company
Home of Fanatical Support /o/
Second biggest Public Cloud in the world
OpenStack Project co-founder

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
What is Metadata?!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
/http://www.flickr.com/photos/boojee/3743753784/
What is Metadata?!
Metadata is “data about data”!
Structural Metadata!
Descriptive Metadata!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/ggvaidya/8256283184/
Metadata in Chef!
Metadata is compiled on the client
at execution time!
Metadata is always stored in its
entirety in the node[] dictionary!
$node[var] = val	

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Metadata in Chef!
You can add metadata in your!
Attribute File!
Node/Recipe!
Environment!
Role!

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Organisation in Chef!

http://docs.opscode.com/essentials_environments.html

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Metadata in Puppet!
Metadata is always compiled on
execution time	
Metadata vars can’t be reassigned!
$::var = val	

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Metadata in Puppet!
You can add metadata in your!
!Class vars!
!Nodes!
!Top Scope!

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Scoping in Puppet!

http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Hiera in Puppet!
Solves metadata fragmentation issues!
Defines hierarchy for easy organisation!
All in clean and properly styled in either
YAML or JSON!

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Hiera in Puppet!
:backends:	
- yaml	
- json	
:yaml:	
:datadir: /etc/puppet/hieradata	
:json:	
:datadir: /etc/puppet/hieradata	
:hierarchy:	
- "%{::clientcert}"	
- "%{::custom_location}"	
- common	

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/musebrarian/3289649684/
Bad metadata practices!
In Chef!
Not keeping consistency between
cookbook scope and environment
scope!
Not removing metadata from nodes
when deprecated!
Directly assigning metadata in a
cookbook without a default attribute!
RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Bad metadata practices!
In Puppet!
Directly assigning metadata in a class
without a default value!
Assigning vars in nodes.pp!
Not assigning global vars in site.pp!

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/deepstereo/9616389940/
Good behaviours!
Separate vars from execution!
Do not store vars on the execution
area (classes/cookbooks)!

High abstraction, good hierarchy!
Metadata is all about context!
!

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Good behaviours!
Global

Environment

Node

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/monica_andre/3119991125/
Metadata Across Providers!
Provisioner

Config Mgmt

cloud-init

Puppet

$dog = cute	

$::dog = big	

WHO IS RIGHT?!?!
RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Metadata Across Providers!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/pss/233841567/
Hard Problem to Solve!
Metadata brain split problem!
Gets worse by adding more players!

System view varies!
Not same info from bios than from a
user level program as facter or ohai!
!
RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Possible Solutions!
Central DB using something fast!
!Redis, mongoDB, mySQL!
Metadata translators might fix it?!
!Write for everything!
Creates new SPoF!
!Hard to solve without throwing $$!
RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Hard Problem to Solve!

http://xkcd.com/927/
RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Possible Solutions!
Scope vars from top of chain!
Pass them over as ext facts!
$dog in cloud-init becomes $::dog in
puppet!

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
@lynxman!
http://slideshare.net/lynxmanuk/!
http://www.rackspace.co.uk/technicalservices/ !
!

RACKSPACE® HOSTING
|
UK SALES: +44 (0)20 8712 6507
RACKSPACE® HOSTING

|

© RACKSPACE US, INC.

|

5 MILLINGTON ROAD
|
HAYES, UNITED KINGDOM UB3 4AZ
|
UK SUPPORT: 0800 988 0300
|
WWW.RACKSPACE.CO.UK

RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES.

|

WWW.RACKSPACE.CO.UK

27

A Metadata Ocean in Chef and Puppet