A metadata ocean in
Puppet and Chef
FOSDEM’14

Marc Cluet

DevOps & Automation Advisory
Rackspace

http://www.flickr.com/p...
Who am I?!
Marc Cluet (@lynxman)
More than 16 years of experience as a
SysAdmin
Founding member of Juju and MAAS
while at ...
What is Rackspace?!
Founded in 1998 in San Antonio, TX by
three guys that wanted to create a hosting
company
Home of Fanat...
What is Metadata?!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
/http://www.flickr.com/photos/boojee/3743753784/
What is Metadata?!
Metadata is “data about data”!
Structural Metadata!
Descriptive Metadata!

RACKSPACE® HOSTING | WWW.RAC...
Metadata in Chef!
Metadata is compiled on the client
at execution time!
Metadata is always stored in its
entirety in the n...
Metadata in Chef!
You can add metadata in your!
Attribute File!
Node/Recipe!
Environment!
Role!

RACKSPACE® HOSTING

|

WW...
Organisation in Chef!

http://docs.opscode.com/essentials_environments.html

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Metadata in Puppet!
Metadata is always compiled on
execution time	
Metadata vars can’t be reassigned!
$::var = val	

RACKS...
Metadata in Puppet!
You can add metadata in your!
!Class vars!
!Nodes!
!Top Scope!

RACKSPACE® HOSTING

|

WWW.RACKSPACE.C...
Scoping in Puppet!

http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html

RACKSPACE® HOSTING

|

WWW.RACKSPA...
Hiera in Puppet!
Solves metadata fragmentation issues!
Defines hierarchy for easy organisation!
All in clean and properly s...
Hiera in Puppet!
:backends:	
- yaml	
- json	
:yaml:	
:datadir: /etc/puppet/hieradata	
:json:	
:datadir: /etc/puppet/hierad...
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/musebrarian/3289649684/
Bad metadata practices!
In Chef!
Not keeping consistency between
cookbook scope and environment
scope!
Not removing metada...
Bad metadata practices!
In Puppet!
Directly assigning metadata in a class
without a default value!
Assigning vars in nodes...
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/deepstereo/9616389940/
Good behaviours!
Separate vars from execution!
Do not store vars on the execution
area (classes/cookbooks)!

High abstract...
Good behaviours!
Global

Environment

Node

RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/monica_andre/3119991125/
Metadata Across Providers!
Provisioner

Config Mgmt

cloud-init

Puppet

$dog = cute	

$::dog = big	

WHO IS RIGHT?!?!
RAC...
Metadata Across Providers!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/pss/233841567/
Hard Problem to Solve!
Metadata brain split problem!
Gets worse by adding more players!

System view varies!
Not same info...
Possible Solutions!
Central DB using something fast!
!Redis, mongoDB, mySQL!
Metadata translators might fix it?!
!Write for...
Hard Problem to Solve!

http://xkcd.com/927/
RACKSPACE® HOSTING

|

WWW.RACKSPACE.CO.UK
Possible Solutions!
Scope vars from top of chain!
Pass them over as ext facts!
$dog in cloud-init becomes $::dog in
puppet...
@lynxman!
http://slideshare.net/lynxmanuk/!
http://www.rackspace.co.uk/technicalservices/ !
!

RACKSPACE® HOSTING
|
UK SAL...
Upcoming SlideShare
Loading in …5
×

A Metadata Ocean in Chef and Puppet

2,529
-1

Published on

What is Metadata and how it is represented in Chef and Puppet? What are the challenges for metadata sharing across providers and possible solutions.

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,529
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
17
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

A Metadata Ocean in Chef and Puppet

  1. 1. A metadata ocean in Puppet and Chef FOSDEM’14 Marc Cluet DevOps & Automation Advisory Rackspace http://www.flickr.com/photos/elias_daniel/7470265444/
  2. 2. Who am I?! Marc Cluet (@lynxman) More than 16 years of experience as a SysAdmin Founding member of Juju and MAAS while at Canonical Leading a DevOps Engineering Team at Rackspace Likes DevOps, programming and long walks on the beach RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  3. 3. What is Rackspace?! Founded in 1998 in San Antonio, TX by three guys that wanted to create a hosting company Home of Fanatical Support /o/ Second biggest Public Cloud in the world OpenStack Project co-founder RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  4. 4. What is Metadata?! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK /http://www.flickr.com/photos/boojee/3743753784/
  5. 5. What is Metadata?! Metadata is “data about data”! Structural Metadata! Descriptive Metadata! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/ggvaidya/8256283184/
  6. 6. Metadata in Chef! Metadata is compiled on the client at execution time! Metadata is always stored in its entirety in the node[] dictionary! $node[var] = val RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  7. 7. Metadata in Chef! You can add metadata in your! Attribute File! Node/Recipe! Environment! Role! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  8. 8. Organisation in Chef! http://docs.opscode.com/essentials_environments.html RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  9. 9. Metadata in Puppet! Metadata is always compiled on execution time Metadata vars can’t be reassigned! $::var = val RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  10. 10. Metadata in Puppet! You can add metadata in your! !Class vars! !Nodes! !Top Scope! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  11. 11. Scoping in Puppet! http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  12. 12. Hiera in Puppet! Solves metadata fragmentation issues! Defines hierarchy for easy organisation! All in clean and properly styled in either YAML or JSON! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  13. 13. Hiera in Puppet! :backends: - yaml - json :yaml: :datadir: /etc/puppet/hieradata :json: :datadir: /etc/puppet/hieradata :hierarchy: - "%{::clientcert}" - "%{::custom_location}" - common RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  14. 14. RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/musebrarian/3289649684/
  15. 15. Bad metadata practices! In Chef! Not keeping consistency between cookbook scope and environment scope! Not removing metadata from nodes when deprecated! Directly assigning metadata in a cookbook without a default attribute! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  16. 16. Bad metadata practices! In Puppet! Directly assigning metadata in a class without a default value! Assigning vars in nodes.pp! Not assigning global vars in site.pp! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  17. 17. RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/deepstereo/9616389940/
  18. 18. Good behaviours! Separate vars from execution! Do not store vars on the execution area (classes/cookbooks)! High abstraction, good hierarchy! Metadata is all about context! ! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  19. 19. Good behaviours! Global Environment Node RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  20. 20. RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/monica_andre/3119991125/
  21. 21. Metadata Across Providers! Provisioner Config Mgmt cloud-init Puppet $dog = cute $::dog = big WHO IS RIGHT?!?! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  22. 22. Metadata Across Providers! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/pss/233841567/
  23. 23. Hard Problem to Solve! Metadata brain split problem! Gets worse by adding more players! System view varies! Not same info from bios than from a user level program as facter or ohai! ! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  24. 24. Possible Solutions! Central DB using something fast! !Redis, mongoDB, mySQL! Metadata translators might fix it?! !Write for everything! Creates new SPoF! !Hard to solve without throwing $$! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  25. 25. Hard Problem to Solve! http://xkcd.com/927/ RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  26. 26. Possible Solutions! Scope vars from top of chain! Pass them over as ext facts! $dog in cloud-init becomes $::dog in puppet! RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
  27. 27. @lynxman! http://slideshare.net/lynxmanuk/! http://www.rackspace.co.uk/technicalservices/ ! ! RACKSPACE® HOSTING | UK SALES: +44 (0)20 8712 6507 RACKSPACE® HOSTING | © RACKSPACE US, INC. | 5 MILLINGTON ROAD | HAYES, UNITED KINGDOM UB3 4AZ | UK SUPPORT: 0800 988 0300 | WWW.RACKSPACE.CO.UK RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.CO.UK 27

×