First session of Casual Config Fridays including an introduction to Hiera and a brief overview to the features of its latest version from the point of view of a service manager at CERN.

2. A brief introduction to Casual Config Fridays
seminars and the idea behind them

3. S M T W T F S
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31
< October >

4. CONFIG TRAINING
A comprehensive introduction
to the Puppet infrastructure
Hostgroups
And we can keep specializing…
It’s everything about grouping
it-puppet-hostgroup-webstuff
webstuff/backendwebstuff/frontend
webstuff/frontend/atlas webstuff/frontend/cms
Welcome to the training
Why are we here?
Puppet infrastructure
Get to know the infrastructure and all the
components that allow us to offer
Configuration management at CERN.
High availability
As part of the training you’ll learn how to
define alias to ensure that at any moment
multiple machines can attend your users’
requests.
Handle secrets in your configuration
No one wants to share a password with people
who shouldn’t have access. Let’s use secrets!
Configure just once
Learning how to configure your services is as
important as defining configuration in a way that
you can replicate it easily across your machines.
Learn the mechanism to share code
Why spend time on tasks that other people in
the community have solved before?
Solve your questions
We’ll cover what we think is important for you to
know, but we are here to solve your questions.
Don’t hesitate to raise your hand if you have any!
Environments
Differences between them
10
production qa new
Production
Machines in this environment will
get all the config from the
master branch of the repositories
and is designed for production
machines that focus on stability.
QA
This environment is usually used
by a portion of the production
machines to test new shared
changes. All the code will come
from qa branches.
New
At any point you can create a
new environment to test
changes in a more isolated way.
4 1

5. Introduction to the basic concepts of Hiera, usage
examples and description of the priority chain

6. /haɪra/
06
First things first
Yeah, pretty much everyone says it wrong…

7. 07
Hiera 101
What it is and how it works
A key/value store abstraction based in YAML files
A key lookup resolution mechanism
A hierarchical data organization
A data composition mechanism (defaults,
overrides, merges…)

8. 08
punch/puppet/ps.yaml
---
osrepos_epel_exclude_pkgs:
- puppetserver
pluginsync_filter:
- archive
- puppetdbquery
- puppetserver
cernpuppet::puppetdb::puppetdb_server: "constable.cern.ch"
A simple
Hiera file
A simple
Hiera file

9. Understanding the hierarchy
The global layer from the most to the least specific
Fully qualified domain name
Data can be assigned to a
specific hostname
Subhostgroups
Searched from most to
least specific
Hostgroup and environment
Different data can be assigned to,
for example, production and qa
Hostgroup and operating system
Each hostgroup has hieradata
available for operating sytem
Toplevel hostgroup
Has the last lookup in
the hostgroup tree

10. 10
Environment
Data specific to the
Puppet environment
Operatingsystem
Global operating
system variables
Hardware vendor
Specific to the vendor (or maintainer)
of the hardware
Module
Data for the module
Common
Global data
Datacenter
Target the different
datacenters (i.e. meyrin)

11. Learning to fetch your data using automatic lookups,
useful functions and command line tools

12. 12
Getting your data
Two different lookup mechanisms
Automatic lookup
Puppet automatically looks for
class parameters values using the
fully qualified name when those are
not explicitly provided
Explicit lookup
Uses Hiera to retrieve the value for
a key, allowing data validation and
different strategies to define how to
fetch data

13. 13
Automatic
class
parameter
lookup
hg_webserver/manifests/backend.pp
class hg_webserver::backend {
include ::yum
}
webserver/backend.yaml
---
yum::clean_old_kernels: false
yum/manifests/init.pp
class yum (
Boolean $clean_old_kernels = true,
) {
...
}

14. 14
code/manifest/webserver/frontend.pp
class hg_webserver::frontend {
$backend_url = lookup('backend_url')
$backend_port = lookup('backend_port', {
'default_value' => 80,
'merge' => 'first',
})
}
webserver/backend.yaml
---
backend_port: 8080
backend_url: "mywebserver.cern.ch"
The lookup
function

15. 15
Merging data
Four different merge behaviors
First
No merge. First found, first used
Unique (array merge)
Combines any number of array
and scalar values into an array
Hash
Combines keys and values of any
number of hashes to return a
merged hash
Deep
Similar to hash, but if the same key
exists in multiple source hashes,
Hiera recursively merges them

16. 16
location/pdx.yaml
profile::server::time_servers: "time.pdx.example.com"
common.yaml
profile::server::time_servers:
- 0.pool.ntp.org
- 1.pool.ntp.org
lookup('profile::server::time_servers', {merge => 'first'})
'time.pdx.example.com’
First
strategy

17. 17
location/pdx.yaml
profile::server::time_servers: "time.pdx.example.com"
common.yaml
profile::server::time_servers:
- 0.pool.ntp.org
- 1.pool.ntp.org
lookup('profile::server::time_servers', {merge => 'unique'})
[
'time.pdx.example.com’,
'0.pool.ntp.org',
'1.pool.ntp.org’,
]
Unique
strategy

18. 18
lookup('site_users', {merge => 'hash'})
{
"ash" => { group => "common", uid => 502, shell => "/bin/zsh" }
"bob" => { group => "ops", uid => 1000 },
"jen" => { group => "ops", uid => 503, shell => "/bin/zsh" },
}
groups/ops.yaml
site_users:
bob:
group: ops
uid: 1000
jen:
group: ops
shell: /bin/zsh
uid: 503
common.yaml
site_users:
ash:
group: common
shell: /bin/zsh
uid: 502
bob:
shell: /bin/bash
uid: 501
Hash
strategy

19. 19
lookup('site_users', {merge => 'deep'})
{
"ash" => { group => "common", uid => 502, shell => "/bin/zsh" }
"bob" => { group => "ops", uid => 1000, shell => "/bin/bash" },
"jen" => { group => "ops", uid => 503, shell => "/bin/zsh" },
}
groups/ops.yaml
site_users:
bob:
group: ops
uid: 1000
jen:
group: ops
shell: /bin/zsh
uid: 503
common.yaml
site_users:
ash:
group: common
shell: /bin/zsh
uid: 502
bob:
shell: /bin/bash
uid: 501
Deep
strategy

20. 20
Deprecated functions
Time to update your manifests
hiera_array
(141 uses)
hiera
(4381 uses)
hiera_hash
(258 uses)
hiera_include
(6 uses)

21. Definition of not-so-static Hiera values using
variables and functions interpolations

22. 23
Interpolating variables
Four different possible sources
Puppet variables
Most common way. Get’s the
value from a Puppet variable
Trusted hash
Accurate values extracted from
the node’s certificate
Facts hash
Contains all node’s facts.
Structured ones are shown up as a
nested structure

23. 22
webserver/backend.yaml
---
# Puppet variable interpolation
server_name: "%{servername}"
# Facts hash interpolation
smtpserver: "mail.%{facts.networking.domain}" # mail.cern.ch
# Trusted hash interpolation
webserver::frontend::backend_url: "%{trusted.hostname}.cern.ch"
Interpolating
variables

24. 24
Interpolating functions
Lookups and beyond
lookup
Looks up a key using Hiera, and
interpolates the values into a string
scope
An alternative way to interpolate a
variable. Not generally useful
literal
A way to write a literal percent
sign (%) without accidentally
interpolating something
alias
Looks up a key using Hiera, and
uses the value as a replacement
for the enclosing

25. 25
webserver/backend.yaml
---
# lookup interpolation
webserver::backend::database_server: "%{lookup('mysql::public_hostname')}"
# scope interpolation
smtpserver: "mail.%{facts.domain}"
smtpserver: "mail.%{scope('facts.domain')}"
# literal interpolation
server_name_string: "%{literal('%')}{SERVER_NAME}"
# alias interpolation
original:
- 'one'
- 'two'
aliased: "%{alias('original’)}"
Interpolating
functions

26. Upgrading and cleaning our code with Hiera 5 and
defining custom strategies

27. 27
Global, environment and module
Three different layers of configuration
Global layer
Define all the levels of
the hierarchy
Environment layer
Merged with the global
layer in our deployment
Module layer
Allows to set defaults for a
module’s class parameters

28. 28
Implementing Hiera 5
The hiera.yaml format
Module level data is defined in a hiera.yaml file
Must include the version (v5)
The hierarchy key configures the data hierarchy
The defaults key define default values for the
backend and datadir keys

29. 29
hiera.yaml
---
version: 5
defaults:
datadir: 'data'
data_hash: 'yaml_data'
hierarchy:
- name: 'Full Version'
path: '%{facts.os.name}-%{facts.os.release.full}.yaml'
- name: 'Major Version'
path: '%{facts.os.name}-%{facts.os.release.major}.yaml'
- name: 'Operating System Family'
path: '%{facts.os.family}-family.yaml'
- name: 'common'
path: 'common.yaml'
A hiera.yaml
example

30. 30
Defining strategies
The lookup_options key
Any data source can set a lookup_options key
This key controls the merge behavior of other keys
Puppet lookups will first check for lookup_options
The lookup_options keys are merged by Puppet
using hash merge before deciding a merge behavior

31. 31
Defining
strategies
code/data/common.yaml
lookup_options:
ntp::servers:
merge: unique
"^profile::(.*)::users$":
merge: deep
ntp::servers: "ntp.cern.ch"
code/data/rhel-7.yaml
lookup_options:
"^profile::(.*)::users$":
merge: hash
# Actual values after the hash_merge
{
"ntp::servers" => { merge => "unique" }
"^profile::(.*)::users$" => { merge => "hash" },
}

32. Things to remember…
Just an small summary
32

34. 34
Some useful links
Click & Go
Migrating to Hiera 5
Best module to use us an example of best practices
Automatic class parameter lookup
Hiera documentation in configdocs

35. 35
And some more…
Again, Click & Go
Interesting talk on Hiera by Hendrik Lindberg
Further options for deep strategy behaviour
More information on interpolation
Deprecated functions and alternatives