This document discusses using role-based naming with Puppet to scale node classification. It recommends generating DNS records from inventory data and using Puppet facts and Hiera to assign node roles and classes based on the DNS names. This allows nodes to be treated as disposable "cattle" while maintaining descriptive, human-friendly names. The document provides examples of role name formats and implementing the approach with Facter, Hiera, monitoring configurations, and a CLI tool.

1. Presented by
Cameron Nicholson, Systems Engineer
What's in a name?
•
Using DNS, Facter, and Hiera to scale node classification

2. Two Hard Problems
• Naming things
- today!
• Cache invalidation
- also today, but not as much

3. Names Have Power
• Embody taxonomies

5. Names Have Power
• Embody taxonomies
• Shape thinking

7. Names Have Power
• Embody taxonomies
• Shape thinking
• Potentially limiting
- Misnomers!

8. Source: openstreetmap.org
Rhode Island

9. Honey Badger Mountain Chicken Koala Bear
Mantis Shrimp Red Panda
image source: wikipedia

10. • Static nodes.pp
node srv1.dc1.foo.com { include bar::baz }
node default { }
- Hardware swaps required manual intervention
hand-edit nodes.pp
- node srv1.site.foo.com { include bar::baz }
+ node srv2.site.foo.com { include bar::baz }
• Generated nodes.pp
- Manage class assignments in inventory
- Long generation runs (20+ mins for 5k nodes)
- Required puppet code push
The bad old days
Wherefore art thou, Rolename?

11. Bad times
Wherefore art thou, Rolename?
source https://www.flickr.com/photos/versageek/493800514

12. • Abolish nodes.pp
- Inventory system cannot handle query volume
• DNS is performant. Use that!
- Generate zone files from inventory data
- Use Hiera with custom Puppet facts to assign classes
- DNS easily handles incremental updates
- Scaling DNS for query volume is a solved problem
Better days: Using DNS for ENC
Wherefore art thou, Rolename?

13. Much better
Wherefore art thou, Rolename?
source https://commons.wikimedia.org/wiki/File:Sdtpa_wmf-6.jpg

14. Designing better names
Wherefore art thou, Rolename?
• RFC2100 - machines have many names
https://tools.ietf.org/html/rfc2100
• Nodes are interchangeable - therefore disposable
- nobody is special
- cattle, not pets
http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/
• Rolenames should be regular and descriptive
Good: api0005, web1020, db-redis2004
Bad: srv3502, andromeda, hermes

15. Wherefore art thou, Rolename?
• Primary key / counter - srv#### - srv3287
(-) not inherently meaningful
(+) easy for automated inventory
• Encoded location - {DC ID}{Room/Hall}{Rack}{Slot} - SFO1H5R13U30
(-) difficult to remember
(-) physical location often not relevant to assignment - network gear may
be forgiven
• Assignment - lb0005, api2231
(-) changes / moves with hw changes
(+) easy to understand, human friendly
Common naming schemes

16. The best of both
• Indelible static names - one name per server, forever
easy audit
names can be arbitrary (AWS, for example)
• Descriptive, portable rolenames
CNAMES are easily moved
Human-friendly name scheme
REGEX-parseable
- Example: api0005 -> srv1234
• Cloud-like flexibility on bare metal
Wherefore art thou, Rolename?

17. Anatomy of a rolename
Types
rolename
cluster name
node type node
number
memc-a0001
grouped type
[a-z]+-[a-z][0-9]*
rolename
node type
node
number
api0001
simple type
[a-z]+[0-9]*
rolename
node type
cluster
name
node
number
lb-web0001
cluster
[a-z]+d*-[a-z]+d*

18. Anatomy of a rolename
Example rolenames
simple type
api0001 - api0050 : api logical group 1
api1001 - api1150 : api logical group 2
grouped type
memc-a0001 - memc-a0050 : memcache cluster memc-a
zk-w0001 - zk-w0003 : zookeeper cluster zk-web
cluster
lb-web0001 - lb-web0010 : load balancers for a web service
db-redis0001 - db-redis0050 : redis db cluster
hadoop1-nn0001 - hadoop1-nn0002 : namenodes
hadoop1-zk0001 - hadoop1-zk0003 : zookeepers
hadoop1-dn0001 - hadoop1-dn1500 : datanodes

19. The other hard problem
Cache invalidation
• Rolename TTLs should be short
1h is good
• Static name TTLs should be long
1w works
• Danger! Carriers/Telcos/Public resolvers often ignore short TTLs!

20. • DevOps writes ‘glue’ to bind products together
• Inventory control + DevOps tools
- Use inventory APIs to write zone files and monitoring configs
• DNS update publishes name change
• DNS is “eventual consistency”
- Nuke local caches if necessary
Implementing rolenames
Running the glue factory

21. Implementing rolenames
Ready your adhesives!
source https://commons.wikimedia.org/wiki/File:FiveMinEpoxy.jpg

22. Monitoring
Monitoring
gen script
Puppet
server
Facter
Inventory
DNS
generate git publish
DNS gen
script
Implementing rolenames
Running the glue factory

23. Monitoring
Implementing rolenames
• Run checks against rolenames
- zk0001, NOT srv1234
• Assign checks based on cluster names and node types
- Example: zk* and *-zk* hosts should get zookeeper checks
• Enumerate via inventory API, or parse DNS files
- Can use puppet exec to auto generate on run
- Notify monitoring service resource to reload on change

24. site.pp
Implementing rolenames
Hiera3: hiera_include(‘classes’)
Hiera5: lookup('classes', {merge => unique}).include

25. DNS
site.foo.zone
SOA
include /var/bind/foo.main
include /var/bind/foo.roles
include /var/bind/foo.extra
site.foo.main
srv1234.site.foo.com. A 10.9.8.7
site.foo.roles
api0001.site.foo.com. CNAME srv1234.foo.com.
srv1234.site.foo.com. TXT rolename=api0001
site.foo.extra
puppet.site.foo.com. CNAME puppet0001.site.foo.com.
graphite.site.foo.com. CNAME graphite0001.site.foo.com.
Implementing rolenames

26. Facter
Implementing rolenames
lib/facter/rolename.rb
# use hostname to generate facts
require 'facter'
require 'resolv'
['rolename', 'cluster_name', 'node_type'].each do |fact|
Facter.add(fact) do
setcode do
begin
dns = Resolv::DNS.new()
rec = dns.getresource(Facter.value('fqdn'), Resolv::DNS::Resource::IN::TXT)
txt = rec.data.split('=').pop
rx1 = /(?<rolename>(?<cluster_name>(?<node_type>[a-z]+)-[a-z])d+)$/
rx2 = /(?<rolename>(?:(?<cluster_name>[a-z]+d*)-)?(?<node_type>[a-z]+)d*)$/
data = (rx1.match(txt) or rx2.match(txt))
data[fact]
rescue
fact == 'rolename' ? Facter.value('fqdn').split('.').shift : nil
end
end
end
end
Facter.add(:rolenumber) do
setcode do
Facter.value(:rolename)[/d+$/].to_i
end
end

27. Hiera
:hierarchy:
- '%{::site}/%{::rolename}'
- '%{::site}/%{::cluster_name}/%{::rolename}'
- '%{::site}/%{::cluster_name}/%{::node_type}'
- '%{::site}/%{::cluster_name}'
- '%{::site}/%{::node_type}'
- '%{::site}'
- 'common/%{::rolename}'
- 'common/%{::cluster_name}/%{::rolename}'
- 'common/%{::cluster_name}/%{::node_type}'
- 'common/%{::cluster_name}'
- 'common/%{::node_type}'
- common
Implementing rolenames

28. CLI
Implementing rolenames
bin/rolename
#!/bin/bash
OPT=`getopt -o a:lfx -n rolename -- "$@"`
if [ $? != 0 ] ; then
exit 1
fi
eval set -- "$OPT"
ATTR="rolename"
while true; do
case "$1" in
-a) ATTR=$2; shift 2;;
-l) LONG=1; shift;;
-f) FAIL=1; shift;;
-x) TESTFAIL=1; shift;;
--) shift; break;;
*) shift;;
esac
done
HOST=${1:-`/bin/hostname -f`}
OUT=`/usr/bin/host -t txt $HOST 2>&1`
if [ $? -ne 0 ] || [ -n "$TESTFAIL" ]; then
if [ -n "$FAIL" ]; then
echo "dns error"
exit 1
fi
if [ -n "$LONG" ] || [ `expr $HOST : "[1-9]"` -ne 0 ]; then
echo $HOST
else
echo ${HOST%%.*}
fi
exit 0
fi
IFS="
"
for line in $OUT; do
case $line in
*" descriptive text "$ATTR="*)
if [ -n "$LONG" ]; then
N=`echo $line | cut -d" " -f 1`
FULL=.${N#*.}
fi
echo ${line:$((47+${#ATTR})):-1}${FULL}
exit 0
;;
*" has no TXT record")
if [ -n "$LONG" ]; then
echo $HOST
else
echo ${HOST%%.*}
fi
exit 0
;;
*"domain name pointer "*)
eval set -- "$OPT"
hostname=${line/* domain name pointer /}
exec ${@:0:$#} ${hostname%.}
;;
*)
continue
;;
esac
done
if [ -z "$found" ] && [ -n "$FAIL" ]; then
echo "$ATTR record not found"
exit 1
fi
echo ${HOST%%.*}${FULL}
exit 0

29. • Role-based naming
- Puppet facts / regex parse
• Portable DNS names
• Hiera hierarchy
Implementing rolenames
Modularity

30. Rolenames and profiles pattern
common/api.yaml - (api####.*.foo.com)
classes:
- profiles::api
common/db-redis.yaml - (db-redis####.*.foo.com)
- profiles::redis
Implementing rolenames
Rolenames and profiles
Rolenames as profiles pattern (don’t do this - that way lies madness!)
common/db-redis.yaml
- redis
- logrotate::redis
- internal::dbusers
- collectd::redis

31. simple keys
Examples
common.yaml (*.foo.com)
classes:
- ’profiles::java’
profiles::java::version: ‘8u131’
lab.yaml (*.lab.foo.com)
profiles::java::version: ‘8u144’
lab/hadoop1.yaml (hadoop1-*.lab.foo.com)
profiles::java::version: ‘8u131’
:hierarchy:
- '%{::site}/%{::cluster_name}'
- '%{::site}'
- common

32. hashed configs
Examples
common/lb.yaml (lb-*.*.foo.com)
classes:
- profiles::lb
profiles::lb::config:
anycast_ip: ’10.10.10.10/32’
listen_port: ‘443’
sfo/lb-a.yaml (lb-a####.sfo.foo.com)
profiles::lb::config:
anycast_ip: ‘172.10.10.10/32’
lab/lb-t0001.yaml (lb-t0001.lab.foo.com)
profiles::lb::config:
anycast_ip: ‘192.10.10.10/32’
listen_port: ‘8080’
Puppet3:
/etc/puppet/heira.yaml
:merge_behavior: deeper
profiles/lb.pp
conf = hiera_hash(‘profiles::lb::config’)
Puppet5:
profiles/lb.pp
conf = lookup(‘profiles::lb::config’, {merge =>
deep}).include
:hierarchy:
- '%{::site}/%{::rolename}'
- '%{::site}/%{::cluster_name}'
- 'common/%{::cluster_name}'

33. hadoop cluster
Examples
dc1.yaml (*.dc1.foo.com)
profiles::hadoop::client::cluster: ‘hadoop1’
dc1/worker.yaml (worker####.dc1.foo.com)
classes:
- profiles::hadoop::client
dc1/hadoop1.yaml (hadoop1-*.dc1.foo.com)
hadoop::version: ‘5.12.1’ #cdh version
hadoop::cluster::namenodes:
- “%{::cluster_name}-nn0001.%{::domain}”
- “%{::cluster_name}-nn0002.%{::domain}”
hadoop::cluster::zookeepers:
- “%{::cluster_name}-zk0001.%{::domain}”
- “%{::cluster_name}-zk0002.%{::domain}”
- “%{::cluster_name}-zk0003.%{::domain}”
hadoop::cluster::data_volumes: [‘0’, ‘1’, ‘2’]
common/nn.yaml (hadoop1-nn####.*.foo.com)
classes:
- profiles::hadoop::namenode
common/zk.yaml (hadoop1-zk####.*.foo.com)
classes:
- profiles::hadoop::zookeeper
common/dn.yaml (hadoop1-dn####.*.foo.com)
classes:
- profiles::hadoop::datanode
:hierarchy:
- '%{::site}/%{::cluster_name}'
- '%{::site}/%{::node_type}'
- '%{::site}'
- 'common/%{::node_type}'

34. memcache cluster
Examples
common/memc-a.yaml (memc-a*.*.foo.com)
classes:
- profiles::memcache
profiles::memcache::options:
listen_address: ‘0.0.0.0’
max_connections: 1000
common/memc-a/memc-a0001.yaml (memc-a0001.*.foo.com)
profiles::memcache::dashboard: enable
profiles::memcache::dashboard_url: “%{::rolename}”
lab/memc-a0001.yaml (memc-a0001.lab.foo.com)
profiles::memcache::debug: true
:hierarchy:
- '%{::site}/%{::rolename}'
- 'common/%{::cluster_name}/%{::rolename}'
- 'common/%{::cluster_name}'

35. TM and © 2017 Apple Inc. All rights reserved.