Erik Dalén's talk "Puppet at Spotify" from Puppet Camp Stockholm 2013.

1. Puppet at Spotify
Erik Dalén
February 7, 2013

2. Spotify
• Over 20 million monthly active users
• Launched in 17 countries
• Over 20 million songs
• More than 1 billion playlists
Growing quickly

3. Puppet Infrastructure
• Rougly 3500 nodes
• 3 different Puppet installations, each with
their own PuppetDB and CA
• One puppetmaster per data centre

4. Puppet Infrastructure
Puppet users since 2.5 years
• More than 300 changes per month
• 137 committers to our Puppet repository
• 240 puppet modules
Code review by operations using Gerrit

5. Puppet Infrastructure
Git branch
==
Puppet environment

6. Puppet Infrastructure
Backend services built as debian packages
Deployed using Puppet

7. Node Classification
At the moment using different ENCs in different
Puppet installations
Will be switching to using hiera to classify
nodes

8. Node Classification
Hostname standard:
sto1-puppet-a1
<site>-<service>-<pool><poolid>
Mainly service fact used to determine the
classes a machine should have

9. Node classification
• fqdn/%{fqdn}
• service/%{service}/%{spenvironment}/%{service_pool}'
• service/%{service}/%{service_pool}
• service/%{service}/%{spenvironment}
• service/%{service}
• lsbdistcodename/%{lsbdistcodename}
• domain/%{domain}
• site/%{site}
• spenvironment/%{spenvironment}
• common

10. Service discovery
Using SRV records to discover services
Puppet module dalen-dnsquery can be used to
lookup them from inside Puppet manifests.

11. Service discovery
wasd: github.com/spotify/wasd
Ingests Cassandra rings and DNS records across
multiple sites at a regular interval, then exposes
information about hosts and services.
One can ask what a host does, optionally for which sites;
which hosts in a site offer a service; which hosts offer
a service for a site.
Using a REST API.

12. Questions?