Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
August 24, 2013
Erik Dalén
Puppet at Spotify
2Section name
● Over 24 million monthly active users
● Launched in 28 countries
● Over 20 million songs
● More than 1 bill...
SystemEngineer in Site Reliability Engineering at Spotify
Operational systemowner for Puppet, playlist systemand Cassandra...
● Morethan 450 changes per month
● 220committers to ourPuppet git repository
● 325puppet modules
Codereview by SRE team us...
Puppet Infrastructure
● Roughly 5500
nodes
● 3 different Puppet
installations
● Each with their own
CAand PuppetDB
● One o...
Gitbranch = = Puppet environment
Everyonecanpush to private branches and run puppet against those
Codereview mandatory to ...
Built as Debian Packages
Deployed using Puppet
Backend services
●Client connections are proxied through the accesspoints
●Most other backend services are stateless
●Storage in Cassandra,...
●Puppet module for the service
●Deployed and tested in test environment
●Hardware requested from SRE team and service depl...
UsingSRV records to discoverservices
Puppet module dalen-dnsquery canbe used tolookup them from inside
Puppet manifests.
S...
Atthe moment using different ENCs in different Puppet installations
Will be switching to using Hiera for node classificati...
Hostname standard:
sto1-puppet-a1
<site>-<role>-<pool><poolid>
Mainly role fact used to determine the classes amachine sho...
25pull requests from Spotify merged to corePuppet andFacter
inlast 12 months.
Many improvements to puppetlabs modules sent...
Tiredof writing queries like this?
["and", ["in", "name", ["extract", "certname", ["select-resources",
["and", ["=", "expo...
Finding nodes using dalen-puppetdbquery:
$ puppet query nodes ‘Class[Cassandra]{version=‚1.1‛} and site=lon’
$ puppet quer...
UsePuppetDB as a backendto the datamapper ORM
Node.get(’foo.example.com’).facts.each do |fact|
puts “#{fact.name}: #{fact....
APuppet face to list files managed by puppet
# puppet ls /etc/systemd/system
nagios-nrpe-server.service
declared in /etc/p...
● Splitting the repo out
● RemoveSRE review requirement on large parts
● Support testing using vagrant
● Building images u...
Consists of anode terminus and a forge implementation
Builds a per node environment dynamically on demand
Will be open sou...
Questions?
August 24, 2013
Check out spotify.com/jobs or @Spotifyjobs for
more information.
Want to join the band?
Upcoming SlideShare
Loading in …5
×

Puppet at Spotify

10,951 views

Published on

Learn how Spotify uses Puppet to manage the large and growing amount of servers used to stream music to millions of users. The presenter will also give an introduction to other technologies used to power Spotify.

Erik Dalén
System Engineer, Spotify
Erik is a system engineer within the site reliability engineering at Spotify with a focus on Puppet and automation. He is also a community contributor to Puppet and author of the puppetdbquery tool. Can be found at IRC and Github as dalen.

Published in: Technology, Spiritual
  • Be the first to comment

Puppet at Spotify

  1. 1. August 24, 2013 Erik Dalén Puppet at Spotify
  2. 2. 2Section name ● Over 24 million monthly active users ● Launched in 28 countries ● Over 20 million songs ● More than 1 billion playlists Growing quickly Spotify
  3. 3. SystemEngineer in Site Reliability Engineering at Spotify Operational systemowner for Puppet, playlist systemand Cassandra Community contributor to Puppet whoami
  4. 4. ● Morethan 450 changes per month ● 220committers to ourPuppet git repository ● 325puppet modules Codereview by SRE team using Gerrit Puppet users since 3 years
  5. 5. Puppet Infrastructure ● Roughly 5500 nodes ● 3 different Puppet installations ● Each with their own CAand PuppetDB ● One or more puppetmasters per data centre ● Run using cron
  6. 6. Gitbranch = = Puppet environment Everyonecanpush to private branches and run puppet against those Codereview mandatory to pushto “production” branch Puppet Infrastructure
  7. 7. Built as Debian Packages Deployed using Puppet Backend services
  8. 8. ●Client connections are proxied through the accesspoints ●Most other backend services are stateless ●Storage in Cassandra, PostgreSQL or Tokyo Cabinet Architecture overview Accesspoint Service 1 DB Service 2 DB Service 3
  9. 9. ●Puppet module for the service ●Deployed and tested in test environment ●Hardware requested from SRE team and service deployed in production Backend service deployment
  10. 10. UsingSRV records to discoverservices Puppet module dalen-dnsquery canbe used tolookup them from inside Puppet manifests. Service Discovery
  11. 11. Atthe moment using different ENCs in different Puppet installations Will be switching to using Hiera for node classification Node Classification
  12. 12. Hostname standard: sto1-puppet-a1 <site>-<role>-<pool><poolid> Mainly role fact used to determine the classes amachine should have Node Classification
  13. 13. 25pull requests from Spotify merged to corePuppet andFacter inlast 12 months. Many improvements to puppetlabs modules sent upstream. Puppet contributions
  14. 14. Tiredof writing queries like this? ["and", ["in", "name", ["extract", "certname", ["select-resources", ["and", ["=", "exported", false], ["=", "type", "Class"], ["=", "title", "Cassandra"], ["=", ["parameter", "version"], "1.1"]]]]], ["in", "name", ["extract", "certname", ["select-facts", ["and", ["=", "name", "site"], ["=", "value", "lon"]]]]]] Querying PuppetDB
  15. 15. Finding nodes using dalen-puppetdbquery: $ puppet query nodes ‘Class[Cassandra]{version=‚1.1‛} and site=lon’ $ puppet query nodes ‘processorcount > 16 and manufacturer ~‛Dell.*‛’ github.com/dalen/puppet-puppetdbquery Querying PuppetDB
  16. 16. UsePuppetDB as a backendto the datamapper ORM Node.get(’foo.example.com’).facts.each do |fact| puts “#{fact.name}: #{fact.value}” end dm-puppetdb-adapter
  17. 17. APuppet face to list files managed by puppet # puppet ls /etc/systemd/system nagios-nrpe-server.service declared in /etc/puppet/environments/production/modules/systemd/manifests/unit.pp:15 content from a "content" parameter puppet ls
  18. 18. ● Splitting the repo out ● RemoveSRE review requirement on large parts ● Support testing using vagrant ● Building images using amasterless puppet apply The future
  19. 19. Consists of anode terminus and a forge implementation Builds a per node environment dynamically on demand Will be open sourcedReal Soon Now™ Spikor
  20. 20. Questions?
  21. 21. August 24, 2013 Check out spotify.com/jobs or @Spotifyjobs for more information. Want to join the band?

×