August 24, 2013
Erik Dalén
Puppet at Spotify
2Section name
● Over 24 million monthly active users
● Launched in 28 countries
● Over 20 million songs
● More than 1 bill...
SystemEngineer in Site Reliability Engineering at Spotify
Operational systemowner for Puppet, playlist systemand Cassandra...
● Morethan 450 changes per month
● 220committers to ourPuppet git repository
● 325puppet modules
Codereview by SRE team us...
Puppet Infrastructure
● Roughly 5500
nodes
● 3 different Puppet
installations
● Each with their own
CAand PuppetDB
● One o...
Gitbranch = = Puppet environment
Everyonecanpush to private branches and run puppet against those
Codereview mandatory to ...
Built as Debian Packages
Deployed using Puppet
Backend services
●Client connections are proxied through the accesspoints
●Most other backend services are stateless
●Storage in Cassandra,...
●Puppet module for the service
●Deployed and tested in test environment
●Hardware requested from SRE team and service depl...
UsingSRV records to discoverservices
Puppet module dalen-dnsquery canbe used tolookup them from inside
Puppet manifests.
S...
Atthe moment using different ENCs in different Puppet installations
Will be switching to using Hiera for node classificati...
Hostname standard:
sto1-puppet-a1
<site>-<role>-<pool><poolid>
Mainly role fact used to determine the classes amachine sho...
25pull requests from Spotify merged to corePuppet andFacter
inlast 12 months.
Many improvements to puppetlabs modules sent...
Tiredof writing queries like this?
["and", ["in", "name", ["extract", "certname", ["select-resources",
["and", ["=", "expo...
Finding nodes using dalen-puppetdbquery:
$ puppet query nodes ‘Class[Cassandra]{version=‚1.1‛} and site=lon’
$ puppet quer...
UsePuppetDB as a backendto the datamapper ORM
Node.get(’foo.example.com’).facts.each do |fact|
puts “#{fact.name}: #{fact....
APuppet face to list files managed by puppet
# puppet ls /etc/systemd/system
nagios-nrpe-server.service
declared in /etc/p...
● Splitting the repo out
● RemoveSRE review requirement on large parts
● Support testing using vagrant
● Building images u...
Consists of anode terminus and a forge implementation
Builds a per node environment dynamically on demand
Will be open sou...
Questions?
August 24, 2013
Check out spotify.com/jobs or @Spotifyjobs for
more information.
Want to join the band?
Upcoming SlideShare
Loading in …5
×

Puppet at Spotify

10,651 views

Published on

Learn how Spotify uses Puppet to manage the large and growing amount of servers used to stream music to millions of users. The presenter will also give an introduction to other technologies used to power Spotify.

Erik Dalén
System Engineer, Spotify
Erik is a system engineer within the site reliability engineering at Spotify with a focus on Puppet and automation. He is also a community contributor to Puppet and author of the puppetdbquery tool. Can be found at IRC and Github as dalen.

Published in: Technology, Spiritual
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
10,651
On SlideShare
0
From Embeds
0
Number of Embeds
1,150
Actions
Shares
0
Downloads
41
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • ----- Meeting Notes (2013-04-23 13:26) -----2000 resources per second
  • We run debian on all our servers
  • ----- Meeting Notes (2013-04-23 13:26) -----Simplified, can be in other sitesMany backend services implemented using Java &amp; Python
  • Both for backend services and puppet itself
  • Problem when moving module ownership to different teamsAllows conflicting module variants or versions on different servers
  • Puppet at Spotify

    1. 1. August 24, 2013 Erik Dalén Puppet at Spotify
    2. 2. 2Section name ● Over 24 million monthly active users ● Launched in 28 countries ● Over 20 million songs ● More than 1 billion playlists Growing quickly Spotify
    3. 3. SystemEngineer in Site Reliability Engineering at Spotify Operational systemowner for Puppet, playlist systemand Cassandra Community contributor to Puppet whoami
    4. 4. ● Morethan 450 changes per month ● 220committers to ourPuppet git repository ● 325puppet modules Codereview by SRE team using Gerrit Puppet users since 3 years
    5. 5. Puppet Infrastructure ● Roughly 5500 nodes ● 3 different Puppet installations ● Each with their own CAand PuppetDB ● One or more puppetmasters per data centre ● Run using cron
    6. 6. Gitbranch = = Puppet environment Everyonecanpush to private branches and run puppet against those Codereview mandatory to pushto “production” branch Puppet Infrastructure
    7. 7. Built as Debian Packages Deployed using Puppet Backend services
    8. 8. ●Client connections are proxied through the accesspoints ●Most other backend services are stateless ●Storage in Cassandra, PostgreSQL or Tokyo Cabinet Architecture overview Accesspoint Service 1 DB Service 2 DB Service 3
    9. 9. ●Puppet module for the service ●Deployed and tested in test environment ●Hardware requested from SRE team and service deployed in production Backend service deployment
    10. 10. UsingSRV records to discoverservices Puppet module dalen-dnsquery canbe used tolookup them from inside Puppet manifests. Service Discovery
    11. 11. Atthe moment using different ENCs in different Puppet installations Will be switching to using Hiera for node classification Node Classification
    12. 12. Hostname standard: sto1-puppet-a1 <site>-<role>-<pool><poolid> Mainly role fact used to determine the classes amachine should have Node Classification
    13. 13. 25pull requests from Spotify merged to corePuppet andFacter inlast 12 months. Many improvements to puppetlabs modules sent upstream. Puppet contributions
    14. 14. Tiredof writing queries like this? ["and", ["in", "name", ["extract", "certname", ["select-resources", ["and", ["=", "exported", false], ["=", "type", "Class"], ["=", "title", "Cassandra"], ["=", ["parameter", "version"], "1.1"]]]]], ["in", "name", ["extract", "certname", ["select-facts", ["and", ["=", "name", "site"], ["=", "value", "lon"]]]]]] Querying PuppetDB
    15. 15. Finding nodes using dalen-puppetdbquery: $ puppet query nodes ‘Class[Cassandra]{version=‚1.1‛} and site=lon’ $ puppet query nodes ‘processorcount > 16 and manufacturer ~‛Dell.*‛’ github.com/dalen/puppet-puppetdbquery Querying PuppetDB
    16. 16. UsePuppetDB as a backendto the datamapper ORM Node.get(’foo.example.com’).facts.each do |fact| puts “#{fact.name}: #{fact.value}” end dm-puppetdb-adapter
    17. 17. APuppet face to list files managed by puppet # puppet ls /etc/systemd/system nagios-nrpe-server.service declared in /etc/puppet/environments/production/modules/systemd/manifests/unit.pp:15 content from a "content" parameter puppet ls
    18. 18. ● Splitting the repo out ● RemoveSRE review requirement on large parts ● Support testing using vagrant ● Building images using amasterless puppet apply The future
    19. 19. Consists of anode terminus and a forge implementation Builds a per node environment dynamically on demand Will be open sourcedReal Soon Now™ Spikor
    20. 20. Questions?
    21. 21. August 24, 2013 Check out spotify.com/jobs or @Spotifyjobs for more information. Want to join the band?

    ×