Puppet at Spotify


Published on

Learn how Spotify uses Puppet to manage the large and growing amount of servers used to stream music to millions of users. The presenter will also give an introduction to other technologies used to power Spotify.

Erik Dalén
System Engineer, Spotify
Erik is a system engineer within the site reliability engineering at Spotify with a focus on Puppet and automation. He is also a community contributor to Puppet and author of the puppetdbquery tool. Can be found at IRC and Github as dalen.

Published in: Technology, Spiritual
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • ----- Meeting Notes (2013-04-23 13:26) -----2000 resources per second
  • We run debian on all our servers
  • ----- Meeting Notes (2013-04-23 13:26) -----Simplified, can be in other sitesMany backend services implemented using Java & Python
  • Both for backend services and puppet itself
  • Problem when moving module ownership to different teamsAllows conflicting module variants or versions on different servers
  • Puppet at Spotify

    1. 1. August 24, 2013 Erik Dalén Puppet at Spotify
    2. 2. 2Section name ● Over 24 million monthly active users ● Launched in 28 countries ● Over 20 million songs ● More than 1 billion playlists Growing quickly Spotify
    3. 3. SystemEngineer in Site Reliability Engineering at Spotify Operational systemowner for Puppet, playlist systemand Cassandra Community contributor to Puppet whoami
    4. 4. ● Morethan 450 changes per month ● 220committers to ourPuppet git repository ● 325puppet modules Codereview by SRE team using Gerrit Puppet users since 3 years
    5. 5. Puppet Infrastructure ● Roughly 5500 nodes ● 3 different Puppet installations ● Each with their own CAand PuppetDB ● One or more puppetmasters per data centre ● Run using cron
    6. 6. Gitbranch = = Puppet environment Everyonecanpush to private branches and run puppet against those Codereview mandatory to pushto “production” branch Puppet Infrastructure
    7. 7. Built as Debian Packages Deployed using Puppet Backend services
    8. 8. ●Client connections are proxied through the accesspoints ●Most other backend services are stateless ●Storage in Cassandra, PostgreSQL or Tokyo Cabinet Architecture overview Accesspoint Service 1 DB Service 2 DB Service 3
    9. 9. ●Puppet module for the service ●Deployed and tested in test environment ●Hardware requested from SRE team and service deployed in production Backend service deployment
    10. 10. UsingSRV records to discoverservices Puppet module dalen-dnsquery canbe used tolookup them from inside Puppet manifests. Service Discovery
    11. 11. Atthe moment using different ENCs in different Puppet installations Will be switching to using Hiera for node classification Node Classification
    12. 12. Hostname standard: sto1-puppet-a1 <site>-<role>-<pool><poolid> Mainly role fact used to determine the classes amachine should have Node Classification
    13. 13. 25pull requests from Spotify merged to corePuppet andFacter inlast 12 months. Many improvements to puppetlabs modules sent upstream. Puppet contributions
    14. 14. Tiredof writing queries like this? ["and", ["in", "name", ["extract", "certname", ["select-resources", ["and", ["=", "exported", false], ["=", "type", "Class"], ["=", "title", "Cassandra"], ["=", ["parameter", "version"], "1.1"]]]]], ["in", "name", ["extract", "certname", ["select-facts", ["and", ["=", "name", "site"], ["=", "value", "lon"]]]]]] Querying PuppetDB
    15. 15. Finding nodes using dalen-puppetdbquery: $ puppet query nodes ‘Class[Cassandra]{version=‚1.1‛} and site=lon’ $ puppet query nodes ‘processorcount > 16 and manufacturer ~‛Dell.*‛’ github.com/dalen/puppet-puppetdbquery Querying PuppetDB
    16. 16. UsePuppetDB as a backendto the datamapper ORM Node.get(’foo.example.com’).facts.each do |fact| puts “#{fact.name}: #{fact.value}” end dm-puppetdb-adapter
    17. 17. APuppet face to list files managed by puppet # puppet ls /etc/systemd/system nagios-nrpe-server.service declared in /etc/puppet/environments/production/modules/systemd/manifests/unit.pp:15 content from a "content" parameter puppet ls
    18. 18. ● Splitting the repo out ● RemoveSRE review requirement on large parts ● Support testing using vagrant ● Building images using amasterless puppet apply The future
    19. 19. Consists of anode terminus and a forge implementation Builds a per node environment dynamically on demand Will be open sourcedReal Soon Now™ Spikor
    20. 20. Questions?
    21. 21. August 24, 2013 Check out spotify.com/jobs or @Spotifyjobs for more information. Want to join the band?
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.