Puppet Camp April 9th 2013
What’s in a name?Andrew Fong and Gary Josack     fong@dropbox.com     gary@dropbox.com
About Dropbox•  Thousands of instances/servers•  Mostly Python Stack•  EC2 and Large Self Datacenters•  Over a billion fil...
A story of a startup... Hostapuppet.com
Chapter One•  1 or 2 teams•  Couple of hosts•  Webserver and a database•  Maybe one ops guy
What Ops People Like•  Simplicity•  Repeatability•  Assurances that things remain consistent•  Puppet / Configuration mana...
Config Managementnode www1.example.com {    include common    include apache    include squid  }node db1.example.com {    ...
My First Puppet ConfigNode ‘mickey.hostapuppet.com’ {      include common      include webserver      include sudoers}Node...
Sudoers Module…file { "/etc/sudoers":           owner => root,           group => root,           mode => "440",          ...
Sudoers File itselfHost_Alias DONALD=donald.hostapuppet.comHost_Alias MICKEY=mickey.hostapuppet.comdb_guy DONALD=(all) NOP...
Chapter Two: A growing service•  A few teams•  2 or 3 services•  multiple types of hosts  –  Web  –  API  –  DB
Hostnames•  sjc-web[1-N]•  sjc-db[1-N]•  sjc-api[1-N]
Host Regex$hosttype = inline_template(<%=hostname.sub(/w+-([a-z]+)d*/){$1} %>’)
Hosttypes   $hosttype = inline_template(<%= hostname.sub(/w+-([a-z]+)d*/){$1} %>’)if $hosttype == ‘web’ {                 ...
Back to sudoers…file { "/etc/sudoers":           owner => root,           group => root,           mode => "440",         ...
Sudoers File itselfHost_Alias      WEB=sjc-web*Host_Alias      DB=sjc-db*Host_Alias      API=sjc-api*database_guy DB=(all)...
Hypergrowth           Users (millions)120100 80 60 40 20  0      0      2                  3   5             Users (millio...
Chapter 3: An Expanding            Infrastructure•  Lots of new hires!•  A bunch more developers•  Some PMs•  Some Designers
All Kinds Of Problems…•  Boxes of same hardware class running different services•  Boxes serving more than one role (remem...
Renaming a host•  Update dns•  Update dhcpd.conf•  Push both•  Update puppet configs•  Update code
OMG I JUST RENAME HOSTS!
Sudoers File From Chapter Two…Host_Alias   WEB= sjc-web*Host_Alias   API=sjc-api*Host_Alias   DB=sjc-db*database_guy DB=(a...
Sudoers File in Chapter 3Host_Alias   WEB= sjc-web*Host_Alias   API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553Ho...
Dropbox•  We did all that.•  We’re still paying the taxes for doing  that.•  But there is a light at the end of the  tunnel…
ABSTRACT THE    SERVICEFROM THE HOST!
So what does that mean?• Make hosts role agnostic• Do not require invasive changes• Simple interfaces
Making hosts role agnostic• Positional• Serial Numbers• Anything that doesnt change
The Dropbox Plan• Positional names• Custom Machine Database• External Node Classifier• Transitioning Puppet configs• Namin...
Service/Machine Management             Database• Universal Source of Truth• Manage roles / attributes• Generated configs  ...
What exactly is the ENC• External Node Classifier• Inject variables (and other) from externalprocess• YAML Output
Part 2: External Node Encoders
Sudoers++• Move from monolithic to modular• Includes! (Weird caveats)• Just use ALL for Host_Lists
Sudoers at Dropbox
Part 3: Helper Functions
Sudoers with tags
Sudoers with tags
Provisioning• Preload MDB, DNS, DHCPD, etc.  - Set   it and forget it• Have spares ready for any roles• Assigning a role i...
Dynamic Naming w/ PowerDNS
Dynamic Naming w/ PowerDNS
Zookeeper• ZKNS included with the Vitess project• ZK is in use at various different companies(YouTube, Twitter, AirBnB)
Q&A FAQ #1: Are you hiring?λ    - Yes! Come talk to us. :)
Upcoming SlideShare
Loading in...5
×

Functional Hostnames and Why they are Bad

8,491

Published on

"Functional Hostnames and Why they are Bad" by Andrew Fong and Gary Josack of Dropbox at Puppet Camp SF 2013. Find a Puppet Camp near you and learn more about configuration management: puppetlabs.com/community/puppet-camp/

Published in: Technology

Functional Hostnames and Why they are Bad

  1. 1. Puppet Camp April 9th 2013
  2. 2. What’s in a name?Andrew Fong and Gary Josack fong@dropbox.com gary@dropbox.com
  3. 3. About Dropbox•  Thousands of instances/servers•  Mostly Python Stack•  EC2 and Large Self Datacenters•  Over a billion file syncs per day•  Thousands of MySQL Shards•  4 SREs and 1 DBA
  4. 4. A story of a startup... Hostapuppet.com
  5. 5. Chapter One•  1 or 2 teams•  Couple of hosts•  Webserver and a database•  Maybe one ops guy
  6. 6. What Ops People Like•  Simplicity•  Repeatability•  Assurances that things remain consistent•  Puppet / Configuration management
  7. 7. Config Managementnode www1.example.com { include common include apache include squid }node db1.example.com { include common include mysql }
  8. 8. My First Puppet ConfigNode ‘mickey.hostapuppet.com’ { include common include webserver include sudoers}Node ‘donald.hostapuppet.com’{ include common include db include sudoers}
  9. 9. Sudoers Module…file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", }…
  10. 10. Sudoers File itselfHost_Alias DONALD=donald.hostapuppet.comHost_Alias MICKEY=mickey.hostapuppet.comdb_guy DONALD=(all) NOPASSWD: ALLops_guy MICKEY=(all) NOPASSWD: ALL
  11. 11. Chapter Two: A growing service•  A few teams•  2 or 3 services•  multiple types of hosts –  Web –  API –  DB
  12. 12. Hostnames•  sjc-web[1-N]•  sjc-db[1-N]•  sjc-api[1-N]
  13. 13. Host Regex$hosttype = inline_template(<%=hostname.sub(/w+-([a-z]+)d*/){$1} %>’)
  14. 14. Hosttypes $hosttype = inline_template(<%= hostname.sub(/w+-([a-z]+)d*/){$1} %>’)if $hosttype == ‘web’ { if $hosttype == ‘api’ { include sudoers include sudoers include web include api }}If $hosttype == ‘db’ { include sudoers include db}
  15. 15. Back to sudoers…file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", }…
  16. 16. Sudoers File itselfHost_Alias WEB=sjc-web*Host_Alias DB=sjc-db*Host_Alias API=sjc-api*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API=(all) NOPASSWD: ALL
  17. 17. Hypergrowth Users (millions)120100 80 60 40 20 0 0 2 3 5 Users (millions)
  18. 18. Chapter 3: An Expanding Infrastructure•  Lots of new hires!•  A bunch more developers•  Some PMs•  Some Designers
  19. 19. All Kinds Of Problems…•  Boxes of same hardware class running different services•  Boxes serving more than one role (remember sudoers?)•  Deploying or moving hosts quickly
  20. 20. Renaming a host•  Update dns•  Update dhcpd.conf•  Push both•  Update puppet configs•  Update code
  21. 21. OMG I JUST RENAME HOSTS!
  22. 22. Sudoers File From Chapter Two…Host_Alias WEB= sjc-web*Host_Alias API=sjc-api*Host_Alias DB=sjc-db*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API(all) NOPASSWD: ALL
  23. 23. Sudoers File in Chapter 3Host_Alias WEB= sjc-web*Host_Alias API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553Host_Alias DB=sjc-db*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API(all) NOPASSWD: ALL
  24. 24. Dropbox•  We did all that.•  We’re still paying the taxes for doing that.•  But there is a light at the end of the tunnel…
  25. 25. ABSTRACT THE SERVICEFROM THE HOST!
  26. 26. So what does that mean?• Make hosts role agnostic• Do not require invasive changes• Simple interfaces
  27. 27. Making hosts role agnostic• Positional• Serial Numbers• Anything that doesnt change
  28. 28. The Dropbox Plan• Positional names• Custom Machine Database• External Node Classifier• Transitioning Puppet configs• Naming service(s) for convenient names
  29. 29. Service/Machine Management Database• Universal Source of Truth• Manage roles / attributes• Generated configs - Gmond, Nagios, etc
  30. 30. What exactly is the ENC• External Node Classifier• Inject variables (and other) from externalprocess• YAML Output
  31. 31. Part 2: External Node Encoders
  32. 32. Sudoers++• Move from monolithic to modular• Includes! (Weird caveats)• Just use ALL for Host_Lists
  33. 33. Sudoers at Dropbox
  34. 34. Part 3: Helper Functions
  35. 35. Sudoers with tags
  36. 36. Sudoers with tags
  37. 37. Provisioning• Preload MDB, DNS, DHCPD, etc. - Set it and forget it• Have spares ready for any roles• Assigning a role is one command• No more renames!
  38. 38. Dynamic Naming w/ PowerDNS
  39. 39. Dynamic Naming w/ PowerDNS
  40. 40. Zookeeper• ZKNS included with the Vitess project• ZK is in use at various different companies(YouTube, Twitter, AirBnB)
  41. 41. Q&A FAQ #1: Are you hiring?λ  - Yes! Come talk to us. :)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×