Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Functional Hostnames and Why they are Bad


Published on

"Functional Hostnames and Why they are Bad" by Andrew Fong and Gary Josack of Dropbox at Puppet Camp SF 2013. Find a Puppet Camp near you and learn more about configuration management:

Published in: Technology
  • Be the first to comment

Functional Hostnames and Why they are Bad

  1. Puppet Camp April 9th 2013
  2. What’s in a name?Andrew Fong and Gary Josack
  3. About Dropbox•  Thousands of instances/servers•  Mostly Python Stack•  EC2 and Large Self Datacenters•  Over a billion file syncs per day•  Thousands of MySQL Shards•  4 SREs and 1 DBA
  4. A story of a startup...
  5. Chapter One•  1 or 2 teams•  Couple of hosts•  Webserver and a database•  Maybe one ops guy
  6. What Ops People Like•  Simplicity•  Repeatability•  Assurances that things remain consistent•  Puppet / Configuration management
  7. Config Managementnode { include common include apache include squid }node { include common include mysql }
  8. My First Puppet ConfigNode ‘’ { include common include webserver include sudoers}Node ‘’{ include common include db include sudoers}
  9. Sudoers Module…file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", }…
  10. Sudoers File itselfHost_Alias DONALD=donald.hostapuppet.comHost_Alias MICKEY=mickey.hostapuppet.comdb_guy DONALD=(all) NOPASSWD: ALLops_guy MICKEY=(all) NOPASSWD: ALL
  11. Chapter Two: A growing service•  A few teams•  2 or 3 services•  multiple types of hosts –  Web –  API –  DB
  12. Hostnames•  sjc-web[1-N]•  sjc-db[1-N]•  sjc-api[1-N]
  13. Host Regex$hosttype = inline_template(<%=hostname.sub(/w+-([a-z]+)d*/){$1} %>’)
  14. Hosttypes $hosttype = inline_template(<%= hostname.sub(/w+-([a-z]+)d*/){$1} %>’)if $hosttype == ‘web’ { if $hosttype == ‘api’ { include sudoers include sudoers include web include api }}If $hosttype == ‘db’ { include sudoers include db}
  15. Back to sudoers…file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", }…
  16. Sudoers File itselfHost_Alias WEB=sjc-web*Host_Alias DB=sjc-db*Host_Alias API=sjc-api*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API=(all) NOPASSWD: ALL
  17. Hypergrowth Users (millions)120100 80 60 40 20 0 0 2 3 5 Users (millions)
  18. Chapter 3: An Expanding Infrastructure•  Lots of new hires!•  A bunch more developers•  Some PMs•  Some Designers
  19. All Kinds Of Problems…•  Boxes of same hardware class running different services•  Boxes serving more than one role (remember sudoers?)•  Deploying or moving hosts quickly
  20. Renaming a host•  Update dns•  Update dhcpd.conf•  Push both•  Update puppet configs•  Update code
  22. Sudoers File From Chapter Two…Host_Alias WEB= sjc-web*Host_Alias API=sjc-api*Host_Alias DB=sjc-db*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API(all) NOPASSWD: ALL
  23. Sudoers File in Chapter 3Host_Alias WEB= sjc-web*Host_Alias API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553Host_Alias DB=sjc-db*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API(all) NOPASSWD: ALL
  24. Dropbox•  We did all that.•  We’re still paying the taxes for doing that.•  But there is a light at the end of the tunnel…
  26. So what does that mean?• Make hosts role agnostic• Do not require invasive changes• Simple interfaces
  27. Making hosts role agnostic• Positional• Serial Numbers• Anything that doesnt change
  28. The Dropbox Plan• Positional names• Custom Machine Database• External Node Classifier• Transitioning Puppet configs• Naming service(s) for convenient names
  29. Service/Machine Management Database• Universal Source of Truth• Manage roles / attributes• Generated configs - Gmond, Nagios, etc
  30. What exactly is the ENC• External Node Classifier• Inject variables (and other) from externalprocess• YAML Output
  31. Part 2: External Node Encoders
  32. Sudoers++• Move from monolithic to modular• Includes! (Weird caveats)• Just use ALL for Host_Lists
  33. Sudoers at Dropbox
  34. Part 3: Helper Functions
  35. Sudoers with tags
  36. Sudoers with tags
  37. Provisioning• Preload MDB, DNS, DHCPD, etc. - Set it and forget it• Have spares ready for any roles• Assigning a role is one command• No more renames!
  38. Dynamic Naming w/ PowerDNS
  39. Dynamic Naming w/ PowerDNS
  40. Zookeeper• ZKNS included with the Vitess project• ZK is in use at various different companies(YouTube, Twitter, AirBnB)
  41. Q&A FAQ #1: Are you hiring?λ  - Yes! Come talk to us. :)