Functional Hostnames and Why they are Bad

  • 7,874 views
Uploaded on

"Functional Hostnames and Why they are Bad" by Andrew Fong and Gary Josack of Dropbox at Puppet Camp SF 2013. Find a Puppet Camp near you and learn more about configuration management: …

"Functional Hostnames and Why they are Bad" by Andrew Fong and Gary Josack of Dropbox at Puppet Camp SF 2013. Find a Puppet Camp near you and learn more about configuration management: puppetlabs.com/community/puppet-camp/

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
7,874
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
47
Comments
0
Likes
16

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Puppet Camp April 9th 2013
  • 2. What’s in a name?Andrew Fong and Gary Josack fong@dropbox.com gary@dropbox.com
  • 3. About Dropbox•  Thousands of instances/servers•  Mostly Python Stack•  EC2 and Large Self Datacenters•  Over a billion file syncs per day•  Thousands of MySQL Shards•  4 SREs and 1 DBA
  • 4. A story of a startup... Hostapuppet.com
  • 5. Chapter One•  1 or 2 teams•  Couple of hosts•  Webserver and a database•  Maybe one ops guy
  • 6. What Ops People Like•  Simplicity•  Repeatability•  Assurances that things remain consistent•  Puppet / Configuration management
  • 7. Config Managementnode www1.example.com { include common include apache include squid }node db1.example.com { include common include mysql }
  • 8. My First Puppet ConfigNode ‘mickey.hostapuppet.com’ { include common include webserver include sudoers}Node ‘donald.hostapuppet.com’{ include common include db include sudoers}
  • 9. Sudoers Module…file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", }…
  • 10. Sudoers File itselfHost_Alias DONALD=donald.hostapuppet.comHost_Alias MICKEY=mickey.hostapuppet.comdb_guy DONALD=(all) NOPASSWD: ALLops_guy MICKEY=(all) NOPASSWD: ALL
  • 11. Chapter Two: A growing service•  A few teams•  2 or 3 services•  multiple types of hosts –  Web –  API –  DB
  • 12. Hostnames•  sjc-web[1-N]•  sjc-db[1-N]•  sjc-api[1-N]
  • 13. Host Regex$hosttype = inline_template(<%=hostname.sub(/w+-([a-z]+)d*/){$1} %>’)
  • 14. Hosttypes $hosttype = inline_template(<%= hostname.sub(/w+-([a-z]+)d*/){$1} %>’)if $hosttype == ‘web’ { if $hosttype == ‘api’ { include sudoers include sudoers include web include api }}If $hosttype == ‘db’ { include sudoers include db}
  • 15. Back to sudoers…file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", }…
  • 16. Sudoers File itselfHost_Alias WEB=sjc-web*Host_Alias DB=sjc-db*Host_Alias API=sjc-api*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API=(all) NOPASSWD: ALL
  • 17. Hypergrowth Users (millions)120100 80 60 40 20 0 0 2 3 5 Users (millions)
  • 18. Chapter 3: An Expanding Infrastructure•  Lots of new hires!•  A bunch more developers•  Some PMs•  Some Designers
  • 19. All Kinds Of Problems…•  Boxes of same hardware class running different services•  Boxes serving more than one role (remember sudoers?)•  Deploying or moving hosts quickly
  • 20. Renaming a host•  Update dns•  Update dhcpd.conf•  Push both•  Update puppet configs•  Update code
  • 21. OMG I JUST RENAME HOSTS!
  • 22. Sudoers File From Chapter Two…Host_Alias WEB= sjc-web*Host_Alias API=sjc-api*Host_Alias DB=sjc-db*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API(all) NOPASSWD: ALL
  • 23. Sudoers File in Chapter 3Host_Alias WEB= sjc-web*Host_Alias API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553Host_Alias DB=sjc-db*database_guy DB=(all) NOPASSWD: ALLops_guy WEB=(all) NOPASSWD: ALLapi_team API(all) NOPASSWD: ALL
  • 24. Dropbox•  We did all that.•  We’re still paying the taxes for doing that.•  But there is a light at the end of the tunnel…
  • 25. ABSTRACT THE SERVICEFROM THE HOST!
  • 26. So what does that mean?• Make hosts role agnostic• Do not require invasive changes• Simple interfaces
  • 27. Making hosts role agnostic• Positional• Serial Numbers• Anything that doesnt change
  • 28. The Dropbox Plan• Positional names• Custom Machine Database• External Node Classifier• Transitioning Puppet configs• Naming service(s) for convenient names
  • 29. Service/Machine Management Database• Universal Source of Truth• Manage roles / attributes• Generated configs - Gmond, Nagios, etc
  • 30. What exactly is the ENC• External Node Classifier• Inject variables (and other) from externalprocess• YAML Output
  • 31. Part 2: External Node Encoders
  • 32. Sudoers++• Move from monolithic to modular• Includes! (Weird caveats)• Just use ALL for Host_Lists
  • 33. Sudoers at Dropbox
  • 34. Part 3: Helper Functions
  • 35. Sudoers with tags
  • 36. Sudoers with tags
  • 37. Provisioning• Preload MDB, DNS, DHCPD, etc. - Set it and forget it• Have spares ready for any roles• Assigning a role is one command• No more renames!
  • 38. Dynamic Naming w/ PowerDNS
  • 39. Dynamic Naming w/ PowerDNS
  • 40. Zookeeper• ZKNS included with the Vitess project• ZK is in use at various different companies(YouTube, Twitter, AirBnB)
  • 41. Q&A FAQ #1: Are you hiring?λ  - Yes! Come talk to us. :)