1. Puppet Camp
April 9th 2013

2. What’s in a name?
Andrew Fong and Gary Josack
fong@dropbox.com
gary@dropbox.com

3. About Dropbox
• Thousands of instances/servers
• Mostly Python Stack
• EC2 and Large Self Datacenters
• Over a billion file syncs per day
• Thousands of MySQL Shards
• 4 SREs and 1 DBA

4. A story of a startup...
Hostapuppet.com

5. Chapter One
• 1 or 2 teams
• Couple of hosts
• Webserver and a database
• Maybe one ops guy

6. What Ops People Like
• Simplicity
• Repeatability
• Assurances that things remain consistent
• Puppet / Configuration management

7. Config Management
node 'www1.example.com' {
include common
include apache
include squid
}
node 'db1.example.com' {
include common
include mysql
}

8. My First Puppet Config
Node ‘mickey.hostapuppet.com’ {
include common
include webserver
include sudoers
}
Node ‘donald.hostapuppet.com’{
include common
include db
include sudoers
}

9. Sudoers Module
…
file { "/etc/sudoers":
owner => root,
group => root,
mode => "440",
source => "puppet:///modules/sudo/sudoers",
}
…

10. Sudoers File itself
Host_Alias DONALD=donald.hostapuppet.com
Host_Alias MICKEY=mickey.hostapuppet.com
db_guy DONALD=(all) NOPASSWD: ALL
ops_guy MICKEY=(all) NOPASSWD: ALL

11. Chapter Two: A growing service
• A few teams
• 2 or 3 services
• multiple types of hosts
– Web
– API
– DB

12. Hostnames
• sjc-web[1-N]
• sjc-db[1-N]
• sjc-api[1-N]

13. Host Regex
$hosttype = inline_template('<%=
hostname.sub(/w+-([a-z]+)d*/){$1} %>’)

14. Hosttypes
$hosttype = inline_template('<%= hostname.sub(/w+-([a-z]+)d*/){$1} %>’)
if $hosttype == ‘web’ { if $hosttype == ‘api’ {
include sudoers include sudoers
include web include api
}
}
If $hosttype == ‘db’ {
include sudoers
include db
}

15. Back to sudoers
…
file { "/etc/sudoers":
owner => root,
group => root,
mode => "440",
source => "puppet:///modules/sudo/sudoers",
}
…

16. Sudoers File itself
Host_Alias WEB=sjc-web*
Host_Alias DB=sjc-db*
Host_Alias API=sjc-api*
database_guy DB=(all) NOPASSWD: ALL
ops_guy WEB=(all) NOPASSWD: ALL
api_team API=(all) NOPASSWD: ALL

17. Hypergrowth
Users (millions)
120
100
80
60
40
20
0
0 2 3 5
Users (millions)

18. Chapter 3: An Expanding
Infrastructure
• Lots of new hires!
• A bunch more developers
• Some PMs
• Some Designers

19. All Kinds Of Problems…
• Boxes of same hardware class running
different services
• Boxes serving more than one role
(remember sudoers?)
• Deploying or moving hosts quickly

20. Renaming a host
• Update dns
• Update dhcpd.conf
• Push both
• Update puppet configs
• Update code

21. OMG I JUST RENAME HOSTS!

22. Sudoers File From Chapter Two…
Host_Alias WEB= sjc-web*
Host_Alias API=sjc-api*
Host_Alias DB=sjc-db*
database_guy DB=(all) NOPASSWD: ALL
ops_guy WEB=(all) NOPASSWD: ALL
api_team API(all) NOPASSWD: ALL

23. Sudoers File in Chapter 3
Host_Alias WEB= sjc-web*
Host_Alias API=sjc-api*,sjc-web550,sjc-
web551,sjc-web552,sjc-web553
Host_Alias DB=sjc-db*
database_guy DB=(all) NOPASSWD: ALL
ops_guy WEB=(all) NOPASSWD: ALL
api_team API(all) NOPASSWD: ALL

24. Dropbox
• We did all that.
• We’re still paying the taxes for doing
that.
• But there is a light at the end of the
tunnel…

25. ABSTRACT THE
SERVICE
FROM THE HOST!

26. So what does that mean?
• Make hosts role agnostic
• Do not require invasive changes
• Simple interfaces

27. Making hosts role agnostic
• Positional
• Serial Numbers
• Anything that doesn't change

28. The Dropbox Plan
• Positional names
• Custom Machine Database
• External Node Classifier
• Transitioning Puppet configs
• Naming service(s) for convenient names

29. Service/Machine Management
Database
• Universal Source of Truth
• Manage roles / attributes
• Generated configs
- Gmond, Nagios, etc

30. What exactly is the ENC
• External Node Classifier
• Inject variables (and other) from external
process
• YAML Output

31. Part 2: External Node Encoders

32. Sudoers++
• Move from monolithic to modular
• Includes! (Weird caveats)
• Just use ALL for Host_Lists

33. Sudoers at Dropbox

34. Part 3: Helper Functions

35. Sudoers with tags

36. Sudoers with tags

37. Provisioning
• Preload MDB, DNS, DHCPD, etc.
- Set it and forget it
• Have spares ready for any roles
• Assigning a role is one command
• No more renames!

38. Dynamic Naming w/ PowerDNS

39. Dynamic Naming w/ PowerDNS

40. Zookeeper
• ZKNS included with the Vitess project
• ZK is in use at various different companies
(YouTube, Twitter, AirBnB)

41. Q&A
FAQ #1: Are you hiring?
λ
- Yes! Come talk to us. :)