Presentation to Box TechOps team in 2011

1. Scalability and Consistency
The Nitty Gritty

2. Overview
• The only sane way to scale is to ensure consistency and
automation
• Build things in a way in which you don't care about
individual servers (don't be afraid to rekick a server)
• You should never need to login to an individual box once
it's in production

3. How to Accomplish This
• Use an automated provisioning tool (such as
kickstart/cobbler)
• Use configuration management (such as puppet)
• Test, test, test

4. Users and Authentication
• Logging in as root directly is bad
• Keys used for automated processes should be separate
from user keys
• Access to a service on a box and SSH access to the box
should not be linked

5. Packaging
• Use a local copy of distribution repositories
• Do not use external repositories directly
• Use RPMs (do not use PECL, PEAR, CPAN, gem,
extracting tarballs, compiling from source, etc.)
External repositories are moving targets. Depending on
when a server is provisioned, it could end up with different
package versions.
Non-RPM packaging methods don't have upgrade, uninstall,
and versioning, which is why they should be avoided.

6. Building RPMs
• It's perfectly fine to build a custom RPM when one is not
readily available
• Don't rebuild just to change file locations (such as
putting stuff under /box)
• Build in a "sterile" environment
• Make sure RPM spec files list all of their direct build and
runtime dependencies

7. General Puppet Tips
• Write modules/classes in a way that they can be reused
• Be specific...don't assume something will happen a certain
way if you don't define it
• Follow the Puppet Style Guide
• Don't hard-code anything (hostnames in particular)
• Use dependencies
http://docs.puppetlabs.com/guides/style_guide.html

8. Puppet Tips - Users
Use this:
group { "bob":
ensure => present,
gid => 503,
}
user { "bob":
ensure => present,
gid => 'bob',
require => Group["bob"],
}
Instead of:
user { "bob":
ensure => present,
}
• Manage the group and the user
• Specify the UID and GID to ensure consistency across
servers

9. Puppet Tips - Packages
Use this:
package { "php-pecl-memcache":
ensure => installed,
}
Instead of:
exec { "install_pecl_memcache":
command => 'pecl install memcache',
}
• Use a single package manager (RPM) as much as possible
• Specify versions when it makes sense

10. Puppet Tips - Dependencies
Use this:
package { "nslcd":
ensure => installed,
}
service { "nslcd":
ensure => running,
require => Package["nslcd"],
subscribe => Package["nslcd"],
}
Instead of:
package { "nslcd":
ensure => installed,
}
service { "nslcd":
ensure => running,
}
• Dependencies ensure that puppet performs operations in
a particular order

11. Puppet Tips - Testing
• Test your changes on a non-production server
• For larger changes in puppet, it's not completely tested
until you can build a server from the ground up and
puppet sets everything up properly without manual
intervention
• If you get errors on the first puppet run and none on the
second, add more dependencies

12. Puppet Tips - Roles
• Instead of defining everything on a per-node basis,
define roles and put servers into them. It's better to
define 500 roles and have servers fit neatly into them
than to have 1000 node definitions, each of them
differing by only a little bit
• All roles should be defined in the "nodetype" module and
inherit from nodetype::generic
• This will allow us to eventually use an ENC (external node
classifier)