Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scalability and consistency


Published on

Presentation to Box TechOps team in 2011

  • Be the first to comment

  • Be the first to like this

Scalability and consistency

  1. 1. Scalability and Consistency The Nitty Gritty
  2. 2. Overview • The only sane way to scale is to ensure consistency and automation • Build things in a way in which you don't care about individual servers (don't be afraid to rekick a server) • You should never need to login to an individual box once it's in production 
  3. 3. How to Accomplish This • Use an automated provisioning tool (such as kickstart/cobbler) •  Use configuration management (such as puppet) •  Test, test, test
  4. 4. Users and Authentication • Logging in as root directly is bad • Keys used for automated processes should be separate from user keys • Access to a service on a box and SSH access to the box should not be linked
  5. 5. Packaging • Use a local copy of distribution repositories • Do not use external repositories directly • Use RPMs (do not use PECL, PEAR, CPAN, gem, extracting tarballs, compiling from source, etc.) External repositories are moving targets. Depending on when a server is provisioned, it could end up with different package versions. Non-RPM packaging methods don't have upgrade, uninstall, and versioning, which is why they should be avoided.
  6. 6. Building RPMs • It's perfectly fine to build a custom RPM when one is not readily available • Don't rebuild just to change file locations (such as putting stuff under /box) • Build in a "sterile" environment • Make sure RPM spec files list all of their direct build and runtime dependencies
  7. 7. General Puppet Tips • Write modules/classes in a way that they can be reused • Be specific...don't assume something will happen a certain way if you don't define it • Follow the Puppet Style Guide • Don't hard-code anything (hostnames in particular) • Use dependencies
  8. 8. Puppet Tips - Users Use this: group { "bob":   ensure => present,   gid      => 503, } user { "bob":   ensure => present,   gid       => 'bob',   require => Group["bob"], } Instead of: user { "bob":   ensure => present, } • Manage the group and the user • Specify the UID and GID to ensure consistency across servers
  9. 9. Puppet Tips - Packages Use this: package { "php-pecl-memcache":   ensure => installed, } Instead of: exec { "install_pecl_memcache":   command => 'pecl install memcache', } • Use a single package manager (RPM) as much as possible • Specify versions when it makes sense
  10. 10. Puppet Tips - Dependencies Use this: package { "nslcd":   ensure => installed, } service { "nslcd":   ensure => running,   require => Package["nslcd"],   subscribe => Package["nslcd"], } Instead of: package { "nslcd":   ensure => installed, } service { "nslcd":   ensure => running, } • Dependencies ensure that puppet performs operations in a particular order
  11. 11. Puppet Tips - Testing • Test your changes on a non-production server • For larger changes in puppet, it's not completely tested until you can build a server from the ground up and puppet sets everything up properly without manual intervention • If you get errors on the first puppet run and none on the second, add more dependencies
  12. 12. Puppet Tips - Roles • Instead of defining everything on a per-node basis, define roles and put servers into them. It's better to define 500 roles and have servers fit neatly into them than to have 1000 node definitions, each of them differing by only a little bit • All roles should be defined in the "nodetype" module and inherit from nodetype::generic • This will allow us to eventually use an ENC (external node classifier)