Upcoming SlideShare
Clearly, I Have Made Some Bad Decisions
- “I don’t have scaling problems”
- Scaling is about change not about quantity
- Problems don’t occur when things are normal
- If things change, you will have scaling problems
- Work takes time to do
- Work takes time to do Email needs to be read
- Work takes time to do Email needs to be readCode runs on a server
- M is t ake !“I don’t have scaling problems”
- M is t ake !“I don’t have scaling problems”Not a mistake we’re making if we’re here?
- Mistakes will be made Problems will happen
- Mistakes will be made Problems will happenBut there are things we can do to be prepared
- #1 Measure Everything
- How do you know ifsomething is wrong?
- How do you know ifsomething is wrong? not wrong?
- # uptime 17:27:18 up 405 days, 2:36, 1 user,load average: 26.93, 10.46, 6.16 !?!?
- # uptime 17:27:18 up 405 days, 2:36, 1 user,load average: 26.93, 10.46, 6.16
- Read your log files
- Read your log files(Exceptions aren’t always exceptional)
- Measure in production(hat tip: Coda, “metrics, metrics everywhere”) That’s the only place where things are really happening
- Measure in production(hat tip: Coda, “metrics, metrics everywhere”) That’s the only place where things are really happening But don’t let your metrics cause performance problems
- PING web (192.168.19.1): 56 data bytesRequest timeout for icmp_seq 0Request timeout for icmp_seq 1Request timeout for icmp_seq 2Request timeout for icmp_seq 3 Sometimes you can just tell things are wrong
- #2 Infrastructure as code (and config management)
- Don’t do this.
- Chef or Puppet (or cfengine or bcfg2)Server config is code
- Chef or Puppet (or cfengine or bcfg2)Server config is code Revision control
- Chef or Puppet (or cfengine or bcfg2)Server config is code Revision control Feature branches
- Chef or Puppet (or cfengine or bcfg2)Server config is code Revision control Feature branches Commenting and authorship
- Chef or Puppet (or cfengine or bcfg2)Server config is code Revision control Feature branches Commenting and authorship Centralized (not in someone’s head)
- Should I choose Chef or Puppet?
- Should I choose Chef or Puppet? Yes (Seriously, this is non-negotiable.)
- How do I switch my servers to start using config management? My advice: build new ones, throw the old ones away.
- Clean Known stateBuild test clusters
- Clean Known state Buil d test clusters troyDes
- Clean Known state Buil d test clusters troyDes Build live machines
- Clean Known state Buil d test clusters troyD es Bu ild e live machines U s
- Clean Known state Buil d test clusters troyD es Bu ild Use live machines! tr oy D es
- One-button servers What about your code?
- #3a Real deployment
- $ svn upU www/index.phpU www/payments.phpU www/settings-live.phpU www/settings-dev.phpA www/specials.php U .Updated to revision 9703. Don’t do this.
- Deployment is more than just putting code in place.
- Deployment is more than just putting code in place. reproducible idempotent rollouts
- Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number
- Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configuration
- Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configurationtriggered non-manually across any number of servers
- Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configurationtriggered non-manually across any number of servers with full dependency management
- Deployment is more than just putting code in place. reproducible idempotent rollouts tied to a known build number with separately-versioned known configurationtriggered non-manually across any number of servers with full dependency management and automated regression testing.
- Etsy’s Deployinator FabricOS Packages Vlad the DeployerCapistrano Roll your own
- #3b Continuous deployment
- Holy Grailtrunk = live tests block commits feature flags? dark launches?
- Cowboy vsPerfectionist
- Fast iteration = fast test results Ten new tiny features tested Two accepted One huge feature tested ... and rejected
- Failure is comfortableConsequences immediately visible Blame out, responsibility in
- Okay, fine:Continuous Integration
- After all thatThings still go wrong
- #4 Plan for failure
- Take backupsTest backups
- Automate serversTest server crashes
- Netflix’s Chaos Monkey And cousins: the Simian Army
- Server failures predicted and foiledWhat about code? New features?
- #5 Future Compatibility
- ALTER TABLE `user` ADD COLUMN `twootr` VARCHAR(16);CREATE INDEX `twootr_idx` ON `user` (`twootr`); Don’t do this. (on live)
- “Future compatible” schemas Normalized tables are performance heavy “Future compatible” code Don’t assume any columns?
- Shiny new Yucky old Migrate ? Write Read
- What about other bad decisions?
- #6 Wing It
- spof.yola.com - Django - MySQL Scheduled for reboot
- spof.yola.com - Django - MySQL Slave Replication - MySQL
- spof.yola.com - Django - MySQL - MySQL
- spof.yola.com - Django - Django Slave replication - MySQL - MySQL
- spof.yola.com LB - Django - Django Slave replication - MySQL - MySQL
- spof.yola.com LB Drop DNS TTL - Django - Django Slave replication - MySQL - MySQL
- spof.yola.com LB - Django - Django Slave replication - MySQL - MySQL
- k ay ’s o t i tB u
- Jonathan Hitchcock @vhata github.com/vhata
Login to see the comments