Too often in the organization of this conference we have heard "but I don't have scalability issues". This talk discusses what scalability issues actually are, and details why we all inevitably have them. Avoiding them, or delaying solutions until they are unavoidable, leads to making many bad "temporary" decisions that cannot be fixed further down the line. I will discuss the methodologies and best practices that are required in order to be scalable, and describe the common mistakes they will temper, and why they should be implemented immediately. Finally, I will briefly touch on how to deal with rectifying the bad decisions that we all inevitably make, no matter how forward-thinking we are.

2. “I don’t have scaling
problems”

3. Scaling is about change
not about quantity

4. Problems don’t occur when
things are normal

5. If things change, you will
have scaling problems

6. Work takes time to do

7. Work takes time to do
Email needs to be read

8. Work takes time to do
Email needs to be read
Code runs on a server

9. M is t
ake
!
“I don’t have scaling
problems”

10. M is t
ake
!
“I don’t have scaling
problems”
Not a mistake we’re making if we’re here?

11. Mistakes will be made
Problems will happen

12. Mistakes will be made
Problems will happen
But there are things we can do
to be prepared

13. #1 Measure Everything

14. How do you know if
something is wrong?

15. How do you know if
something is wrong?
not wrong?

16. # uptime
17:27:18 up 405 days, 2:36, 1 user,
load average: 26.93, 10.46, 6.16
!?!?

17. # uptime
17:27:18 up 405 days, 2:36, 1 user,
load average: 26.93, 10.46, 6.16

19. Read your log files

20. Read your log files
(Exceptions aren’t always exceptional)

21. Measure in production
(hat tip: Coda, “metrics, metrics everywhere”)
That’s the only place where
things are really happening

22. Measure in production
(hat tip: Coda, “metrics, metrics everywhere”)
That’s the only place where
things are really happening
But don’t let your metrics cause
performance problems

23. PING web (192.168.19.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Sometimes you can just tell
things are wrong

24. #2 Infrastructure as code
(and config management)

25. Don’t do this.

26. Chef or Puppet
(or cfengine or bcfg2)
Server config is code

27. Chef or Puppet
(or cfengine or bcfg2)
Server config is code
Revision control

28. Chef or Puppet
(or cfengine or bcfg2)
Server config is code
Revision control
Feature branches

29. Chef or Puppet
(or cfengine or bcfg2)
Server config is code
Revision control
Feature branches
Commenting
and authorship

30. Chef or Puppet
(or cfengine or bcfg2)
Server config is code
Revision control
Feature branches
Commenting
and authorship Centralized
(not in someone’s head)

31. Should I choose Chef or Puppet?

32. Should I choose Chef or Puppet?
Yes
(Seriously, this is non-negotiable.)

33. How do I switch my servers to start
using config management?
My advice:
build new ones, throw the old
ones away.

34. Clean Known state
Build test clusters

35. Clean Known state
Buil
d test clusters
troy
Des

36. Clean Known state
Buil
d test clusters
troy
Des
Build live machines

37. Clean Known state
Buil
d test clusters
troy
D es
Bu
ild
e live machines
U s

38. Clean Known state
Buil
d test clusters
troy
D es
Bu
ild
Use live machines!
tr oy
D es

39. One-button servers
What about your code?

40. #3a Real deployment

41. $ svn up
U www/index.php
U www/payments.php
U www/settings-live.php
U www/settings-dev.php
A www/specials.php
U .
Updated to revision 9703.
Don’t do this.

42. Deployment is more than just putting code in place.

43. Deployment is more than just putting code in place.
reproducible idempotent rollouts

44. Deployment is more than just putting code in place.
reproducible idempotent rollouts
tied to a known build number

45. Deployment is more than just putting code in place.
reproducible idempotent rollouts
tied to a known build number
with separately-versioned known configuration

46. Deployment is more than just putting code in place.
reproducible idempotent rollouts
tied to a known build number
with separately-versioned known configuration
triggered non-manually across any number of servers

47. Deployment is more than just putting code in place.
reproducible idempotent rollouts
tied to a known build number
with separately-versioned known configuration
triggered non-manually across any number of servers
with full dependency management

48. Deployment is more than just putting code in place.
reproducible idempotent rollouts
tied to a known build number
with separately-versioned known configuration
triggered non-manually across any number of servers
with full dependency management
and automated regression testing.

49. Etsy’s Deployinator
Fabric
OS Packages
Vlad the Deployer
Capistrano
Roll your own

50. #3b Continuous
deployment

51. Holy Grail
trunk = live
tests block commits
feature flags?
dark launches?

52. Cowboy
vs
Perfectionist

53. Fast iteration = fast test results
Ten new tiny features tested
Two accepted
One huge feature tested
... and rejected

54. Failure is comfortable
Consequences immediately visible
Blame out, responsibility in

55. Okay, fine:
Continuous Integration

56. After all that
Things still go wrong

57. #4 Plan for failure

58. Take backups
Test backups

59. Automate servers
Test server crashes

60. Netflix’s Chaos Monkey
And cousins: the Simian Army

61. Server failures predicted and foiled
What about code? New features?

62. #5 Future Compatibility

63. ALTER TABLE `user` ADD COLUMN
`twootr` VARCHAR(16);
CREATE INDEX `twootr_idx` ON
`user` (`twootr`);
Don’t do this.
(on live)

64. “Future compatible” schemas
Normalized tables are performance heavy
“Future compatible” code
Don’t assume any columns?

65. Shiny new Yucky old
Migrate
?
Write Read

66. What about other bad decisions?

67. #6 Wing It

68. spof.yola.com
- Django
- MySQL
Scheduled for reboot

69. spof.yola.com
- Django
- MySQL
Slave Replication
- MySQL

70. spof.yola.com
- Django
- MySQL
- MySQL

71. spof.yola.com
- Django - Django
Slave replication
- MySQL - MySQL

72. spof.yola.com
LB
- Django - Django
Slave replication
- MySQL - MySQL

73. spof.yola.com
LB Drop DNS TTL
- Django - Django
Slave replication
- MySQL - MySQL

74. spof.yola.com
LB
- Django - Django
Slave replication
- MySQL - MySQL

76. k ay
’s o
t i t
B u

77. Jonathan Hitchcock
@vhata
github.com/vhata