1. Managing Massive Datacenters
Luke Kanies
Founder and CEO, Puppet Labs
luke@puppetlabs.com

2. Question
Authority

3. Puppet: Open Source
Systems Management
• You explain to Puppet how you want your
infrastructure to look, and it makes it so
• As you change that explanation over time,
Puppet updates all machines to match
Explain Enforce Iterate

4. Managing
Datacenters

5. Thereʼs no pattern
• Enterprise vs. growth company
– E.g., Nokia vs. Zynga
• Technology company vs. startup
– Twitter today vs. Twitter 2 years ago
• Yours vs. Mine
– Rackspace vs. Amazon vs. Google
• Big vs. really really big
– Wal Mart vs. JPMorgan
• Corporate vs. Customer-facing

6. Web ops are
cooler, corporate
is harder

7. Web operations
looks a lot like
HPC
http://www.cfdesign.com/image/legacy/news/2009/tricone-bit-HPC.jpg?w=350&h=288&as=1

8. Lots of apps !=
lots of users

9. It can feel like
Silicon Valley vs.
The World

10. Over time, web ops
companies
become corporate

11. How are
datacenters
managed?

12. Itʼs worse than you
think

13. Worse than that

14. No no - not you
But probably the person next to you

15. Technology still
canʼt solve
political problems

16. $25m application
with 100s of trivial
root exploits

17. Deployment
costing more than
development

18. Six months with no
upgrades because
they forgot how

19. 50-96% of outages
caused by human
error

20. The problem actually
is complex
Monitor
Change Management
Application Service Deployment
Orchestration
Middleware Deployment & Configuration
Release
Audit
System Configuration
Cloud or VM Image
OS Install
Launch
Configuration Repository

21. Disaster is constant

22. Solution: Enterprise-
grade management
solution

23. Built for
protection, not
productivity
http://www.deviantart.com/download/97345411/Straight_jacket_Joker_by_MasterDrawer.jpg

24. Claim to solve
every problem

25. Reality: Shell
scripts and cron
jobs

26. Infrastructure is
more coral reef,
than Eiffel Tower

27. Simplicity scales,
complexity
crumbles

28. Scalable complexity
requires simplicity
http://freeassociationdesign.files.wordpress.com/2010/04/neuron-map1.jpg

29. Itʼs even more so
the cloud

30. Simplicity provides
frictionless scale

31. High cost, no
agility, big
resources kill you

32. How did we get
here?

33. Jenga
administration
http://www.thecolorcurve.com/blog/wp-content/uploads/2011/02/jenga_negativeskew.jpg

34. Theory: Deploy, use,
decommision

35. Reality: Deploy, back
away slowly

36. People fear
change, rather
than enable it
http://www.freefoto.com/images/04/28/04_28_57---Pile-of-Coins_web.jpg

37. Poor teamwork

38. Development
produces the
worldʼs best app

39. Ops has no idea
how to deploy it

40. Auditors wonʼt
sign off on it

41. No oneʼs fault,
everyoneʼs
problem

42. Consultants are
happy to “help”

43. Devops : Ops ::
Agile : Development

44. Cultural change
takes forever

45. The best have
learned

46. It canʼt go on like this
forever
Image source: http://sloblogs.thetribunenews.com/slovault/files/2008/08/pacific-telephone.jpg

47. They build their own

48. => Lots of
duplicated effort
And thrown away code

49. Theyʼre still the
exception

50. Normal:
Purchased, hand-
rolled, and broken

51. Operational
Antipatterns

52. Direct from
keyboard to
production

53. Alerts delivered
via email

54. Expecting tools to
protect you from
incompetence

55. Golden Images
Image from http://www.flickr.com/photos/fungep/2516767121/sizes/l_

56. What you should
do instead

57. Step 1
SSH
Scripts

58. Make everyone
responsible for
operational success

59. See the change,
not the state

60. Think services, not
hosts

61. Use the stupidest
tool that can
possibly work

62. Find where
peopleʼs time goes

63. Solve the simple
problems first

64. Focus on value/
cost

65. Only monitor
things you care
about

66. Continuous
integration, even in
the infrastructure

67. Keep humans at
the console

68. Automate tasks,
not decisions

69. A little more about
Puppet

70. Constant Configuration
Node
1 Facts
!"#$%&'#$(#%'($
%&)*+,-.#'$'+/+$
+0&1/$-/(#,2$/&$/"#$
3144#/$5+(/#)6
SSL secure 2 Catalog
3144#/$1(#($/"#$7+8/($/&
encryption 8&*4-,#$+$9+/+,&:$/"+/
on all data (4#8-2-#($"&;$/"#$%&'#
transport ("&1,'$0#$8&%2-:1)#'6
Report 3
!"#$%&'#$
)#4&)/($0+8=$
/&$3144#/$
-%'-8+/-%:$/"#$
8&%2-:1)+/-&%$-($
8&*4,#/#>$;"-8"$
-($?-(-0,#$-%$/"#$ Puppet
3144#/$@+("0&+)'6 Master 4 Report Collector
A3144#/$&)$B)'$4+)/<$/&&,C
Report
!"##$%&'()#$*(+!,(
8+%$+,(&$(#%'$'+/+$
/&$/"-)'$4+)/<$/&&,(6

71. Simple Configuration
Language

72. System Portability
with the Resource
Abstraction Layer

73. Weʼre building Iron Man, not
robots

74. Usability first,
capability second

75. Separation of
specification and
execution

76. Massive amounts
of data

77. Portability enables
DRY Operations

78. The Future of the Puppet
Platform
APPLICATION
OWNER
Ela
sti
cS
A
ca
SL
CxO
lin
g
ics SYS
lyt
ADMIN
Tim
na Pr
Real-time Status
Auditing and A
e to
g
Roll-out Monitorin
ov
Reportin
isioning
Value
Co
n fi g r a ti o
n
Ma u
n a ge m ent
g
Co m p li a n ce
IT n
Ser ti o
v i ce De l ega ! ! "#$$%&'
! ! '$ (&)*'*+&(,
! ! -#'#$%

79. Summary

80. The state of
operations is still
really bad

81. The cloud is making
it worse, or better,
or something

82. Trying to solve it
yourself makes you
a software company

83. But you might
have no choice

84. So build something
simple and be
ready to toss it

85. Or just use
Puppet :)

86. Questions?