Managing Massive Datacenters           Luke Kanies       Founder and CEO, Puppet Labs           luke@puppetlabs.com
QuestionAuthority
Puppet: Open Source   Systems Management • You explain to Puppet how you want your   infrastructure to look, and it makes ...
ManagingDatacenters
Thereʼs no pattern• Enterprise vs. growth company  – E.g., Nokia vs. Zynga• Technology company vs. startup  – Twitter toda...
Web ops arecooler, corporate    is harder
Web operationslooks a lot like     HPC http://www.cfdesign.com/image/legacy/news/2009/tricone-bit-HPC.jpg?w=350&h=288&as=1
Lots of apps != lots of users
It can feel likeSilicon Valley vs.    The World
Over time, web ops   companiesbecome corporate
How aredatacenters managed?
Itʼs worse than you        think
Worse than that
No no - not you   But probably the person next to you
Technology still    canʼt solvepolitical problems
$25m applicationwith 100s of trivial   root exploits
Deploymentcosting more than  development
Six months with noupgrades because  they forgot how
50-96% of outagescaused by human      error
The problem actually     is complex                                     Monitor                                Change Mana...
Disaster is constant
Solution: Enterprise- grade management      solution
Built for                         protection, not                          productivityhttp://www.deviantart.com/download/...
Claim to solveevery problem
Reality: Shellscripts and cron      jobs
Infrastructure is more coral reef,than Eiffel Tower
Simplicity scales,   complexity    crumbles
Scalable complexity requires simplicityhttp://freeassociationdesign.files.wordpress.com/2010/04/neuron-map1.jpg
Itʼs even more so     the cloud
Simplicity provides frictionless scale
High cost, no    agility, bigresources kill you
How did we get    here?
Jengaadministrationhttp://www.thecolorcurve.com/blog/wp-content/uploads/2011/02/jenga_negativeskew.jpg
Theory: Deploy, use,   decommision
Reality: Deploy, back    away slowly
People fearchange, ratherthan enable it http://www.freefoto.com/images/04/28/04_28_57---Pile-of-Coins_web.jpg
Poor teamwork
Development produces theworldʼs best app
Ops has no ideahow to deploy it
Auditors wonʼt sign off on it
No oneʼs fault, everyoneʼs   problem
Consultants arehappy to “help”
Devops : Ops ::Agile : Development
Cultural change takes forever
The best have   learned
It canʼt go on like this                 foreverImage source: http://sloblogs.thetribunenews.com/slovault/files/2008/08/pa...
They build their own
=> Lots ofduplicated effort     And thrown away code
Theyʼre still the  exception
Normal:Purchased, hand-rolled, and broken
OperationalAntipatterns
Direct fromkeyboard toproduction
Alerts delivered   via email
Expecting tools to protect you from  incompetence
Golden ImagesImage from http://www.flickr.com/photos/fungep/2516767121/sizes/l_
What you should  do instead
Step 1 SSHScripts
Make everyone  responsible foroperational success
See the change, not the state
Think services, not      hosts
Use the stupidest  tool that can possibly work
Find wherepeopleʼs time goes
Solve the simple problems first
Focus on value/     cost
Only monitorthings you care     about
Continuousintegration, even in the infrastructure
Keep humans at  the console
Automate tasks, not decisions
A little more about       Puppet
Constant Configuration                                                Node                     1   Facts                   ...
Simple Configuration     Language
System Portabilitywith the ResourceAbstraction Layer
Weʼre building Iron Man, not           robots
Usability first,capability second
Separation ofspecification and   execution
Massive amounts    of data
Portability enables DRY Operations
The Future of the Puppet       Platform                                                               APPLICATION         ...
Summary
The state ofoperations is still   really bad
The cloud is making it worse, or better,    or something
Trying to solve ityourself makes youa software company
But you mighthave no choice
So build something   simple and be  ready to toss it
Or just use Puppet :)
Questions?
Upcoming SlideShare
Loading in …5
×

Managing massive datacenters

1,483 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,483
On SlideShare
0
From Embeds
0
Number of Embeds
67
Actions
Shares
0
Downloads
26
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Humans are very bad at typing the same thing 1000 times\n
  • \n
  • \n
  • \n
  • \n
  • They have to, to charge what they do\nThe result is massive complexity and poor integration\n
  • \n
  • It accretes over time\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Sysadmins hate menial work even more than management\nToo long spent on this slide\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • It doesn’t matter how “important” the function is - if it’s taking time, get rid of it\n
  • \n
  • \n
  • \n
  • JPMorgan has 1 billion lines of code under continuous integration\n
  • \n
  • \n
  • \n
  • \n
  • This is shareable, releasable code.\nClasses are analogous with tags\n
  • \n
  • Devops\n
  • \n
  • This allows more control points as needs grow\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Managing massive datacenters

    1. 1. Managing Massive Datacenters Luke Kanies Founder and CEO, Puppet Labs luke@puppetlabs.com
    2. 2. QuestionAuthority
    3. 3. Puppet: Open Source Systems Management • You explain to Puppet how you want your infrastructure to look, and it makes it so • As you change that explanation over time, Puppet updates all machines to matchExplain Enforce Iterate
    4. 4. ManagingDatacenters
    5. 5. Thereʼs no pattern• Enterprise vs. growth company – E.g., Nokia vs. Zynga• Technology company vs. startup – Twitter today vs. Twitter 2 years ago• Yours vs. Mine – Rackspace vs. Amazon vs. Google• Big vs. really really big – Wal Mart vs. JPMorgan• Corporate vs. Customer-facing
    6. 6. Web ops arecooler, corporate is harder
    7. 7. Web operationslooks a lot like HPC http://www.cfdesign.com/image/legacy/news/2009/tricone-bit-HPC.jpg?w=350&h=288&as=1
    8. 8. Lots of apps != lots of users
    9. 9. It can feel likeSilicon Valley vs. The World
    10. 10. Over time, web ops companiesbecome corporate
    11. 11. How aredatacenters managed?
    12. 12. Itʼs worse than you think
    13. 13. Worse than that
    14. 14. No no - not you But probably the person next to you
    15. 15. Technology still canʼt solvepolitical problems
    16. 16. $25m applicationwith 100s of trivial root exploits
    17. 17. Deploymentcosting more than development
    18. 18. Six months with noupgrades because they forgot how
    19. 19. 50-96% of outagescaused by human error
    20. 20. The problem actually is complex Monitor Change Management Application Service Deployment Orchestration Middleware Deployment & Configuration Release Audit System Configuration Cloud or VM Image OS Install Launch Configuration Repository
    21. 21. Disaster is constant
    22. 22. Solution: Enterprise- grade management solution
    23. 23. Built for protection, not productivityhttp://www.deviantart.com/download/97345411/Straight_jacket_Joker_by_MasterDrawer.jpg
    24. 24. Claim to solveevery problem
    25. 25. Reality: Shellscripts and cron jobs
    26. 26. Infrastructure is more coral reef,than Eiffel Tower
    27. 27. Simplicity scales, complexity crumbles
    28. 28. Scalable complexity requires simplicityhttp://freeassociationdesign.files.wordpress.com/2010/04/neuron-map1.jpg
    29. 29. Itʼs even more so the cloud
    30. 30. Simplicity provides frictionless scale
    31. 31. High cost, no agility, bigresources kill you
    32. 32. How did we get here?
    33. 33. Jengaadministrationhttp://www.thecolorcurve.com/blog/wp-content/uploads/2011/02/jenga_negativeskew.jpg
    34. 34. Theory: Deploy, use, decommision
    35. 35. Reality: Deploy, back away slowly
    36. 36. People fearchange, ratherthan enable it http://www.freefoto.com/images/04/28/04_28_57---Pile-of-Coins_web.jpg
    37. 37. Poor teamwork
    38. 38. Development produces theworldʼs best app
    39. 39. Ops has no ideahow to deploy it
    40. 40. Auditors wonʼt sign off on it
    41. 41. No oneʼs fault, everyoneʼs problem
    42. 42. Consultants arehappy to “help”
    43. 43. Devops : Ops ::Agile : Development
    44. 44. Cultural change takes forever
    45. 45. The best have learned
    46. 46. It canʼt go on like this foreverImage source: http://sloblogs.thetribunenews.com/slovault/files/2008/08/pacific-telephone.jpg
    47. 47. They build their own
    48. 48. => Lots ofduplicated effort And thrown away code
    49. 49. Theyʼre still the exception
    50. 50. Normal:Purchased, hand-rolled, and broken
    51. 51. OperationalAntipatterns
    52. 52. Direct fromkeyboard toproduction
    53. 53. Alerts delivered via email
    54. 54. Expecting tools to protect you from incompetence
    55. 55. Golden ImagesImage from http://www.flickr.com/photos/fungep/2516767121/sizes/l_
    56. 56. What you should do instead
    57. 57. Step 1 SSHScripts
    58. 58. Make everyone responsible foroperational success
    59. 59. See the change, not the state
    60. 60. Think services, not hosts
    61. 61. Use the stupidest tool that can possibly work
    62. 62. Find wherepeopleʼs time goes
    63. 63. Solve the simple problems first
    64. 64. Focus on value/ cost
    65. 65. Only monitorthings you care about
    66. 66. Continuousintegration, even in the infrastructure
    67. 67. Keep humans at the console
    68. 68. Automate tasks, not decisions
    69. 69. A little more about Puppet
    70. 70. Constant Configuration Node 1 Facts !"#$%&#$(#%($ %&)*+,-.#$+/+$ +0&1/$-/(#,2$/&$/"#$ 3144#/$5+(/#)6 SSL secure 2 Catalog 3144#/$1(#($/"#$7+8/($/& encryption 8&*4-,#$+$9+/+,&:$/"+/ on all data (4#8-2-#($"&;$/"#$%&# transport ("&1,$0#$8&%2-:1)#6Report 3!"#$%&#$)#4&)/($0+8=$/&$3144#/$-%-8+/-%:$/"#$8&%2-:1)+/-&%$-($8&*4,#/#>$;"-8"$-($?-(-0,#$-%$/"#$ Puppet3144#/$@+("0&+)6 Master 4 Report Collector A3144#/$&)$B)$4+)/<$/&&,C Report !"##$%&()#$*(+!,( 8+%$+,(&$(#%$+/+$ /&$/"-)$4+)/<$/&&,(6
    71. 71. Simple Configuration Language
    72. 72. System Portabilitywith the ResourceAbstraction Layer
    73. 73. Weʼre building Iron Man, not robots
    74. 74. Usability first,capability second
    75. 75. Separation ofspecification and execution
    76. 76. Massive amounts of data
    77. 77. Portability enables DRY Operations
    78. 78. The Future of the Puppet Platform APPLICATION OWNER Ela sti cS A ca SL CxO lin g ics SYS lyt ADMIN Tim na Pr Real-time Status Auditing and A e to g Roll-out Monitorin ov Reportin isioning Value Co n fi g r a ti o n Ma u n a ge m ent g Co m p li a n ce IT n Ser ti o v i ce De l ega ! ! "#$$%& ! ! $ (&)**+&(, ! ! -##$%
    79. 79. Summary
    80. 80. The state ofoperations is still really bad
    81. 81. The cloud is making it worse, or better, or something
    82. 82. Trying to solve ityourself makes youa software company
    83. 83. But you mighthave no choice
    84. 84. So build something simple and be ready to toss it
    85. 85. Or just use Puppet :)
    86. 86. Questions?

    ×