Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1
3 Years of Puppet at Cisco:
The Secrets to Ou...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
•  Worked at Cisco for 13 years
-  7 years i...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
•  Worked at Cisco for 3 years
-  Software E...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
•  Overview
•  Experience
•  Plumbing
•  His...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
•  Enterprise Social Media
-  Create content...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Persistence
•  Traditional tiered web servic...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
•  Virtual appliance delivery model
•  Roles...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
1.  Customer downloads a single OVA from Ci...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
1.  Customer downloads a single OVA from Ci...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
1.  Customer downloads a single OVA from Ci...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
1.  Customer downloads a single OVA from Ci...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
1.  Customer downloads a single OVA from Ci...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
1.  Customer downloads a single OVA from Ci...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
1.  Customer download a single release imag...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
1.  Customer download a single release imag...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
1.  Customer download a single release imag...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
1.  Customer download a single release imag...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
OVF
OS
Bootstrap
Config
Network
Settings
Di...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
•  Package everything as an RPM
•  RPMs dis...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
•  Versioned build artifact containing:
-  ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
Director
OS
Analytics
Store
QUAD OS
JSON St...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Director
OS
Analytics
Store
OS
BSON
Store
O...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
Director
OS
Analytics
Store
OS
BSON
Store
O...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
Phase 1
Mid 2010
•  20GB VM
•  CentOS 5
•  ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Phase 1
Mid 2010
•  20GB VM
•  CentOS 5
•  ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Phase 1
Mid 2010
•  20GB VM
•  CentOS 5
•  ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Phase 1
Mid 2010
•  20GB VM
•  CentOS 5
•  ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Phase 1
Mid 2010
•  20GB VM
•  CentOS 5
•  ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
Phase 1
Mid 2010
•  20GB VM
•  CentOS 5
•  ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
Keep it simple
•  80/20 rule
•  Build only ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
•  Excellent at configuration management
• ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
•  Alex Honor of DTO Solutions and Simplify...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
Upcoming SlideShare
Loading in...5
×

3 Years of Puppet at Cisco: The Secrets to Our Success - PuppetConf 2013

1,706
-1

Published on

"3 Years of Puppet at Cisco: The Secrets to Our Success" by Keith Chambers and Ryan Uber, Cisco WebEx.

Presentation Overview: WebEx Social (WxS) is an enterprise social networking product by Cisco. Customers can subscribe to WxS as a cloud service or they can purchase a license and run WxS on-premise. Puppet is at the core of our technology stack that includes MongoDB, Solr, RabbitMQ, memcached, ZooKeeper, Nagios, and Graphite. We love Puppet. We pioneered embedding Puppet in Cisco products 3 years ago and have been tireless advocates ever since. We have learned valuable technology/process/culture lessons and developed reusable patterns for: - Scaling to thousands hosts with Puppet - Ensuring host upgrades are 100% reliable - Avoiding Puppet codebase spaghetti - Detecting and eliminating system drift at scale - Creating a culture of shared responsibility through architecture design.

Speaker Bio: Keith Chambers,Technical Leader, Cisco
Keith Chambers is the platform architect for the WebEx Social product at Cisco. He joined Cisco in 2000 and is based out of Seattle Washington. In his current role, Keith leads the engineering team responsible for deploying and operating WebEx Social. Prior to joining the WebEx Social team, Keith worked in Cisco's Unified Communication Business Unit and Technical Assistance Center. Keith was instrumental in the successful virtualization of Cisco's entire Unified Communications portfolio and spoke at VMworld 2009 on the topic of "virtualizing real-time applications". VMware presented Keith with their prestigious vExpert award in 2009. Outside of Cisco, Keith is a house deejay and producer. He enjoys traveling the world and spending time with his family, friends, and beautiful wife Jackie.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,706
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
44
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

3 Years of Puppet at Cisco: The Secrets to Our Success - PuppetConf 2013

  1. 1. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1 3 Years of Puppet at Cisco: The Secrets to Our Success PuppetConf 2013 Keith Chambers & Ryan Uber
  2. 2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2 •  Worked at Cisco for 13 years -  7 years in Technical Assistance Center (support) -  2 years in Product Marketing -  4 years as WebEx Social Platform Architect •  Passionate about technology •  Passionate about people •  Always looking to raise the bar
  3. 3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3 •  Worked at Cisco for 3 years -  Software Engineer -  Has played a critical role in our success •  Work at a managed host provider for 5 years -  Designed and implemented most of the infrastructure -  Supported everything he built •  Puppet user for 6 years
  4. 4. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4 •  Overview •  Experience •  Plumbing •  History & Evolution •  Secrets to our success
  5. 5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
  6. 6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6 •  Enterprise Social Media -  Create content -  Communities -  Follow -  Like -  Activity feed -  Recommendations •  On-premise deployment model •  Later added cloud hosted
  7. 7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7 Persistence •  Traditional tiered web service •  Large number of services -  App Server -  Worker -  OpenFire -  RabbitMQ -  Memcached -  MongoDB -  Oracle -  Solr Caching Application Web Server Worker
  8. 8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8 •  Virtual appliance delivery model •  Roles based architecture -  1 service per VM -  25 VM deployments are common •  Director provides system wide: -  Configuration -  Software maintenance -  Diagnostics and logging -  Health monitoring •  Everything works out of the box Oracle OS MongoDB OS Solr OS Worker OS Memcached OS App OS RabbitMQ OS OpenFire OS Virtualized Compute & Storage Director OS
  9. 9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
  10. 10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10 1.  Customer downloads a single OVA from Cisco
  11. 11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11 1.  Customer downloads a single OVA from Cisco 2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option
  12. 12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
  13. 13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13 1.  Customer downloads a single OVA from Cisco 2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option 3.  Customer configures Director VM network settings
  14. 14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14 1.  Customer downloads a single OVA from Cisco 2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option 3.  Customer configures Director VM network settings 4.  Customer browses to the Director UI and configures system settings and defines the topology
  15. 15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
  16. 16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
  17. 17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17 1.  Customer downloads a single OVA from Cisco 2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option 3.  Customer configures Director VM network settings 4.  Customer browses to the Director UI and configures system settings and defines the topology 5.  Customer deploy all VMs defined in the topology from the OVA by selecting the appropriate configuration
  18. 18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18 1.  Customer downloads a single OVA from Cisco 2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option 3.  Customer configures Director VM network settings 4.  Customer browses to the Director UI and configures system settings and defines the topology 5.  Customer deploy all VMs defined in the topology from the OVA by selecting the appropriate configuration •  System handles all other setup and configuration
  19. 19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19 1.  Customer download a single release image from file from Cisco
  20. 20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20 1.  Customer download a single release image from file from Cisco 2.  Customer uploads the release image to the Director
  21. 21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
  22. 22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22 1.  Customer download a single release image from file from Cisco 2.  Customer uploads the release image to the Director 3.  Customer clicks upgrade
  23. 23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23 1.  Customer download a single release image from file from Cisco 2.  Customer uploads the release image to the Director 3.  Customer clicks upgrade •  System handles the upgrade
  24. 24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
  25. 25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
  26. 26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
  27. 27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27 OVF OS Bootstrap Config Network Settings Director FQDN Glue Code
  28. 28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
  29. 29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29 •  Package everything as an RPM •  RPMs distributed from Director via Yum •  Controller is a Puppet Master Director OS Analytics Store OS Puppet Master Yum
  30. 30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30 •  Versioned build artifact containing: -  All RPMs for all roles -  All Puppet modules •  Upgrade mechanics: -  New image uploaded to Director -  Old image unmounted and new image mounted -  Director calls Puppet run for all hosts -  On success the updated version is stored in on VM file system -  Salt used to fetch version number from all VMs
  31. 31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31 Director OS Analytics Store QUAD OS JSON Store QUAD OS RDBMS Store QUAD OS Graph Store QUAD OS Rsyslog •  Rsyslog -  High performance -  Reliable -  Part of CentOS 6 Rsyslogx Rsyslogx Rsyslogx Rsyslogx
  32. 32. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32 Director OS Analytics Store OS BSON Store OS RDBMS Store OS Social Graph OS monit monit monit monit Nagios •  Nagios: -  Reliable -  Extremely flexible -  Available in EPEL •  Monit -  Reliable -  Flexible and extensible -  Enables self healing -  HTTP API -  Available in EPEL
  33. 33. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33 Director OS Analytics Store OS BSON Store OS RDBMS Store OS Social Graph OS Graphite collectd collectd collectd collectd •  Graphite: -  Reliable -  Extremely flexible -  Available in EPEL •  Collectd -  Reliable -  100+ plugins -  Extensible -  In EPEL
  34. 34. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
  35. 35. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35 Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0 •  100% proprietary •  Time to market focused •  No DevOps tooling experience in the team
  36. 36. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36 Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0 •  100% proprietary •  Time to market focused •  No DevOps tooling experience in the team Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint •  Major pushback on Puppet •  Your team matters
  37. 37. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37 Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0 •  100% proprietary •  Time to market focused •  No DevOps tooling experience in the team Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint •  Major pushback on Puppet •  Your team matters Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/ Rsyslog + Scribe •  Rsyslog was resisted due to name association with syslog
  38. 38. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38 Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0 •  100% proprietary •  Time to market focused •  No DevOps tooling experience in the team Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint •  Major pushback on Puppet •  Your team matters Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/ Rsyslog + Scribe •  Rsyslog was resisted due to name association with syslog Phase 4 Early 2012 •  CentOS 6 upgrade •  Monit •  Salt •  Remove Scribe •  Maintain fewer libraries •  Further reduce VM footprint •  Started replacing Monitoring 1.0 •  Salt proved buggy
  39. 39. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39 Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0 •  100% proprietary •  Time to market focused •  No DevOps tooling experience in the team Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint •  Major pushback on Puppet •  Your team matters Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/ Rsyslog + Scribe •  Rsyslog was resisted due to name association with syslog Phase 4 Early 2012 •  CentOS 6 upgrade •  Monit •  Salt •  Remove Scribe •  Maintain fewer libraries •  Further reduce VM footprint •  Started replacing Monitoring 1.0 •  Salt proved buggy Phase 5 Late 2012 •  Replaced Monitoring 1.0 with Nagios, Collectd, Graphite •  Last minute decision to go with Graphite •  Puppet code spaghetti •  VM tampering undercutting upgrade reliability •  Other Cisco groups want to reuse our code
  40. 40. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40 Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0 •  100% proprietary •  Time to market focused •  No DevOps tooling experience in the team Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint •  Major pushback on Puppet •  Your team matters Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/ Rsyslog + Scribe •  Rsyslog was resisted due to name association with syslog Phase 4 Early 2012 •  CentOS 6 upgrade •  Monit •  Salt •  Remove Scribe •  Maintain fewer libraries •  Further reduce VM footprint •  Started replacing Monitoring 1.0 •  Salt proved buggy Phase 5 Late 2012 •  Replaced Monitoring 1.0 with Nagios, Collectd, Graphite •  Last minute decision to go with Graphite •  Puppet code spaghetti •  VM tampering undercutting upgrade reliability •  Other Cisco groups want to reuse our code Phase 6 Started 2012 •  Generic programmable Controller •  Deploy VMs via IaaS API •  Orchestrate deploy and upgrade with Fabric •  Baked VM images •  Hiera for module parameters
  41. 41. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
  42. 42. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42 Keep it simple •  80/20 rule •  Build only what you need •  What works is boring Learn and adapt •  Don’t fall in love with what you’ve built Unix tool chain pattern •  Loosely couple with best of bread tools •  End-to-end solutions lock you in
  43. 43. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43 •  Excellent at configuration management •  Bulletproof reliability •  Long term investment protection •  Vibrant community and ecosystem •  Commitment to open source
  44. 44. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44 •  Alex Honor of DTO Solutions and Simplify Ops -  Insightful program leadership -  Walking encyclopedia of proven patterns -  Always there when I’m in a jam
  45. 45. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×