Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 1
Puppet at Cisco CCATG
Aug 23, 2013
Reinhardt ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Millions of Meetings
for
10s of Millions of ...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
U
S
U
K
Indi
a
Australi
a
China
Hong
Kong
Am...
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 4Cisco Confidential 4© 2011 Cisco and/or its af...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Files
Packages
=
Users
Services
…
Etc.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Manifests
- nodes.pp
- site.pp
Classes, Mod...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
BaseOS_Hardening v1.1
ElasticSearch v0.20.6...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
DC1 DC2
Multiple DC Pairs
Multiple Clusters...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
By DC
Or by Node
v1
v1 v2
v2
v1 v1
v2v2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
“Blueprints” or “Models”
• JSON/YAML
• TOSC...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
Puppet
Master
Manifests &
Modules
DC1 DC2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
Puppet
Master
Manifests &
Modules
DC1 DC2
G...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
Puppet
Master
Manifests &
Modules
DC1 DC2
P...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Puppet
Master
Manifests &
Modules
DC1 DC2
P...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
WebEx
Meetings
WebEx
Connect
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
puppet apply 
-–modulepath=/opt/puppet_loca...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Manifests &
Modules .rpm or .deb
yum instal...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
ssh node81 „yum install app_pp_v1 && puppet...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
• Application Stacks/Deployment are NOT Hom...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
• Every artifact (module, manifest, Hiera f...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
• Modules are the atomic packages of config...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
• Build loosely coupled modules that can wo...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
• The singleton ServerType defines which pr...
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
• Dependencies in Puppet, not RPM/DEB
• Pac...
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 36Cisco Confidential© 2011 Cisco and/or its aff...
Upcoming SlideShare
Loading in...5
×

Puppet for Production in WebEx - PuppetConf 2013

20,965

Published on

"Puppet for Production in WebEx" by Reinhardt Quelle, Cloud Services Architect, Cisco.

Presentation Overview: Getting started with Puppet configuring an individual machine is straightforward. Managing a cluster of machines across multiple data centers, supporting upgrades while running a 7x24 service, and building for collaboration is significantly more challenging. The WebEx team will discuss the problems and some strategies they are using to manage this complexity.

Speaker Bio: Reinhardt Quelle is a Cloud Services Architect in the Cloud Collaboration Applications group at Cisco, where he’s responsible for defining infrastructure architecture and deployment automation . His group manages thousands of servers across multiple data centers around the world serving multiple applications, including WebEx conferencing, to tens of millions of users. In prior roles, he’s worked extensively in SaaS operations, delivering diverse applications from email security through social media applications.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
20,965
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
71
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • We don’t rely upon OS package management dependencies; these should be explicitly listed in Puppet manifests.Nothing precludes installing RPM containing Puppet config onto a puppet master; use “Environments” and yum --installrootModules are designed for transparency, simplicity: “4AM-proofing”Composition usually trumps inheritanceTim Bell and the CERN folks talk of “Pets” and “Cattle”You can only shoot a system in the head if you can create another at will“Fried” or “Baked”? YES. Even for systems which we launch from snapshots, the system has to come from version control.
  • Puppet for Production in WebEx - PuppetConf 2013

    1. 1. Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 1 Puppet at Cisco CCATG Aug 23, 2013 Reinhardt Quelle, Cloud Services Architect
    2. 2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2 Millions of Meetings for 10s of Millions of Users totaling Billions of Minutes each month 7x24x365 Cisco Social WebEx Connect
    3. 3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3 U S U K Indi a Australi a China Hong Kong Amsterdam Japa n ~ 7K Hosts ~ 8 Data Centers > 12 iPOPs Private Backbone
    4. 4. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 4Cisco Confidential 4© 2011 Cisco and/or its affiliates. All rights reserved.
    5. 5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
    6. 6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
    7. 7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
    8. 8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
    9. 9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9 Files Packages = Users Services … Etc.
    10. 10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10 Manifests - nodes.pp - site.pp Classes, Modules =
    11. 11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
    12. 12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
    13. 13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13 BaseOS_Hardening v1.1 ElasticSearch v0.20.6 JRE v1.7.0_25 BaseOS_Hardening v1.1 ElasticSearch v0.90.2-1 JRE v1.7.0_25 • Some systems can simply be knocked over the head and recreated with fresh versions • Others – notably most database servers – cannot; updates are performed in-place • “Big Bang” upgrades don’t often happen; we step methodically through groups of machines
    14. 14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
    15. 15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15 DC1 DC2 Multiple DC Pairs Multiple Clusters of each Service type • By Customer Class • By Lifecycle Stage • By Special Needs
    16. 16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16 By DC Or by Node v1 v1 v2 v2 v1 v1 v2v2
    17. 17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17 “Blueprints” or “Models” • JSON/YAML • TOSCA • CMDB * Orchestration • Fabric • SLiM • Mcollective
    18. 18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18 Puppet Master Manifests & Modules DC1 DC2
    19. 19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19 Puppet Master Manifests & Modules DC1 DC2 Guess when you’ll need to push infrastructure changes the most!
    20. 20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20 Puppet Master Manifests & Modules DC1 DC2 Puppet Master Manifests & Modules
    21. 21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21 Puppet Master Manifests & Modules DC1 DC2 Puppet Master Manifests & Modules
    22. 22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22 WebEx Meetings WebEx Connect
    23. 23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23 puppet apply -–modulepath=/opt/puppet_local --execute “include servertype::front-end” Manifests & Modules copy [/etc/puppet/*] to each node
    24. 24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24 Manifests & Modules .rpm or .deb yum install app_pp_v1 && puppet apply …‟ private package repository
    25. 25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25 ssh node81 „yum install app_pp_v1 && puppet apply …‟ fab dfw-frontends pp_apply:latest‟ Fabric Mcollective Salt Ansible
    26. 26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26 • Application Stacks/Deployment are NOT Homogenous • The “right” solution for one stack not always right for another • Share as much as possible, but don’t force it • Tightly coupled systems are often rigid, brittle • Solving big, general problems is hard; small bites are easily digested “A foolish consistency is the hobgoblin of little minds” – Emerson
    27. 27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27 • Every artifact (module, manifest, Hiera file) is checked into version control • Versions are packaged and released and should go through same promotion process as application code • All good coding practices apply Modular Well defined interfaces Tested Shared
    28. 28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28 • Modules are the atomic packages of configuration • “Profiles” bundle modules into commonly used sets for ease of consumption: BaseOS JavaApp Tomcat App • A given machine has exactly one “ServerType” • Inspired by Chef’s “roles”, and similar to Craig Dunn’s Role/Profile/Modules • At the code level, these are actually all just modules
    29. 29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29 • Build loosely coupled modules that can work together if installed together, but that can stand on own, too • Example: standard monit config includes /etc/monit/conf.d/* Application that wants to be monitored just drops file in this location • logrotate, collectd, apache, nginx, etc all support class elasticsarch { … if $monit::include_dir != undef { validate_absolute_path($monit::include_dir) file { "${monit::include_dir}/${monit_config_file_name}": ensure => present, content => template("elasticsearch/${monit_config_file_name}.erb"), notify => Service['monit'], } } }
    30. 30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30 • The singleton ServerType defines which profiles and modules are included; structure and order • Puppet Librarian and its Puppetfile describe which version of a module is used, and where it comes from • After Puppet Librarian has run and downloaded all required assets, FPM is called upon to build the package • Dev cycle includes doing local builds/tests against Vagrant in both develpers personal machines as well as the Jenkin’s build farm • Upon successful build, packages are uploaded to repositories
    31. 31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31 • Dependencies in Puppet, not RPM/DEB • Packaging works for deploying to Puppetmasters, too. • Modules are designed for transparency, simplicity: “4AM-proofing” • Composition usually trumps inheritance • Tim Bell and the CERN folks talk of “Pets” and “Cattle” You can only shoot a system in the head if you can create another at will • “Fried” or “Baked”? YES.
    32. 32. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 36Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 36
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×