More Related Content Similar to Puppet for Production in WebEx - PuppetConf 2013 (20) Puppet for Production in WebEx - PuppetConf 20131. Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 1
Puppet at Cisco CCATG
Aug 23, 2013
Reinhardt Quelle, Cloud Services Architect
2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Millions of Meetings
for
10s of Millions of Users
totaling
Billions of Minutes
each month
7x24x365
Cisco Social
WebEx
Connect
3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
U
S
U
K
Indi
a
Australi
a
China
Hong
Kong
Amsterdam
Japa
n
~ 7K Hosts
~ 8 Data Centers
> 12 iPOPs
Private Backbone
4. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 4Cisco Confidential 4© 2011 Cisco and/or its affiliates. All rights reserved.
5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Files
Packages
=
Users
Services
…
Etc.
10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Manifests
- nodes.pp
- site.pp
Classes, Modules
=
11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
BaseOS_Hardening v1.1
ElasticSearch v0.20.6
JRE v1.7.0_25
BaseOS_Hardening v1.1
ElasticSearch v0.90.2-1
JRE v1.7.0_25
• Some systems can simply be knocked over the head and recreated with
fresh versions
• Others – notably most database servers – cannot; updates are
performed in-place
• “Big Bang” upgrades don’t often happen; we step methodically through
groups of machines
14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
DC1 DC2
Multiple DC Pairs
Multiple Clusters of each Service type
• By Customer Class
• By Lifecycle Stage
• By Special Needs
16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
By DC
Or by Node
v1
v1 v2
v2
v1 v1
v2v2
17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
“Blueprints” or “Models”
• JSON/YAML
• TOSCA
• CMDB *
Orchestration
• Fabric
• SLiM
• Mcollective
18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
Puppet
Master
Manifests &
Modules
DC1 DC2
19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
Puppet
Master
Manifests &
Modules
DC1 DC2
Guess when you’ll need to push infrastructure changes the most!
20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
Puppet
Master
Manifests &
Modules
DC1 DC2
Puppet
Master
Manifests &
Modules
21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Puppet
Master
Manifests &
Modules
DC1 DC2
Puppet
Master
Manifests &
Modules
22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
WebEx
Meetings
WebEx
Connect
23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
puppet apply
-–modulepath=/opt/puppet_local
--execute “include servertype::front-end”
Manifests &
Modules copy [/etc/puppet/*] to each node
24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Manifests &
Modules .rpm or .deb
yum install app_pp_v1 && puppet apply …‟
private
package
repository
25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
ssh node81 „yum install app_pp_v1 && puppet apply …‟
fab dfw-frontends pp_apply:latest‟
Fabric
Mcollective
Salt
Ansible
26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
• Application Stacks/Deployment are NOT Homogenous
• The “right” solution for one stack not always right for another
• Share as much as possible, but don’t force it
• Tightly coupled systems are often rigid, brittle
• Solving big, general problems is hard; small bites are easily
digested
“A foolish consistency is the hobgoblin of little minds” – Emerson
27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
• Every artifact (module, manifest, Hiera file) is checked into
version control
• Versions are packaged and released and should go through same
promotion process as application code
• All good coding practices apply
Modular
Well defined interfaces
Tested
Shared
28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
• Modules are the atomic packages of configuration
• “Profiles” bundle modules into commonly used sets for ease of
consumption:
BaseOS
JavaApp
Tomcat App
• A given machine has exactly one “ServerType”
• Inspired by Chef’s “roles”, and similar to Craig Dunn’s
Role/Profile/Modules
• At the code level, these are actually all just modules
29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
• Build loosely coupled modules that can work together if installed
together, but that can stand on own, too
• Example: standard monit config includes /etc/monit/conf.d/*
Application that wants to be monitored just drops file in this location
• logrotate, collectd, apache, nginx, etc all support
class elasticsarch {
…
if $monit::include_dir != undef {
validate_absolute_path($monit::include_dir)
file { "${monit::include_dir}/${monit_config_file_name}":
ensure => present,
content => template("elasticsearch/${monit_config_file_name}.erb"),
notify => Service['monit'],
}
}
}
30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
• The singleton ServerType defines which profiles and modules are
included; structure and order
• Puppet Librarian and its Puppetfile describe which version of a
module is used, and where it comes from
• After Puppet Librarian has run and downloaded all required
assets, FPM is called upon to build the package
• Dev cycle includes doing local builds/tests against Vagrant in both
develpers personal machines as well as the Jenkin’s build farm
• Upon successful build, packages are uploaded to repositories
31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
• Dependencies in Puppet, not RPM/DEB
• Packaging works for deploying to Puppetmasters, too.
• Modules are designed for transparency, simplicity: “4AM-proofing”
• Composition usually trumps inheritance
• Tim Bell and the CERN folks talk of “Pets” and “Cattle”
You can only shoot a system in the head if you can create another at will
• “Fried” or “Baked”? YES.
32. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 36Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 36
Editor's Notes We don’t rely upon OS package management dependencies; these should be explicitly listed in Puppet manifests.Nothing precludes installing RPM containing Puppet config onto a puppet master; use “Environments” and yum --installrootModules are designed for transparency, simplicity: “4AM-proofing”Composition usually trumps inheritanceTim Bell and the CERN folks talk of “Pets” and “Cattle”You can only shoot a system in the head if you can create another at will“Fried” or “Baked”? YES. Even for systems which we launch from snapshots, the system has to come from version control.