"Puppet for Production in WebEx" by Reinhardt Quelle, Cloud Services Architect, Cisco. Presentation Overview: Getting started with Puppet configuring an individual machine is straightforward. Managing a cluster of machines across multiple data centers, supporting upgrades while running a 7x24 service, and building for collaboration is significantly more challenging. The WebEx team will discuss the problems and some strategies they are using to manage this complexity. Speaker Bio: Reinhardt Quelle is a Cloud Services Architect in the Cloud Collaboration Applications group at Cisco, where he’s responsible for defining infrastructure architecture and deployment automation . His group manages thousands of servers across multiple data centers around the world serving multiple applications, including WebEx conferencing, to tens of millions of users. In prior roles, he’s worked extensively in SaaS operations, delivering diverse applications from email security through social media applications.

1. Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 1
Puppet at Cisco CCATG
Aug 23, 2013
Reinhardt Quelle, Cloud Services Architect

2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Millions of Meetings
for
10s of Millions of Users
totaling
Billions of Minutes
each month
7x24x365
Cisco Social
WebEx
Connect

3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
U
S
U
K
Indi
a
Australi
a
China
Hong
Kong
Amsterdam
Japa
n
~ 7K Hosts
~ 8 Data Centers
> 12 iPOPs
Private Backbone

4. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 4Cisco Confidential 4© 2011 Cisco and/or its affiliates. All rights reserved.

5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5

6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6

7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7

8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Files
Packages
=
Users
Services
…
Etc.

10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Manifests
- nodes.pp
- site.pp
Classes, Modules
=

11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11

12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12

13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
BaseOS_Hardening v1.1
ElasticSearch v0.20.6
JRE v1.7.0_25
BaseOS_Hardening v1.1
ElasticSearch v0.90.2-1
JRE v1.7.0_25
• Some systems can simply be knocked over the head and recreated with
fresh versions
• Others – notably most database servers – cannot; updates are
performed in-place
• “Big Bang” upgrades don’t often happen; we step methodically through
groups of machines

14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14

15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
DC1 DC2
Multiple DC Pairs
Multiple Clusters of each Service type
• By Customer Class
• By Lifecycle Stage
• By Special Needs

16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
By DC
Or by Node
v1
v1 v2
v2
v1 v1
v2v2

17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
“Blueprints” or “Models”
• JSON/YAML
• TOSCA
• CMDB *
Orchestration
• Fabric
• SLiM
• Mcollective

18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
Puppet
Master
Manifests &
Modules
DC1 DC2

19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
Puppet
Master
Manifests &
Modules
DC1 DC2
Guess when you’ll need to push infrastructure changes the most!

20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
Puppet
Master
Manifests &
Modules
DC1 DC2
Puppet
Master
Manifests &
Modules

21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Puppet
Master
Manifests &
Modules
DC1 DC2
Puppet
Master
Manifests &
Modules

22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
WebEx
Meetings
WebEx
Connect

23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
puppet apply
-–modulepath=/opt/puppet_local
--execute “include servertype::front-end”
Manifests &
Modules copy [/etc/puppet/*] to each node

24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Manifests &
Modules .rpm or .deb
yum install app_pp_v1 && puppet apply …‟
private
package
repository

25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
ssh node81 „yum install app_pp_v1 && puppet apply …‟
fab dfw-frontends pp_apply:latest‟
Fabric
Mcollective
Salt
Ansible

26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
• Application Stacks/Deployment are NOT Homogenous
• The “right” solution for one stack not always right for another
• Share as much as possible, but don’t force it
• Tightly coupled systems are often rigid, brittle
• Solving big, general problems is hard; small bites are easily
digested
“A foolish consistency is the hobgoblin of little minds” – Emerson

27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
• Every artifact (module, manifest, Hiera file) is checked into
version control
• Versions are packaged and released and should go through same
promotion process as application code
• All good coding practices apply
Modular
Well defined interfaces
Tested
Shared

28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
• Modules are the atomic packages of configuration
• “Profiles” bundle modules into commonly used sets for ease of
consumption:
BaseOS
JavaApp
Tomcat App
• A given machine has exactly one “ServerType”
• Inspired by Chef’s “roles”, and similar to Craig Dunn’s
Role/Profile/Modules
• At the code level, these are actually all just modules

29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
• Build loosely coupled modules that can work together if installed
together, but that can stand on own, too
• Example: standard monit config includes /etc/monit/conf.d/*
Application that wants to be monitored just drops file in this location
• logrotate, collectd, apache, nginx, etc all support
class elasticsarch {
…
if $monit::include_dir != undef {
validate_absolute_path($monit::include_dir)
file { "${monit::include_dir}/${monit_config_file_name}":
ensure => present,
content => template("elasticsearch/${monit_config_file_name}.erb"),
notify => Service['monit'],
}
}
}

30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
• The singleton ServerType defines which profiles and modules are
included; structure and order
• Puppet Librarian and its Puppetfile describe which version of a
module is used, and where it comes from
• After Puppet Librarian has run and downloaded all required
assets, FPM is called upon to build the package
• Dev cycle includes doing local builds/tests against Vagrant in both
develpers personal machines as well as the Jenkin’s build farm
• Upon successful build, packages are uploaded to repositories

31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
• Dependencies in Puppet, not RPM/DEB
• Packaging works for deploying to Puppetmasters, too.
• Modules are designed for transparency, simplicity: “4AM-proofing”
• Composition usually trumps inheritance
• Tim Bell and the CERN folks talk of “Pets” and “Cattle”
You can only shoot a system in the head if you can create another at will
• “Fried” or “Baked”? YES.

32. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 36Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 36