The document outlines strategies for scaling infrastructure management to 5000 agents using Puppet, emphasizing the importance of consistency and automation in setup processes. It details best practices in environment configuration, version control, and module management, alongside the transition to effective tools like Hiera for data management. The evolution from a lengthy, error-prone deployment process to a streamlined approach is highlighted, showcasing significant improvements in efficiency and reliability.

1. Learning Lessons Scaling to
5000 Agents
Russ Johnson
rjohnson@stubhub.com
@professoruss
Confidential Slide 1

2. #whoami
Started out in the mid 90’s
Recovering Windows Admin
Storage Guy
Datacenter Monkey
Once upon a time network guy
At StubHub since December 2006
Working on puppet adoption in a crazy infrastructure
Puppet certified
April 9, 2013 2

3. 4 Puppet Masters
DEV/QA PROD
DR CORP
April 9, 2013 3

4. The road to sanity
April 9, 2013 4

5. Set up your master properly
Apache/Passenger
Tune Passenger
PassengerMaxPoolSize 32
PassengerMinInstances 4
PassengerMaxRequests 10000
PassengerStatThrottleRate 30
16 cores, 32GB
load average: 4.03, 3.71, 3.45
4000+ agents
April 9, 2013 5

6. Thundering herds
April 9, 2013 6

7. Build your hosts the same way!
Old way:
systemimager, vmware clones, manual installs
Results:
INCONSISTENCY!
April 9, 2013 7

8. Build your hosts the same way!
New way:
Cobbler < 5m bare metal to on the network
Results:
Same results every time! No drift between base
April 9, 2013 8

9. Set up your working environment properly
Geppetto – eclipse based IDE
http://cloudsmith.github.com/geppetto/index.html
VIM
• Pathogen – For autoloading vim plugins
• Snipmate – Snippets
• Tabular – Text filtering and alignment
• Syntastic – Syntax checking
• mv-vim-puppet – Make vim puppet friendly
• puppet-lint – Syntax checker (gem)
April 9, 2013 9

10. Syntastic/puppet-lint
April 9, 2013 10

11. Set up your working environment properly
+
April 9, 2013 11

12. Version Control is not enough
Ever do a 4 way diff across 60 modules to find most of
them different?
April 9, 2013 12

13. What to do?
How do I stop manual edits?
How do I deal with 80+ Dev/QA Environments?
Versioning? Librarian?
Pulp?
Branching?
Internal Forge?
Dynamic Environments?
Puppet Module Tool?
What does PuppetLabs do?
April 9, 2013 13

14. PuppetLabs seems to know what to do
Let’s investigate
puppet module tool
http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013 14

15. Generate a module
http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013 15

16. Edit Modulefile
http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html
April 9, 2013 16

17. Document the manifest
http://rdoc.sourceforge.net/
April 9, 2013 17

18. Write Documentation?
April 9, 2013 18

19. Free Docs!
puppet doc -a -o /var/www/html/puppetdocs --mode rdoc
April 9, 2013 19

20. What’s actually installed?
April 9, 2013 20

21. Catching live edits and preventing them
Splunk -> puppet module changes -> alerting
The NOC will hunt you down!
April 9, 2013 21

22. Build and install the module
tar –xzf /tmp/work/stubhub-puppetserver/pkg/stubhub-
puppetserver-0.0.1.tar.gz –C /etc/puppet/environments/
staging/modules/puppetserver
April 9, 2013 22

23. Releasing like that?
April 9, 2013 23

24. Internal Forge
mod_rewrite:
Simulate the api – redirect to json metadata files
$htmlroot/api/v1/releases.json?module=user/module
ruby script:
Generate metadata files for each module release and
all modules.
Similar to createrepo (yum)
April 9, 2013 24

25. Internal Forge - Search
April 9, 2013 25

26. Internal Forge - install
April 9, 2013 26

27. Internal Forge - upgrade
April 9, 2013 27

28. Case statements? How bout Hiera?
April 9, 2013 28

29. Avoid case statement insanity
case $::system_role {
‘browse’, ‘search’: {
…do some stuff…
}
‘db’: {
…other stuff…
}
‘otherrole’: {
…please make it stop!!!!
}
}
April 9, 2013 29

30. hieradata
$hieradata/browse.yaml:
---
module::parameter: ‘foo’
$hieradata/search.yaml:
---
module::parameter: ‘bar’
$hieradata/defaults.yaml:
---
module::parameter: ‘I want this everywhere unless there are overrides’
April 9, 2013 30

31. Case -> variables -> hiera
§ 9000 lines of case statements
§ 1000 lines with case/variables
§ ~20 lines with defined type
Code compression FTW!
April 9, 2013 31

32. Dynamic Environments
§ puppet.conf:
modulepath = /etc/puppet/environments/$environment/modules
manifest = /etc/puppet/environments/$environment/manifests/site.pp
manifestdir = /etc/puppet/environments/$environment/manifests
§ hiera.yaml:
:datadir: '/etc/puppet/environments/%{environment}/hieradata'
April 9, 2013 32

33. Release process
§ Syntax check/validate
§ Test on VMs
§ Build module package
§ Release to internal forge
§ puppet module install to staging environment
§ Test again!
§ puppet module install to production environment
April 9, 2013 33

34. The road to yesop
§ Staging
§ Process
§ Repeatability
§ Consistency
§ Document everything
§ Breaking things where it’s cheap
§ Test everything!
April 9, 2013 34

35. Then VS now
§ Environment build time:
– Then: 3+ weeks
• It was wrong
• It didn’t work
• Nobody knew what to expect
– Now: < 1 day
• It’s the same every time
• We know exactly what’s installed
• Internal consumers get what they expect
• Less outages from human error
April 9, 2013 35

36. Questions?
April 9, 2013 36