This document discusses a solution for providing Puppet services globally across multiple regions with poor WAN connectivity. The solution involves building a "Puppeteer" master that acts as a central point of entry for code updates and certificate management. It ensures Puppet masters in each region are in sync. LDAP is used as an external node classifier to provide node definitions across regions. The Puppet file server replicates configuration between masters. F5 load balancers route clients to the nearest master and provide high availability if any master fails. Workflows for adding new servers and masters are also summarized.

1. MultiMaster
scaling for multiple regions
Greg Cockburn
@gergnz

2. problem:
How do we provide a
Puppet Service Globally
When WAN pipes suck

3. what's in our tool box?
VMware ESX
LDAP
F5 Load Balancers
Puppet Enterprise Edition

4. Items that need to be addressed
• Puppet Certificate management
• Node Classification and ENC replication
• Master Replication
• Master Availability
• Master Scalability
• Reporting and notifications
• Change Control

5. One Solution that Worked

6. Build a Puppeteer:
• This is a Puppet Master Master
• No Client Access
• Acts as a PuppetCA
• Central Point of Entry for Code Updates
• Ensures that the Puppet Masters are in sync

7. LDAP as an ENC:
• Existing highly available UNIX/Linux backbone service
• Already replicated to every region
• Masters are configured to speak with their nearest LDAP
replica
• Provides an effective audit trail
• Node definitions are abstracted away from the Puppet
manifests

8. Replicating Puppet Configuration:
• The Puppet Master is effective at syncing files
• Use the Puppet Fileserver to replicate the masters
o manifests
o modules
o files
o templates
• The Puppeteer can 'kick' the other masters to force a run
• Create a puppet::master class to ensure, masters are
fully controlled

9. F5 Global Traffic Management (GTM)
& DNS:
• Local Puppet Master addresses are returned to
clients based on the DNS server the request
came from
• If a Master is down then next nearest is returned
• Any Puppet Master globally can answer the
client

10. F5 Local Traffic Management (LTM):
• On sites with heavy loads this can be used to
rapidly scale the local Puppet Master service
• If a local Master is taken out of service F5 will
automatically send you to the nearest local
Master

11. All Tied Together:

12. Workflow – Adding a New Server
• Define the client characteristics in the LDAP ENC (eg.
Datacentre, Environment, Server Flavour)
• Configure the build tools
• PXE boot then server, OS is installed and puppet
bootstraps
• Once the client certificate is signed the server is
configured

13. Workflow (adding a master):
• Build a 'standard' client
• Redefine in ENC (LDAP) as a puppetmaster
• Destroy local certificates
• generate special certificates on puppetmaster using --
dns_alt_names
• rerun puppet and Master configurations will sync down

14. So What’s New:
Since this configuration was deployed Puppet Labs have
been busy:
• Puppet Sites - Will soon be released and addresses a lot
of the issues here
• PuppetDB – The new standard for stored configs

15. Special thanks to Jon Spinks @ Sourced
Group
Sourced Group are a Puppet Labs partner
providing integration services for Puppet
Enterprise Edition

16. Q&A
Please go and bother Jon Spinks to find out what Sourced
have been doing with Puppet to automate Amazon Web
Services