Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Puppet at Google

3,573 views

Published on

Gordon Rowell's talk "Puppet at Google" from Puppet Camp Sydney 2013.

Published in: Technology
  • Be the first to comment

Puppet at Google

  1. 1. Puppet at Google Gordon RowellPuppet Camp Sydney 2013 gordonr@google.com
  2. 2. Non-GoalsNot here to to talk about● Hiring practices● Release schedules● Puppet configs● Monitoring● Compliance● Auditing● ...See also Jason Wrights talk from PuppetConf 2011
  3. 3. BackgroundPuppet at Google is offered as an infrastructure service● Run by a Site Reliability Engineering (SRE) team● Customers are OS teams● Does not manage Googles customer facing infrastructure (search, Gmail, etc.)!● Manages internal laptops, desktops and servers
  4. 4. How Many Nodes?Clients: ● "Lots" of Mac desktops and laptops ● "Lots" of Ubuntu desktops, laptops and servers ● "Some" othersServers: ● "Tens" of puppet config servers ● "Units" of puppet CAs ● Deployed in five globally distributed VIPs ● Clients use Anycast to find closest "server"
  5. 5. Scaling is fun● We dont deploy "a server" ○ Servers break, power fails ○ Clients/DNS need to be reconfigured● We dont deploy "a cluster" ○ Networks break, servers break, power fails ○ Clients/DNS need to be reconfigured● We deploy redundant clusters ○ Attempt to send clients to nearest serving cluster ○ Anycast means unified client configuration
  6. 6. Load balancing is funDo you have enough capacity? ● How many backends do you need? ● What happens if half of your backends lose power? ● What about when half are already out for repairs?How do you send clients to the right cluster? ● Client configuration ● DNS round-robin (simple global load balancing) ● DNS views (give best answer for client IP) ● Anycast (portable IP, routed to "nearest" cluster) ● Consider: DNS views plus Anycast
  7. 7. Anycast is fun● Anycast is "coarse-grain" load balancing ○ It normally sends traffic to closest serving cluster● Networks break ○ Physical issues ○ Routing issues ○ Configuration issues ○ VIP load balancer bugs● All clients could be sent to the same cluster ○ Be ready for that ○ Can a single cluster handle worldwide traffic? ○ What do you do if you cant?
  8. 8. Puppet problems: Thundering herds● "Lots" + "lots" + "some" == "thundering herds"● What if they all want to do a puppet run?● What about every hour?● What about every five minutes?● Masterless puppet is being considered
  9. 9. Puppet problems: Release tracks● OS releases have unstable, testing, stable branches ○ Maintained by OS platform teams● Addons also have unstable, testing, stable branches ○ Maintained by service owners● Using different tracks for OS and addons is hard ○ However, thats common - testing a new addon release ○ Puppets global namespace is part of the problem
  10. 10. Puppet problems: Namespaces● Lots of developers moving fast == conflicts● Conflicts mean surprises● Qualify everything● Testing with rspec-puppet helps to catch issues early
  11. 11. Questions? Gordon Rowell gordonr@google.com

×