Your SlideShare is downloading. ×
Puppet at Google
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Puppet at Google


Published on

Gordon Rowell's talk "Puppet at Google" from Puppet Camp Sydney 2013.

Gordon Rowell's talk "Puppet at Google" from Puppet Camp Sydney 2013.

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Puppet at Google Gordon RowellPuppet Camp Sydney 2013
  • 2. Non-GoalsNot here to to talk about● Hiring practices● Release schedules● Puppet configs● Monitoring● Compliance● Auditing● ...See also Jason Wrights talk from PuppetConf 2011
  • 3. BackgroundPuppet at Google is offered as an infrastructure service● Run by a Site Reliability Engineering (SRE) team● Customers are OS teams● Does not manage Googles customer facing infrastructure (search, Gmail, etc.)!● Manages internal laptops, desktops and servers
  • 4. How Many Nodes?Clients: ● "Lots" of Mac desktops and laptops ● "Lots" of Ubuntu desktops, laptops and servers ● "Some" othersServers: ● "Tens" of puppet config servers ● "Units" of puppet CAs ● Deployed in five globally distributed VIPs ● Clients use Anycast to find closest "server"
  • 5. Scaling is fun● We dont deploy "a server" ○ Servers break, power fails ○ Clients/DNS need to be reconfigured● We dont deploy "a cluster" ○ Networks break, servers break, power fails ○ Clients/DNS need to be reconfigured● We deploy redundant clusters ○ Attempt to send clients to nearest serving cluster ○ Anycast means unified client configuration
  • 6. Load balancing is funDo you have enough capacity? ● How many backends do you need? ● What happens if half of your backends lose power? ● What about when half are already out for repairs?How do you send clients to the right cluster? ● Client configuration ● DNS round-robin (simple global load balancing) ● DNS views (give best answer for client IP) ● Anycast (portable IP, routed to "nearest" cluster) ● Consider: DNS views plus Anycast
  • 7. Anycast is fun● Anycast is "coarse-grain" load balancing ○ It normally sends traffic to closest serving cluster● Networks break ○ Physical issues ○ Routing issues ○ Configuration issues ○ VIP load balancer bugs● All clients could be sent to the same cluster ○ Be ready for that ○ Can a single cluster handle worldwide traffic? ○ What do you do if you cant?
  • 8. Puppet problems: Thundering herds● "Lots" + "lots" + "some" == "thundering herds"● What if they all want to do a puppet run?● What about every hour?● What about every five minutes?● Masterless puppet is being considered
  • 9. Puppet problems: Release tracks● OS releases have unstable, testing, stable branches ○ Maintained by OS platform teams● Addons also have unstable, testing, stable branches ○ Maintained by service owners● Using different tracks for OS and addons is hard ○ However, thats common - testing a new addon release ○ Puppets global namespace is part of the problem
  • 10. Puppet problems: Namespaces● Lots of developers moving fast == conflicts● Conflicts mean surprises● Qualify everything● Testing with rspec-puppet helps to catch issues early
  • 11. Questions? Gordon Rowell