Keeping AWS in check with Puppet 
or 
“Managing cloud networking with Puppet with a healthy dose of 
trolling yourself on ...
HELLO.
What’s all this about then? 
● We use AWS extensively, especially EC2 
● We needed a way to organise our AWS networking (V...
Spoiler 
We (Tom) decided to use Puppet 
We wrote a set of types and providers to use 
with the AWS API
What I’m going to cover 
YOU ARE HERE 
1. Intro 
2. AWS networking basics 
3. Why we chose Puppet 
4. How it actually work...
What I hope you’ll get from this 
● A sense of Puppet as an extensible framework that can 
manage dependencies in external...
To serve as a lesson to others.
SOME AWS BASICS.
Some 
of this 
isn’t 
exactly 
retina 
resolution
The (simplified) Hierarchy 
Account 
Region 
VPC 
Routetable 
Subnet
The (needlessly complicated simplified) 
Hierarchy 
Account: 
• Region 
• dopt: DHCP Options 
• vpc: Virtual Private Cloud...
How do we make all these objects? 
Just make them in the console!
How do we make all these objects? 
IF YOU HATE YOURSELF 
Just make them in the console!
How do we actually make all of these objects? 
• AWS::SDK! 
• We’ll have to ensure that the resources get created (and 
pu...
USE PUPPET!
Create Puppet types for VPC objects! 
● Resources for each of the objects 
○ All API calls made on this level 
○ Will cont...
(Actually though) Why Puppet? 
• With the ability to query and create and modify objects 
through the API we can state the...
Why Puppet? 
Using this model you can even collect dependencies and 
resources: 
• Make all resources of a type with a par...
Why Puppet? 
Admittedly it seems a little asymmetrical: 
• Puppet runs on nodes which creates 
resources in AWS 
• AWS net...
HOW IT WORKS.
Puppet Types 
• The bit that you specify in 
the manifest 
• Really just a DDL for the 
resource’s metadata 
• The “front ...
Puppet Providers 
• The business end 
• This is where the API 
code lives 
• Quite a lot longer 
• Handles all application...
What’s a “prefetch”? 
• Some resources are 
expensive to read, so 
you read them all once 
when you first come 
across one...
Manifest examples
Associated Infrastructure 
• How do you handle multiple accounts? 
• Dedicated AWS admin box within each 
account to apply...
LESSONS LEARNED. 
(the hard way)
Don’t 
do 
what 
Donny 
Don’t 
does
Why not put credentials in the code?
Why not put credentials in the code? 
• We (I) tried to make an 
aws_credentials type 
• This requires access to the 
cata...
Why not put credentials in the code? 
• The instances method suddenly 
requires an argument 
• In fact, so does anything t...
On second thoughts...
Prefetch isn’t… Exceptional 
• https://tickets.puppetlabs.com/browse/PUP-3656 
• This means that if anything goes awry in ...
So we did a bad thing...
Asynchronous APIs tell lies 
• When you create an object you get 
a 200 OK 
• This doesn’t mean “I’ve done it”, 
this mean...
SO IN SUMMARY...
What I learned (the hard way) 
• The “resources” resource 
• Is hard to understand unless 
you read the code (type only) 
...
What I learned (the hard way) 
• Instances vs. class methods paradigm gets very confusing 
because it is tied to applying ...
What really cool things I learned 
• You can reproduce AWS networking (and other objects 
thanks to other contributors!) 
...
THANKS FOR LISTENING! 
github.com/bobtfish/puppet-aws_api 
Matt Carroll 
SRE at Yelp 
@oholiab 
mattc@yelp.com 
yelp.com/c...
Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)
Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)
Upcoming SlideShare
Loading in …5
×

Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)

966 views

Published on

How we (at Yelp) use Puppet to manage AWS VPCs and what we learned along the way

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
966
On SlideShare
0
From Embeds
0
Number of Embeds
401
Actions
Shares
0
Downloads
7
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)

  1. 1. Keeping AWS in check with Puppet or “Managing cloud networking with Puppet with a healthy dose of trolling yourself on the side” Matt Carroll
  2. 2. HELLO.
  3. 3. What’s all this about then? ● We use AWS extensively, especially EC2 ● We needed a way to organise our AWS networking (VPCs) ● It needs to be: ○ Centrally managed ○ Reproducible ○ Declarative (idempotent) ○ Ideally not another solution on top of a stack of solutions
  4. 4. Spoiler We (Tom) decided to use Puppet We wrote a set of types and providers to use with the AWS API
  5. 5. What I’m going to cover YOU ARE HERE 1. Intro 2. AWS networking basics 3. Why we chose Puppet 4. How it actually works 5. Some of the “interesting” things we learned 6. Summary 7. Questions
  6. 6. What I hope you’ll get from this ● A sense of Puppet as an extensible framework that can manage dependencies in external APIs ● Nodes are just units of computing capacity - an operation does not have to be a subset of a node ● Some insight into the types and providers system ● Some of the strange things we learned along the way … and as always...
  7. 7. To serve as a lesson to others.
  8. 8. SOME AWS BASICS.
  9. 9. Some of this isn’t exactly retina resolution
  10. 10. The (simplified) Hierarchy Account Region VPC Routetable Subnet
  11. 11. The (needlessly complicated simplified) Hierarchy Account: • Region • dopt: DHCP Options • vpc: Virtual Private Cloud • igw: Internet Gateway • cgw: Customer Gateway • vgw: Virtual Gateway • vpn: VPN • routetable • subnet
  12. 12. How do we make all these objects? Just make them in the console!
  13. 13. How do we make all these objects? IF YOU HATE YOURSELF Just make them in the console!
  14. 14. How do we actually make all of these objects? • AWS::SDK! • We’ll have to ensure that the resources get created (and purged)... • Idempotently… • With all their dependencies… • Remind you of something?
  15. 15. USE PUPPET!
  16. 16. Create Puppet types for VPC objects! ● Resources for each of the objects ○ All API calls made on this level ○ Will contain all code for reading and creating individual resources ○ No dependencies other than autorequires in the types - those in the hierarchy earlier ● Business logic in manifests ○ For your site-specific dependencies and network structure ○ e.g. we have a separate VPC for each environment ● Data in hiera
  17. 17. (Actually though) Why Puppet? • With the ability to query and create and modify objects through the API we can state them declaratively • We can thus create resources which can be included or purged idempotently • Rather than specify order, we can state dependencies and allow Puppet to figure order out
  18. 18. Why Puppet? Using this model you can even collect dependencies and resources: • Make all resources of a type with a parameter evaluate after another • Control purge and no-op of all aws resources
  19. 19. Why Puppet? Admittedly it seems a little asymmetrical: • Puppet runs on nodes which creates resources in AWS • AWS networking is not a subset of a node • Nodes just act as executors for creating AWS resource But actually this plays to our advantage (more on this later)
  20. 20. HOW IT WORKS.
  21. 21. Puppet Types • The bit that you specify in the manifest • Really just a DDL for the resource’s metadata • The “front end” for the pluggable “back end” (the provider) • Interface to specify all properties and parameters
  22. 22. Puppet Providers • The business end • This is where the API code lives • Quite a lot longer • Handles all application and querying of the resource including prefetch
  23. 23. What’s a “prefetch”? • Some resources are expensive to read, so you read them all once when you first come across one • This is done before the catalog is completely compiled
  24. 24. Manifest examples
  25. 25. Associated Infrastructure • How do you handle multiple accounts? • Dedicated AWS admin box within each account to apply the resources on • IAM roles to handle credentials • Logging resource changes separately • Why not do it in the code...?
  26. 26. LESSONS LEARNED. (the hard way)
  27. 27. Don’t do what Donny Don’t does
  28. 28. Why not put credentials in the code?
  29. 29. Why not put credentials in the code? • We (I) tried to make an aws_credentials type • This requires access to the catalog in the prefetch phase so other resources can query it • You ALSO need to guarantee type evaluation order and access to credentials in the catalog so that you can use prefetch
  30. 30. Why not put credentials in the code? • The instances method suddenly requires an argument • In fact, so does anything that isn’t an instance method • Actually ended up copying and pasting the resources resource to aws_resources and adding in a “credentials” parameter
  31. 31. On second thoughts...
  32. 32. Prefetch isn’t… Exceptional • https://tickets.puppetlabs.com/browse/PUP-3656 • This means that if anything goes awry in prefetch, puppet will swallow it • In our case, we hit the API limit occasionally, meaning we got duplicate resources
  33. 33. So we did a bad thing...
  34. 34. Asynchronous APIs tell lies • When you create an object you get a 200 OK • This doesn’t mean “I’ve done it”, this means “I’ll do it” • In a dependency chain, this can mean that a resource is about to be created, but when it’s checked by another resource it’s not there yet • In the Puppet paradigm it’s best just to run until convergence.
  35. 35. SO IN SUMMARY...
  36. 36. What I learned (the hard way) • The “resources” resource • Is hard to understand unless you read the code (type only) because Googling it is impossible. • Could do with being able to apply other arbitrary properties and parameters?* *this may be an awful idea
  37. 37. What I learned (the hard way) • Instances vs. class methods paradigm gets very confusing because it is tied to applying catalogue vs. prefetch • Prefetch is generally pretty confusing when you add in dependencies TO the prefetch
  38. 38. What really cool things I learned • You can reproduce AWS networking (and other objects thanks to other contributors!) • A really cool insight into the types and providers system and how it could grow in the future • Learning to treat servers not like pets OR cattle but as a medium by which you do useful stuff … and you could do this with a whole bunch of APIs
  39. 39. THANKS FOR LISTENING! github.com/bobtfish/puppet-aws_api Matt Carroll SRE at Yelp @oholiab mattc@yelp.com yelp.com/careers (we’re hiring)

×