Puppet at scale

Rhommel Lamas

November 2013
Thursday, November 28, 13
Who am I?

Thursday, November 28, 13
Who am I?
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.

T...
Who am I?
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Co...
Who am I?
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Co...
What is 3scale?
3scale provides tools such as traffic
management, API administration, traffic
reports, and more, to help com...
What does scaling stands for?

Thursday, November 28, 13
What does scaling stands for?
Scalability is the ability of a system, network,
or process to handle a growing amount of
wo...
What are our scaling needs?

Thursday, November 28, 13
What are our scaling needs?
• How many people do we need to scale our
system infrastructure?

Thursday, November 28, 13
What are our scaling needs?
• How many people do we need to scale our
system infrastructure?

• How many environments and ...
What are our scaling needs?
• How many people do we need to scale our
system infrastructure?

• How many environments and ...
Puppet dynamic environments

Thursday, November 28, 13
Puppet dynamic environments
• Purpose of Puppet environments.

Thursday, November 28, 13
Puppet dynamic environments
• Purpose of Puppet environments.
• Puppet base environments at 3scale.

Thursday, November 28...
Puppet environments at 3scale
environments

/etc/puppet
auth.conf

production
Capfile
hieradata

autosign.conf

common.yaml...
[main]
server = puppet.example.com
environment = production
[master]
environment = production
manifest
= $confdir/environm...
[main]
server = puppet.example.com
environment = production
[master]
environment = production
manifest
= $confdir/environm...
:hierarchy:
- '%{::osfamily}'
- '%{::environment}'
- common
:backends:
- yaml
- json
:yaml:
:datadir: "/etc/puppet/environ...
Thursday, November 28, 13
Puppet dynamic environments
• Purpose of Puppet environments.
• Puppet environments at 3scale.
• Puppet development workflo...
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Puppet at 3scale.

Thursday, November 28, 13
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.

Thursday, November 28, 13
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.

Thursday, November 28, 13
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.

Thursday, Novembe...
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay...
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay...
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay...
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay...
Puppet at 3scale.
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay...
ROUTE
53

_x-puppet._tcp.3scale.net
_x-puppet-ca._tcp.3scale.net

Puppet
Master
+
RabbitMQ
Cluster
+
MCollective
Client
+
...
Thursday, November 28, 13
What nobody will tell you about
puppet.

Thursday, November 28, 13
What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so U...
What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so U...
What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so U...
What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so U...
What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so U...
What nobody will tell you about
puppet.
• Puppet-lint is your friend and it will help others to
understand your code, so U...
Questions?
at 3scale
we’re hiring
http://3scale.net/jobs

Rhommel Lamas
Twitter: @rhoml
Thursday, November 28, 13
Thank you.
at 3scale
we’re hiring
http://3scale.net/jobs

Rhommel Lamas
Twitter: @rhoml
Thursday, November 28, 13
Thursday, November 28, 13
Useful links

Thursday, November 28, 13
Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records

Thursday, Novembe...
Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.pupp...
Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.pupp...
Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.pupp...
Useful links
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.pupp...
Upcoming SlideShare
Loading in...5
×

Building scalable applications while scaling your infrastructure by rhommel lamas

176

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
176
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Building scalable applications while scaling your infrastructure by rhommel lamas

  1. 1. Puppet at scale Rhommel Lamas November 2013 Thursday, November 28, 13
  2. 2. Who am I? Thursday, November 28, 13
  3. 3. Who am I? Scale experience •Physical servers Xtratelecom.es. •AWS and other cloud providers at Wuaki.tv and 3scale.net. Thursday, November 28, 13
  4. 4. Who am I? Scale experience •Physical servers Xtratelecom.es. •AWS and other cloud providers at Wuaki.tv and 3scale.net. Config Management • Worked with CFEngine2 for 2 years. • Puppet user since 2009. Thursday, November 28, 13
  5. 5. Who am I? Scale experience •Physical servers Xtratelecom.es. •AWS and other cloud providers at Wuaki.tv and 3scale.net. Config Management • Worked with CFEngine2 for 2 years. • Puppet user since 2009. Obsessed about config management and I use Emacs. Thursday, November 28, 13
  6. 6. What is 3scale? 3scale provides tools such as traffic management, API administration, traffic reports, and more, to help companies publicize their APIs easily and power new types of connected applications. Thursday, November 28, 13
  7. 7. What does scaling stands for? Thursday, November 28, 13
  8. 8. What does scaling stands for? Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth. Wikipedia Thursday, November 28, 13
  9. 9. What are our scaling needs? Thursday, November 28, 13
  10. 10. What are our scaling needs? • How many people do we need to scale our system infrastructure? Thursday, November 28, 13
  11. 11. What are our scaling needs? • How many people do we need to scale our system infrastructure? • How many environments and systems can our team maintain? Thursday, November 28, 13
  12. 12. What are our scaling needs? • How many people do we need to scale our system infrastructure? • How many environments and systems can our team maintain? • Can we delegate configurations easily to our team members? Thursday, November 28, 13
  13. 13. Puppet dynamic environments Thursday, November 28, 13
  14. 14. Puppet dynamic environments • Purpose of Puppet environments. Thursday, November 28, 13
  15. 15. Puppet dynamic environments • Purpose of Puppet environments. • Puppet base environments at 3scale. Thursday, November 28, 13
  16. 16. Puppet environments at 3scale environments /etc/puppet auth.conf production Capfile hieradata autosign.conf common.yaml production.yaml config.ru preview.yaml hiera.yaml manifests production.pp hipchat.yaml preview.pp fileserver.conf nodes production unicorn.rb webserver.pp environments preview webserver.pp modules nginx preview staging Thursday, November 28, 13
  17. 17. [main] server = puppet.example.com environment = production [master] environment = production manifest = $confdir/environments/$environment/manifests/site.pp modulepath = $confdir/environments/$environment/modules Thursday, November 28, 13
  18. 18. [main] server = puppet.example.com environment = production [master] environment = production manifest = $confdir/environments/$environment/manifests/$environment.pp modulepath = $confdir/environments/$environment/modules # production.pp import 'nodes/production/nodes.pp' import 'nodes/production/*.pp Thursday, November 28, 13
  19. 19. :hierarchy: - '%{::osfamily}' - '%{::environment}' - common :backends: - yaml - json :yaml: :datadir: "/etc/puppet/environments/%{environment}/hieradata" Thursday, November 28, 13
  20. 20. Thursday, November 28, 13
  21. 21. Puppet dynamic environments • Purpose of Puppet environments. • Puppet environments at 3scale. • Puppet development workflow (Git). • Puppet-lint • Puppet-rspec • Code Review Thursday, November 28, 13
  22. 22. Thursday, November 28, 13
  23. 23. Thursday, November 28, 13
  24. 24. Thursday, November 28, 13
  25. 25. Puppet at 3scale. Thursday, November 28, 13
  26. 26. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. Thursday, November 28, 13
  27. 27. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. Thursday, November 28, 13
  28. 28. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. Thursday, November 28, 13
  29. 29. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. Thursday, November 28, 13
  30. 30. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. Thursday, November 28, 13
  31. 31. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. • The foreman as our ENC. Thursday, November 28, 13
  32. 32. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. • The foreman as our ENC. • Puppetdb and hiera. Thursday, November 28, 13
  33. 33. Puppet at 3scale. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. • The foreman as our ENC. • Puppetdb and hiera. •MCollective with Rabbitmq connector. Thursday, November 28, 13
  34. 34. ROUTE 53 _x-puppet._tcp.3scale.net _x-puppet-ca._tcp.3scale.net Puppet Master + RabbitMQ Cluster + MCollective Client + PuppetDB + The Foreman Puppet Master + RabbitMQ Cluster + MCollective Client + PuppetDB + The Foreman Puppet Master + RabbitMQ Cluster + MCollective Client + PuppetDB + The Foreman RabbitMQ Load Balancer Puppet Agent + Mcollective Server Thursday, November 28, 13 Puppet Agent + Mcollective Server Puppet Agent + Mcollective Server PostgreSQL RDS MultiAZ
  35. 35. Thursday, November 28, 13
  36. 36. What nobody will tell you about puppet. Thursday, November 28, 13
  37. 37. What nobody will tell you about puppet. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. Thursday, November 28, 13
  38. 38. What nobody will tell you about puppet. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. Thursday, November 28, 13
  39. 39. What nobody will tell you about puppet. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. Thursday, November 28, 13
  40. 40. What nobody will tell you about puppet. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. • Read puppet documentation on every version. Thursday, November 28, 13
  41. 41. What nobody will tell you about puppet. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. • Read puppet documentation on every version. • Be careful about dependencies. Thursday, November 28, 13
  42. 42. What nobody will tell you about puppet. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. • Read puppet documentation on every version. • Be careful about dependencies. • Once you start to puppetize you become an Addict. Thursday, November 28, 13
  43. 43. Questions? at 3scale we’re hiring http://3scale.net/jobs Rhommel Lamas Twitter: @rhoml Thursday, November 28, 13
  44. 44. Thank you. at 3scale we’re hiring http://3scale.net/jobs Rhommel Lamas Twitter: @rhoml Thursday, November 28, 13
  45. 45. Thursday, November 28, 13
  46. 46. Useful links Thursday, November 28, 13
  47. 47. Useful links • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records Thursday, November 28, 13
  48. 48. Useful links • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html Thursday, November 28, 13
  49. 49. Useful links • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html • http://puppetlabs.com/blog/git-workflow-and-puppet-environments Thursday, November 28, 13
  50. 50. Useful links • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html • http://puppetlabs.com/blog/git-workflow-and-puppet-environments • http://projects.puppetlabs.com/projects/1/wiki/using_unicorn Thursday, November 28, 13
  51. 51. Useful links • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html • http://puppetlabs.com/blog/git-workflow-and-puppet-environments • http://projects.puppetlabs.com/projects/1/wiki/using_unicorn • https://github.com/cernops/puppetdb_foreman Thursday, November 28, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×