November 2013
Puppet at scale
Rhommel Lamas
Thursday, November 28, 13
Who am I?
Thursday, November 28, 13
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Who am I?
Th...
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Who am I?
Co...
Scale experience
•Physical servers Xtratelecom.es.
•AWS and other cloud providers at Wuaki.tv and
3scale.net.
Who am I?
Co...
What is 3scale?
3scale provides tools such as traffic
management, API administration, traffic
reports, and more, to help com...
What does scaling stands for?
Thursday, November 28, 13
Scalability is the ability of a system, network,
or process to handle a growing amount of
work in a capable manner or its ...
What are our scaling needs?
Thursday, November 28, 13
• How many people do we need to scale our
system infrastructure?
What are our scaling needs?
Thursday, November 28, 13
• How many people do we need to scale our
system infrastructure?
• How many environments and systems can
our team maintain...
• How many people do we need to scale our
system infrastructure?
• How many environments and systems can
our team maintain...
Puppet dynamic environments
Thursday, November 28, 13
• Purpose of Puppet environments.
Puppet dynamic environments
Thursday, November 28, 13
• Purpose of Puppet environments.
• Puppet base environments at 3scale.
Puppet dynamic environments
Thursday, November 28,...
/etc/puppet
auth.conf
autosign.conf
config.ru
hiera.yaml
hipchat.yaml
fileserver.conf
unicorn.rb
environments
Puppet environ...
[main]
server = puppet.example.com
environment = production
[master]
environment = production
manifest = $confdir/environm...
[main]
server = puppet.example.com
environment = production
[master]
environment = production
manifest = $confdir/environm...
:hierarchy:
- '%{::osfamily}'
- '%{::environment}'
- common
:backends:
- yaml
- json
:yaml:
:datadir: "/etc/puppet/environ...
Thursday, November 28, 13
• Purpose of Puppet environments.
• Puppet environments at 3scale.
• Puppet development workflow (Git).
• Puppet-lint
• Pup...
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Puppet at 3scale.
Thursday, November 28, 13
• Since July 2013 we are running Puppet 3.0+.
Puppet at 3scale.
Thursday, November 28, 13
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
Puppet at 3scale.
Thursday, November 28, 13
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
Puppet at 3scale.
Thursday, November...
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit...
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit...
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit...
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit...
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
• Nginx + Unicorn.
• 15 minutes splay with a splaylimit...
ROUTE
53
_x-puppet._tcp.3scale.net
_x-puppet-ca._tcp.3scale.net
Puppet
Master
+
RabbitMQ
Cluster
+
MCollective
Client
+
Pu...
Thursday, November 28, 13
What nobody will tell you about
puppet.
Thursday, November 28, 13
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
What nobody will tell you about
p...
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find ...
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find ...
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find ...
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find ...
• Puppet-lint is your friend and it will help others to
understand your code, so USE IT.
• Puppet-rspec will help you find ...
Rhommel Lamas
Twitter: @rhoml
Questions?
at 3scale
we’re hiring
http://3scale.net/jobs
Thursday, November 28, 13
Rhommel Lamas
Twitter: @rhoml
Thank you.
at 3scale
we’re hiring
http://3scale.net/jobs
Thursday, November 28, 13
Thursday, November 28, 13
Useful links
Thursday, November 28, 13
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
Useful links
Thursday, November...
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/pu...
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/pu...
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/pu...
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
• http://docs.puppetlabs.com/pu...
Upcoming SlideShare
Loading in …5
×

Building scalable applications while scaling your infrastructure by rhommel lamas

853 views

Published on

"Building scalable applications while scaling your infrastructure" presented at Puppet Camp Munich 2013 by Rhommel Lamas 3scale Networks

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
853
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
29
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Building scalable applications while scaling your infrastructure by rhommel lamas

  1. 1. November 2013 Puppet at scale Rhommel Lamas Thursday, November 28, 13
  2. 2. Who am I? Thursday, November 28, 13
  3. 3. Scale experience •Physical servers Xtratelecom.es. •AWS and other cloud providers at Wuaki.tv and 3scale.net. Who am I? Thursday, November 28, 13
  4. 4. Scale experience •Physical servers Xtratelecom.es. •AWS and other cloud providers at Wuaki.tv and 3scale.net. Who am I? Config Management • Worked with CFEngine2 for 2 years. • Puppet user since 2009. Thursday, November 28, 13
  5. 5. Scale experience •Physical servers Xtratelecom.es. •AWS and other cloud providers at Wuaki.tv and 3scale.net. Who am I? Config Management • Worked with CFEngine2 for 2 years. • Puppet user since 2009. Obsessed about config management and I use Emacs. Thursday, November 28, 13
  6. 6. What is 3scale? 3scale provides tools such as traffic management, API administration, traffic reports, and more, to help companies publicize their APIs easily and power new types of connected applications. Thursday, November 28, 13
  7. 7. What does scaling stands for? Thursday, November 28, 13
  8. 8. Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth. Wikipedia What does scaling stands for? Thursday, November 28, 13
  9. 9. What are our scaling needs? Thursday, November 28, 13
  10. 10. • How many people do we need to scale our system infrastructure? What are our scaling needs? Thursday, November 28, 13
  11. 11. • How many people do we need to scale our system infrastructure? • How many environments and systems can our team maintain? What are our scaling needs? Thursday, November 28, 13
  12. 12. • How many people do we need to scale our system infrastructure? • How many environments and systems can our team maintain? • Can we delegate configurations easily to our team members? What are our scaling needs? Thursday, November 28, 13
  13. 13. Puppet dynamic environments Thursday, November 28, 13
  14. 14. • Purpose of Puppet environments. Puppet dynamic environments Thursday, November 28, 13
  15. 15. • Purpose of Puppet environments. • Puppet base environments at 3scale. Puppet dynamic environments Thursday, November 28, 13
  16. 16. /etc/puppet auth.conf autosign.conf config.ru hiera.yaml hipchat.yaml fileserver.conf unicorn.rb environments Puppet environments at 3scale environments production manifests production.pp preview.pp nodes production webserver.pp preview webserver.pp modules nginx hieradata common.yaml production.yaml preview.yaml Capfile preview staging Thursday, November 28, 13
  17. 17. [main] server = puppet.example.com environment = production [master] environment = production manifest = $confdir/environments/$environment/manifests/site.pp modulepath = $confdir/environments/$environment/modules Thursday, November 28, 13
  18. 18. [main] server = puppet.example.com environment = production [master] environment = production manifest = $confdir/environments/$environment/manifests/$environment.pp modulepath = $confdir/environments/$environment/modules # production.pp import 'nodes/production/nodes.pp' import 'nodes/production/*.pp Thursday, November 28, 13
  19. 19. :hierarchy: - '%{::osfamily}' - '%{::environment}' - common :backends: - yaml - json :yaml: :datadir: "/etc/puppet/environments/%{environment}/hieradata" Thursday, November 28, 13
  20. 20. Thursday, November 28, 13
  21. 21. • Purpose of Puppet environments. • Puppet environments at 3scale. • Puppet development workflow (Git). • Puppet-lint • Puppet-rspec • Code Review Puppet dynamic environments Thursday, November 28, 13
  22. 22. Thursday, November 28, 13
  23. 23. Thursday, November 28, 13
  24. 24. Thursday, November 28, 13
  25. 25. Puppet at 3scale. Thursday, November 28, 13
  26. 26. • Since July 2013 we are running Puppet 3.0+. Puppet at 3scale. Thursday, November 28, 13
  27. 27. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. Puppet at 3scale. Thursday, November 28, 13
  28. 28. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. Puppet at 3scale. Thursday, November 28, 13
  29. 29. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. Puppet at 3scale. Thursday, November 28, 13
  30. 30. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. Puppet at 3scale. Thursday, November 28, 13
  31. 31. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. • The foreman as our ENC. Puppet at 3scale. Thursday, November 28, 13
  32. 32. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. • The foreman as our ENC. • Puppetdb and hiera. Puppet at 3scale. Thursday, November 28, 13
  33. 33. • Since July 2013 we are running Puppet 3.0+. • 2 Puppet master’s. • Nginx + Unicorn. • 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records. • The foreman as our ENC. • Puppetdb and hiera. •MCollective with Rabbitmq connector. Puppet at 3scale. Thursday, November 28, 13
  34. 34. ROUTE 53 _x-puppet._tcp.3scale.net _x-puppet-ca._tcp.3scale.net Puppet Master + RabbitMQ Cluster + MCollective Client + PuppetDB + The Foreman RabbitMQ Load Balancer PostgreSQL RDS MultiAZ Puppet Master + RabbitMQ Cluster + MCollective Client + PuppetDB + The Foreman Puppet Master + RabbitMQ Cluster + MCollective Client + PuppetDB + The Foreman Puppet Agent + Mcollective Server Puppet Agent + Mcollective Server Puppet Agent + Mcollective Server Thursday, November 28, 13
  35. 35. Thursday, November 28, 13
  36. 36. What nobody will tell you about puppet. Thursday, November 28, 13
  37. 37. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. What nobody will tell you about puppet. Thursday, November 28, 13
  38. 38. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. What nobody will tell you about puppet. Thursday, November 28, 13
  39. 39. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. What nobody will tell you about puppet. Thursday, November 28, 13
  40. 40. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. • Read puppet documentation on every version. What nobody will tell you about puppet. Thursday, November 28, 13
  41. 41. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. • Read puppet documentation on every version. • Be careful about dependencies. What nobody will tell you about puppet. Thursday, November 28, 13
  42. 42. • Puppet-lint is your friend and it will help others to understand your code, so USE IT. • Puppet-rspec will help you find errors on early stages so TEST your modules. • Don’t code remotely on your servers, use your local development environment and git. • Read puppet documentation on every version. • Be careful about dependencies. • Once you start to puppetize you become an Addict. What nobody will tell you about puppet. Thursday, November 28, 13
  43. 43. Rhommel Lamas Twitter: @rhoml Questions? at 3scale we’re hiring http://3scale.net/jobs Thursday, November 28, 13
  44. 44. Rhommel Lamas Twitter: @rhoml Thank you. at 3scale we’re hiring http://3scale.net/jobs Thursday, November 28, 13
  45. 45. Thursday, November 28, 13
  46. 46. Useful links Thursday, November 28, 13
  47. 47. • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records Useful links Thursday, November 28, 13
  48. 48. • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html Useful links Thursday, November 28, 13
  49. 49. • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html • http://puppetlabs.com/blog/git-workflow-and-puppet-environments Useful links Thursday, November 28, 13
  50. 50. • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html • http://puppetlabs.com/blog/git-workflow-and-puppet-environments • http://projects.puppetlabs.com/projects/1/wiki/using_unicorn Useful links Thursday, November 28, 13
  51. 51. • http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records • http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html • http://puppetlabs.com/blog/git-workflow-and-puppet-environments • http://projects.puppetlabs.com/projects/1/wiki/using_unicorn • https://github.com/cernops/puppetdb_foreman Useful links Thursday, November 28, 13

×