Cooking with Chef, Knife, Vagrant and Co

987 views

Published on

In this talk from SWE Days 2013 you will get a solid understanding of what “Infrastructure-As-Code” means. You will learn about tools like Chef, Vagrant and Librarian and their role in automating your infrastructure. Lots of concrete code examples and short demos are waiting for you. You WILL want to treat your infrastructure as code at the end of this talk! :-)

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
987
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
31
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Cooking with Chef, Knife, Vagrant and Co

  1. 1. © Zühlke 2013 Torben Knerr Cooking with Chef, Knife, Vagrant & Co 4 Ingredients for tasty “Infrastructure-As-Code” 27. September 2013 Slide 1
  2. 2. © Zühlke 2013 With typical components: • Application Servers • Database Servers • Caching Servers • Load Balancer / Reverse Proxy • Clustering & Failover A Typical Web App Project… Cooking with Chef, Knife, Vagrant & Co | Torben Knerr nginx Tomcat Tomcat DB Master DB Slave App App memcached 27. September 2013 Slide 2
  3. 3. © Zühlke 2013 With a typical Deployment Pipeline: Cooking with Chef, Knife, Vagrant & Co | Torben Knerr Local DEV TEST PROD A Typical Web App Project… 27. September 2013 Slide 3
  4. 4. © Zühlke 2013 With typical (?) problems: • All environments are somehow different • Nobody knows the differences in detail • Setting up a new environment takes weeks • And again it looks different from the others • Totally no Dev/Prod Parity • And last but not least… Cooking with Chef, Knife, Vagrant & Co | Torben Knerr A Typical Web App Project… 27. September 2013 Slide 5
  5. 5. © Zühlke 2013 First Approach: Hey, let’s build VMs! This already helps: • Use VMs for a consistent work environment • Manually set up a local developer VM with Tomcat, PostgreSQL, nginx, etc… • Carefully hand-crafted with love! • Distribute it to the team Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 7
  6. 6. © Zühlke 2013 But… • Manual management of VMs is cumbersome • How do you version a 40GB VM? • How do you take care of updates? • Can you diff the changes? • Once it works locally, how can you apply this to the rest of the deployment pipeline? • How do you parameterize such VMs? Better than nothing, but still far from perfect! Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 8 First Approach: Hey, let’s build VMs!
  7. 7. © Zühlke 2013 Why Scripting / Automation? • Lightweight scripts vs. fat VMs – Allows for proper versioning and diffing! – Easy to share • Infrastructure can be reliably recreated – Minimize manual (= boring & error prone) tasks – Basis for “clean” infrastructure deployments – Anywhere in the deployment pipeline • Explicit design of infrastructure – Described as an executable specification – Not formed “by accident” Treat your infrastructure like code! Better Solution: Automate your VMs! Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 9
  8. 8. © Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co Torben Knerr Welcome to the DevOps Kitchen! Infrastructure á la Carte 27. September 2013 Slide 10
  9. 9. © Zühlke 2013 Goal: set up a simple server infrastructure, fully automated • For example on a developer machine • Separate VMs for app server and database server • Anywhere reproducible („clean setup“) Our Example Scenario Cooking with Chef, Knife, Vagrant & Co | Torben Knerr App Server VM Tomcat Database Server VM PostgreSQL .war 27. September 2013 Slide 11
  10. 10. © Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co Torben Knerr Ingredient #1 We need VMs 27. September 2013 Slide 12
  11. 11. © Zühlke 2013 Ingredient #1: we need VMs Enter Vagrant What is Vagrant? • Vagrant is “automation for VMs” • Different VM providers: VirtualBox, aws, rackspace, etc.. • The configuration of VMs is described in a Vagrantfile • Simple commands for interacting with VMs: – vagrant up – vagrant ssh – vagrant halt – … • http://vagrantup.com Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 13
  12. 12. Ingredient #1: we need VMs – A simple Vagrantfile
  13. 13. Ingredient #1: we need VMs – A simple Vagrantfile Basebox
  14. 14. Ingredient #1: we need VMs – A simple Vagrantfile App Server VM
  15. 15. Ingredient #1: we need VMs – A simple Vagrantfile Database Server VM
  16. 16. Ingredient #1: we need VMs – Interacting with Vagrant VMs vagrant up
  17. 17. Ingredient #1: we need VMs – Interacting with Vagrant VMs vagrant ssh
  18. 18. © Zühlke 2013 Ingredient #1: we need VMs A quick look at the VirtualBox GUI Cooking with Chef, Knife, Vagrant & Co | Torben Knerr Is this the App Server VM? Or the Database Server VM? 27. September 2013 Slide 20
  19. 19. Ingredient #1: we need VMs – More VM configuration
  20. 20. Ingredient #1: we need VMs – More VM configuration
  21. 21. Ingredient #1: we need VMs – More VM configuration Name, CPUs, RAM, etc…
  22. 22. Ingredient #1: we need VMs – More VM configuration vagrant reload
  23. 23. © Zühlke 2013 Ingredient #1: we need VMs Looking at the VirtualBox GUI again Cooking with Chef, Knife, Vagrant & Co | Torben Knerr and the VM Parameters are applied too Better names  27. September 2013 Slide 25
  24. 24. © Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co Torben Knerr Ingredient #2: Provisioning 27. September 2013 Slide 26
  25. 25. © Zühlke 2013 Ingredient #2: Provisioning Back to our Example Scenario “Naked” VMs are not enough • On the database server VM we need PostgreSQL • On the app server VM we need Tomcat We need a tool (“provisioner”) for installing and configuring software components on top of naked VMs Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 27
  26. 26. © Zühlke 2013 Ingredient #2: Provisioning Enter Chef What is Chef? • Chef is a “Configuration Management” system • Installs & configures software on nodes • Provides a DSL for describing the configuration of a node in a readable and platform-independent way More information: • http://www.opscode.com/chef/ Vagrant supports provisioning of VMs via Chef out-of-the-box! Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 28
  27. 27. Ingredient #2: Provisioning – Installing PostgreSQL
  28. 28. Ingredient #2: Provisioning – Installing PostgreSQL run the PostgreSQL Server recipe
  29. 29. Ingredient #2: Provisioning – Installing PostgreSQL configuration parameters
  30. 30. vagrant up db
  31. 31. What’s inside the PostgreSQL Recipe?
  32. 32. install the “postgresql“ package
  33. 33. create the config file
  34. 34. Erzeuge die Konfig-Datei
  35. 35. create another config file
  36. 36. start the „postgresql“ service
  37. 37. we also need the postgresql client…
  38. 38. Ingredient #2: Provisioning – Installing Tomcat
  39. 39. Ingredient #2: Provisioning – Installing Tomcat run the Tomcat default recipe (short for “tomcat::default“)
  40. 40. Ingredient #2: Provisioning – Installing Tomcat tomcat specific configuration
  41. 41. vagrant up web
  42. 42. © Zühlke 2013 Chef Philosophy Specification of the target state • Idempotence • Convergence (“self-healing“) Chef DSL abstracts the OS-specific details via • Resources • Providers Naming: kitchen metaphor • Chef, Cookbook, Recipe, Knife, … Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 46
  43. 43. © Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co Torben Knerr Ingredient #3: Managing Cookbooks 27. September 2013 Slide 47
  44. 44. © Zühlke 2013 Ingredient #3: Managing Cookbooks Back to our Example Scenario Cooking with Chef, Knife, Vagrant & Co | Torben Knerr where is the PostgreSQL cookbook / recipe coming from??? 27. September 2013 Slide 48
  45. 45. © Zühlke 2013 Ingredient #3: Managing Cookbooks Typical Sources for “Community Cookbooks” Cooking with Chef, Knife, Vagrant & Co | Torben Knerr http://community.opscode.com/cookbooks/ https://github.com/search?q=cookbook 27. September 2013 Slide 49
  46. 46. © Zühlke 2013 Ingredient #3: Managing Cookbooks Enter Librarian What is Librarian? • Librarian is a Dependency Management tool for Cookbooks • Downloads and extracts cookbooks for you Dependencies are specified in a Cheffile • Transitive dependencies are resolved as well • Different cookbook sources and version constraints are respected More information: • https://github.com/applicationsonline/librarian-chef Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 50
  47. 47. © Zühlke 2013 Ingredient #3: Managing Cookbooks A simple Cheffile Cheffile Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 51
  48. 48. Ingredient #3: Managing Cookbooks – A simple Cheffile Librarian „installs“ the cookbooks
  49. 49. © Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co Torben Knerr Ingredient #4: Application Cookbooks 27. September 2013 Slide 53
  50. 50. © Zühlke 2013 Ingredient #4: Application Cookbooks Back to our Example Scenario Re-using Community Cookbooks is not enough • We need a specific database and database user for our application • We need to deploy our application’s .war file in Tomcat • Existing Community Cookbooks don’t “know” our specific requirements! We need to write our own “Application Cookbook” for this purpose! Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 54
  51. 51. © Zühlke 2013 Did you know? You can create cookbooks with a knife! Cooking with Chef, Knife, Vagrant & Co | Torben Knerr Ingredient #4: Application Cookbooks Enter Knife 27. September 2013 Slide 55
  52. 52. © Zühlke 2013 Ingredient #4: Application Cookbooks Structure of our “myapp” cookbook Single cookbook with separate recipes for setting up the database and webapp Goal: hide implementation details from the user Cooking with Chef, Knife, Vagrant & Co | Torben Knerr myapp attributes recipes db.rb web.rb db.rb default.rb web.rb 27. September 2013 Slide 56
  53. 53. © Zühlke 2013 Ingredient #4: Application Cookbooks Structure of our “myapp” cookbook Attributes files expose the “configuration parameters” of a cookbook and supply sensible defaults Cooking with Chef, Knife, Vagrant & Co | Torben Knerr myapp attributes recipes db.rb web.rb db.rb default.rb web.rb 27. September 2013 Slide 57
  54. 54. © Zühlke 2013 Ingredient #4: Application Cookbooks Structure of our “myapp” cookbook • myapp/attributes/db.rb • myapp/attributes/web.rb Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 58
  55. 55. © Zühlke 2013 Ingredient #4: Application Cookbooks Structure of our “myapp” cookbook The “db” recipe first installs PostgreSQL, then creates the database and database user Cooking with Chef, Knife, Vagrant & Co | Torben Knerr myapp attributes recipes db.rb web.rb db.rb default.rb web.rb 27. September 2013 Slide 59
  56. 56. first install the PostgreSQL server
  57. 57. installs prerequisites for interacting with PostgreSQL from within this recipe
  58. 58. creates the database user
  59. 59. creates the database
  60. 60. © Zühlke 2013 Ingredient #4: Application Cookbooks Use “myapp::db” in the Vagrantfile Vagrantfile Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 65
  61. 61. © Zühlke 2013 Ingredient #4: Application Cookbooks Use “myapp::db” in the Vagrantfile Vagrantfile Cooking with Chef, Knife, Vagrant & Co | Torben Knerr instead of “postgresql::server” 27. September 2013 Slide 66
  62. 62. vagrant provision db
  63. 63. © Zühlke 2013 Ingredient #4: Application Cookbooks Extended Cheffile Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 68
  64. 64. © Zühlke 2013 Ingredient #4: Application Cookbooks Structure of our “myapp” cookbook The “web” recipe first installs tomcat, then configures and deploys the our web app Cooking with Chef, Knife, Vagrant & Co | Torben Knerr myapp attributes recipes db.rb web.rb db.rb default.rb web.rb 27. September 2013 Slide 69
  65. 65. we need to install Tomcat first
  66. 66. there is our .war file located
  67. 67. database configuration (via context.xml)
  68. 68. deploy the webapp to Tomcat
  69. 69. © Zühlke 2013 Ingredient #4: Application Cookbooks Use “myapp::web” in the Vagrantfile Vagrantfile Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 75
  70. 70. © Zühlke 2013 Ingredient #4: Application Cookbooks Use “myapp::web” in the Vagrantfile Vagrantfile Cooking with Chef, Knife, Vagrant & Co | Torben Knerr instead of the “tomcat::default“ recipe 27. September 2013 Slide 76
  71. 71. © Zühlke 2013 Ingredient #4: Application Cookbooks Use “myapp::web” in the Vagrantfile Vagrantfile Cooking with Chef, Knife, Vagrant & Co | Torben Knerr C connect to this database host (here: ip address of the db vm) 27. September 2013 Slide 77
  72. 72. vagrant provision web
  73. 73. Is it running now?
  74. 74. Or whatever is in your context… (we have borrowed the “yabe“ example from Play! here: http://www.playframework.org/documentation/1.0/samples)
  75. 75. © Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co Torben Knerr The DevOps Kitchen – Recap 27. September 2013 Slide 81
  76. 76. © Zühlke 2013 Recap: Ingredients for cooking in the DevOps Kitchen Cooking with Chef, Knife, Vagrant & Co | Torben Knerr Ingr. #1 We need VMs Ingr. #2 Provisioning Ingr. #3 Cookbook Management Ingr. #4 Application Cookbooks + + + 27. September 2013 Slide 82
  77. 77. © Zühlke 2013 Fully Automated Infrastructure via Textual Specification • No shell scripts, but specification of the target state via Chef DSL • Crisp and concise, but not cryptic • SCM-friendly: Kilobytes instead of Gigabytes, diffable • Complex infrastructure available at the push-of-a-button • Reproducible anywhere in the deployment pipeline • Good software engineering principles applied to infrastructure What have we achieved? Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 83
  78. 78. © Zühlke 2013 We have barely touched the surface… There’s much more of Chef, Vagrant & Co we have not seen today: • More Vagrant features – Alternative Providers (e.g. vmware, aws, rackspace, managed-servers, etc…) – Alternative Provisioners (e.g. puppet, shell, ansible, salt, etc…) – Vagrant Plugins (e.g. vagrant-omnibus, vagrant-berkshelf, vagrant-cachier, …) – … • More Chef concepts – Writing Custom Resources und Providers (LWRPs) – Data bags – Environments – Search – Chef Solo vs. Chef Server – … • And most importantly… Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 84
  79. 79. © Zühlke 2013Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 85 …if we treat infrastructure as code, then we have to test! Linting • foodcritic Unit-Level Testing • chefspec • fauxhai Integration Testing (from the inside) • chef-minitest-handler • test-kitchen Acceptance Testing (outside-in) • cucumber-nagios
  80. 80. © Zühlke 2013 Resources Some useful stuff to check out… • Code Examples from this talk https://github.com/tknerr/oop-chef-demo • Newer Example based on Vagrant / Chef / Berkshelf https://github.com/tknerr/sample-application-cookbook • Bills Kitchen – all you need for cooking with Chef & Vagrant on Windows https://github.com/tknerr/bills-kitchen https://ikm.zuehlke.com/topics/Pages/Bill%27s%20Kitchen.aspx • “Chef, Vagrant & Co” Yammer Group https://www.yammer.com/zuehlke.com/#/threads/inGroup?type=in_group&feedId=2051695 • “Test-Driven Infrastructure with Chef (2nd ed.)” – highly recommended! http://shop.oreilly.com/product/0636920030973.do Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 86
  81. 81. © Zühlke 2013 Resources More of the Vagrant Ecosystem… • Bindler – Manage your Vagrant Plugins via `plugins.json` https://github.com/fgrehm/bindler • Selection of useful Vagrant plugins https://github.com/schisamo/vagrant-omnibus – installs Chef on a “provisionerless” basebox https://github.com/riotgames/vagrant-berkshelf – resolves cookbook dependencies on `vagrant up` https://github.com/fgrehm/vagrant-cachier – transparently caches downloaded packages https://github.com/tmatilai/vagrant-proxyconf – sets the proxy inside the VM … • Selection of Vagrant Providers https://github.com/mitchellh/vagrant-aws – provision EC2 instances in the AWS cloud with Vagrant https://github.com/tknerr/vagrant-managed-servers – provision “non-VMs” with Vagrant https://github.com/fgrehm/vagrant-lxc - provision LXC containers … Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 87
  82. 82. © Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co Torben Knerr Discussion Time! 20. Juli 2012 Puppet, Berkshelf, Foo, Bar, Baz DevOps, NoOps, Blablabla Vagrant, Chef, Deployment Stuff Holy Cloud! Slide 88
  83. 83. Thank You!  And let’s continue the discussion here:

×