The Berkshelf Way

25,489 views

Published on

Jamie Winsor and a team of engineers created Berkshelf to help take the sting out of Chef’s learning curve. After encountering numerous challenges while developing Chef cookbooks, Jamie was inspired to create a tool based on criteria that’d be important for a developer’s productivity. Berkshelf, like Rebar, Go, or Mix, is a source code management tool.

Published in: Technology, Self Improvement

The Berkshelf Way

  1. 1. JAMIE WINSOR| SOFTWARE ENGINEERTHE BERKSHELF WAY
  2. 2. LEAGUE OF LEGENDS
  3. 3. ME IN20092009201020112012
  4. 4. 2009201020112012MY CHEFREPO IN2010
  5. 5. 2009201020112012AND IN2011…
  6. 6. 2009201020112012…FROM2012AND ON
  7. 7. WHAT’SBERKSHELF?‣  A COMMAND LINETOOL‣  A SOURCE CODEMANAGEMENT TOOL▾  Similar to ‘Mix’, ‘Go’, or‘Leiningen’‣  A PACKAGE MANAGER▾  Similar to ‘Gem’, ‘Apt’, or ‘Yum’‣  REPLACES PORTIONSOF KNIFE▾  Cookbook Generator, CookbookUploader, Cookbook Downloader
  8. 8. WHY IS IT CALLED BERKSHELF?
  9. 9. INTRO TOCOOKBOOKS:60
  10. 10. COOKBOOKS ARE FED INTO CHEFNGINXCOOKBOOK1.2.3 Chef-Client /Chef-Solo
  11. 11. RECIPES GO IN COOKBOOKSNGINXCOOKBOOK1.2.3RECIPERECIPERECIPE
  12. 12. RECIPES CONTAIN RESOURCECOLLECTIONSDEFAULTPACKAGE NGINXSERVICE NGINXREPOYUM_REPOSITORYNGINXYUM_KEY NGINXNGINXCOOKBOOK1.2.3
  13. 13. EXAMPLE RESOURCE123package “nginx” doaction :installendPACKAGE NGINX
  14. 14. INTRO TOBERKSHELF5min
  15. 15. INSTALL BERKSHELF$ gem install berkshelf Successfully installed berkshelf-1.4.0 1 gem installed
  16. 16. CREATE A NEW COOKBOOK$ berks cookbook myface create myface/files/default create myface/templates/default create myface/attributes create myface/definitions create myface/libraries create myface/providers create myface/recipes create myface/resources create myface/recipes/default.rb create myface/metadata.rb create myface/LICENSE create myface/README.md create myface/Berksfile create myface/chefignore create myface/.gitignore run git init from "./myface" create myface/Gemfile create myface/Vagrantfile Using myface (0.1.0) at path: /Users/reset/code/riot-cookbooks/myface
  17. 17. CONVERT AN EXISTING COOKBOOK$ berks init create Berksfile create chefignore create .gitignore run git init from "." create Gemfile create Vagrantfile Using old_cookbook (0.1.0) at path: /Users/reset/code/riot-cookbooks/old_cookbook Successfully initialized
  18. 18. STUCK? ASK FOR HELP!$ berks cookbook –h Usage: berks cookbook NAME Options: [--foodcritic] # Creates a Thorfile with Foodcritic support to lint test your cookbook [--chef-minitest] # Creates chef-minitest support files and directories, adds minitest-handler … [--scmversion] # Creates a Thorfile with SCMVersion support to manage versions for continuousintegration -L, [--license=LICENSE] # License for cookbook (apachev2, gplv2, gplv3, mit, reserved) # Default: reserved -m, [--maintainer=MAINTAINER] # Name of cookbook maintainer -e, [--maintainer-email=MAINTAINER_EMAIL] # Email address of cookbook maintainer [--no-bundler] # Skips generation of a Gemfile and other Bundler specific support [--skip-vagrant] # Skips adding a Vagrantfile and adding supporting gems to the Gemfile [--skip-git] # Skips adding a .gitignore and running git init in the cookbook directory -c, [--config=PATH] # Path to Berkshelf configuration to use. -F, [--format=FORMAT] # Output format to use. # Default: human -q, [--quiet] # Silence all informational output. -d, [--debug] # Output debug information Create a skeleton for a new cookbook
  19. 19. A COMPLETE EXAMPLECHEF-SERVERCOOKBOOK1.2.3"## Berksfile "## CHANGELOG.md "## CONTRIBUTING.md "## Gemfile "## LICENSE "## README.md "## Thorfile "## Vagrantfile "## attributes $   &## default.rb "## chefignore "## libraries $   "## dev_helper.rb $   &## omnitruck_client.rb "## metadata.rb "## recipes $   "## default.rb $   &## dev.rb &## templates &## default &## chef-server.rb.erb Berksfile metadata.rb
  20. 20. METADATA IS IMPORTANTCOOKBOOK
  21. 21. COOKBOOKMETADATA IS IMPORTANTCHEF-SERVERTHE NAME OF THECOOKBOOK
  22. 22. COOKBOOKCHEF-SERVERMETADATA IS IMPORTANTV2.0.0 THE VERSION OF THECOOKBOOKFOLLOW SEMVER: HTTP://SEMVER.ORG/
  23. 23. METADATA IS IMPORTANTCOOKBOOKYUMV1.2.3COOKBOOKCHEF-SERVERV1.2.3COOKBOOKAPTV1.2.3Red Hat CentOS UbuntuSET YOUR SUPPORTED PLATFORMSCentOS UbuntuRed Hat
  24. 24. METADATA IS IMPORTANTCOOKBOOKYUMV1.2.3SET WHAT COOKBOOKS YOU DEPEND ONCOOKBOOKCHEF-SERVERV1.2.3COOKBOOKAPTV1.2.3SUPPORTEDPLATFORMSSUPPORTEDPLATFORMSSUPPORTEDPLATFORMS
  25. 25. CHEF-SERVER COOKBOOK’S METADATAname "chef-server" maintainer "Opscode, Inc." maintainer_email "cookbooks@opscode.com" license "Apache 2.0" description "Installs and configures Chef Server" long_description IO.read(File.join(File.dirname(__FILE__), README.md)) version "2.0.0" depends "apt", ">= 1.7.0" depends "yum", ">= 0.5.0" %w{ ubuntu redhat centos fedora amazon scientific oracle }.each do |os| supports os end
  26. 26. IT STARTS WITH A BERKSFILE"## Berksfile "## CHANGELOG.md "## CONTRIBUTING.md "## Gemfile "## LICENSE "## README.md "## Thorfile "## Vagrantfile "## attributes $   &## default.rb "## chefignore "## libraries $   "## dev_helper.rb $   &## omnitruck_client.rb "## metadata.rb "## recipes $   "## default.rb $   &## dev.rb &## templates &## default &## chef-server.rb.erb Berksfile site :opscode metadata group :dev do cookbook git end Berksfile  
  27. 27. INSTALLING COOKBOOKSUsing chef-server (2.0.0) at path: /Users/reset/code/riot-cookbooks/chef-server Installing git (2.5.0) from site: http://cookbooks.opscode.com/api/v1/cookbooks Installing apt (1.9.2) from site: http://cookbooks.opscode.com/api/v1/cookbooks Installing yum (2.2.0) from site: http://cookbooks.opscode.com/api/v1/cookbooks Installing dmg (1.1.0) from site: http://cookbooks.opscode.com/api/v1/cookbooks Installing build-essential (1.4.0) from site: http://cookbooks.opscode.com/api/v1/cookbooks Installing windows (1.8.8) from site: http://cookbooks.opscode.com/api/v1/cookbooks Installing chef_handler (1.1.4) from site: http://cookbooks.opscode.com/api/v1/cookbooks Installing runit (1.1.2) from site: http://cookbooks.opscode.com/api/v1/cookbooks $ berks install
  28. 28. INSTALLED TO WHERE?$ ls ~/.berkshelf/cookbooks apache2-1.6.0 chef_handler-1.1.4 known_host-0.1.3 nginx-1.3.0 riot_chef-client-2.0.7 tsf4g-0.0.14 apt-1.9.0 couchdb-2.4.0 liquibase-0.0.1 not_system_ruby-0.1.7 riot_coherence-0.10.12 ucspi-tcp-1.0.0 ark-0.0.11 cron-1.0.4 logrotate-0.8.2 ntp-1.1.8 riot_ejabberd-0.0.152 ulimit-0.1.3 artifact-0.10.10 daemontools-1.0.0 logrotate-1.1.0 ohai-1.1.6 riot_ejabberd-0.0.188 users-1.1.4 artifact-0.10.2 database-1.3.12 lvm-0.8.6 openssl-1.0.0 riot_geoip-0.1.0 vim-1.0.2 artifact-1.2.0 database-1.3.4 magic_shell-0.2.0 postgresql-1.0.0 riot_java-0.1.4 windows-1.8.2 artifact-1.3.1 dmg-1.1.0 man-0.7.0 pvpnet_core-2.3.123 riot_java-0.1.8 xfs-1.0.0 artifact-1.4.0 editor-0.2.0 mysql-1.2.4 pvpnet_core-2.3.131 riot_mysql-2.0.1 xml-1.1.2 aws-0.100.0 erlang-1.2.0 mysql-1.3.0 python-1.2.2 riot_rtview-1.0.1 yum-2.1.0 bluepill-1.1.0 gecode-2.0.0 mysql-2.1.0 rbenv-1.3.2 riot_team-0.9.41 zlib-2.0.0 build-essential-1.1.2 git-2.2.0 mysql-2.1.2 riot_activemq-0.9.28 riot_tomcat-1.0.18 build-essential-1.3.4 hornetq-0.0.27 networking_basic-0.0.5 riot_base-2.7.32 runit-0.16.2 chef-client-2.1.4 java-1.5.0 nexus-1.1.0 riot_base-2.7.37 runit-1.0.4 chef-server-1.1.0 java-1.9.0 nginx-1.2.0 riot_base-2.7.49 sudo-1.2.0
  29. 29. UPLOADING COOKBOOKSUsing chef-server (2.0.0) at path: /Users/reset/code/riot-cookbooks/chef-server Using git (2.2.0) Using apt (1.9.0) Using yum (2.1.0) Using dmg (1.1.0) Using build-essential (1.3.4) Using windows (1.8.2) Using chef_handler (1.1.4) Using runit (0.16.2) Uploading chef-server (2.0.0) to: https://api.opscode.com:443/organizations/vialstudios Uploading git (2.2.0) to: https://api.opscode.com:443/organizations/vialstudios Uploading apt (1.9.0) to: https://api.opscode.com:443/organizations/vialstudios Uploading yum (2.1.0) to: https://api.opscode.com:443/organizations/vialstudios Uploading dmg (1.1.0) to: https://api.opscode.com:443/organizations/vialstudios Uploading build-essential (1.3.4) to: https://api.opscode.com:443/organizations/vialstudios Uploading windows (1.8.2) to: https://api.opscode.com:443/organizations/vialstudios Uploading chef_handler (1.1.4) to: https://api.opscode.com:443/organizations/vialstudios Uploading runit (0.16.2) to: https://api.opscode.com:443/organizations/vialstudios $ berks upload
  30. 30. THEBERKSHELF WAY
  31. 31. WORK INVERTICALSFROM THEOUTSIDE-INTHEBERKSHELFWAY
  32. 32. CREATE ACOOKBOOKFOR YOURAPPLICATIONCOOKBOOK
  33. 33. SERVICES ARE MADE UP OFCOMPONENTS1  app servers2  cache servers3  database servers4  background workers5  load balancers6  etc
  34. 34. APPLY A DESIGN PATTERN
  35. 35. APPLICATIONCOOKBOOKPATTERNCOOKBOOK
  36. 36. COOKBOOKMYFACE
  37. 37. COMPONENTS OF MYFACE1  load balancer2  application server proxy3  caching service4  application server5  background worker6  database server
  38. 38. MAPCOMPONENTSTO RECIPESRECIPERECIPERECIPERECIPE
  39. 39. RECIPES OF MYFACEmyface::defaultmyface::load_balancermyface::app_proxymyface::app_servermyface::cache_servermyface::worker_poolmyface::database_server
  40. 40. CRACK OPENMYFACE::DATABASE_SERVERnode.set[:mysql][:port] = 3309node.set[:mysql][:tunable][:max_connections] = 1000include_recipe "myface::_common_system"include_recipe "mysql::server"
  41. 41. PUBLIC/PRIVATE RECIPESPUBLIC RECIPE‣  What you put in therun_list of a node‣  Documented in theREADME‣  Documented in themetadataPRIVATE RECIPE‣  Not exposed to end user‣  Should never be put inthe run_list of a node‣  Always included intoother recipes‣  Documented in the codefor other developers
  42. 42. WHAT’S INMYFACE::_COMMON_SYSTEM?include_recipe "ntp::default"# . . . MAYBE SETUP SOME USERS . . .# . . . MAYBE SETUP SOME ADDITIONAL REPOS . . .
  43. 43. WORK INVERTICALSFROM THEOUTSIDE-INFAVOR DATA-DRIVENCOOKBOOKSTHEBERKSHELFWAY
  44. 44. COOKBOOKCHANGE ACOOKBOOKSBEHAVIOR ATRUNTIME
  45. 45. TUNABLES OF MYFACE1  listen on a different port or bind address2  set memory usage of workers3  any app config4  set services to started or stopped states
  46. 46. 3 WAYS TO CONFIGUREATTRIBUTESDATA BAGSENCRYPTED DATA BAGS
  47. 47. FAVOR CONFIGURING WITH ATTRIBUTES‣  Path of least resistance for cookbook consumers‣  Provide sensible defaults in the Attribute files▾  Consider creating an attribute file for each recipe‣  Can configure your application on an environmentlevel
  48. 48. WHEN TO USE DATA BAGSUSERSGROUPSYUM/APTTHINGS CONFIGURED BY "BASE"COOKBOOK (IF YOU HAVE ONE)ORGANIZATION LEVELCONFIGURATIONS
  49. 49. USE ENCRYPTED DATA BAGSWHEN STORING SENSITIVEDATA
  50. 50. WORK INVERTICALSFROM THEOUTSIDE-INFAVOR DATA-DRIVENCOOKBOOKSWRITE WELL-ENCAPSULATEDCOOKBOOKSTHEBERKSHELFWAY
  51. 51. NOT VERSIONEDCAN’T BE PACKAGED / DISTRIBUTEDNOT NAMESPACEDSTAHP USING ROLESTHEY ARE ORGANIZATION LEVEL DATAADDITIONAL SETUP COMPLEXITY FORTHE USER
  52. 52. USE DATA BAGS WHERE THEY MAKESENSEDO YOU NEED TO STOREORGANIZATION-WIDE DATA?DO YOU NEED TO STORESENSITIVE DATA?Use oneUse an encrypted one
  53. 53. USE DATA BAGS WHERE THEY MAKESENSEMOST IMPORTANTLY…VALIDATE YOUR DATABAGS!
  54. 54. 5  include_recipe "myface::app_proxy"6  include_recipe "myface::app_server"7  include_recipe "myface::cache_server"8  include_recipe "myface::worker_pool"9  include_recipe "myface::database"DEFAULTS RECIPE IS SPECIAL
  55. 55. LIGHTWEIGHT RESOURCES ANDPROVIDERS (LWRP)LIBRARIESDEFINITIONSCOOKBOOKS ARE MORE THAN RECIPESCOOKBOOKS DON’TNEED TO HAVE RECIPES
  56. 56. COOKBOOK LIBRARYCOOKBOOKPATTERN
  57. 57. LIBRARY COOKBOOKS‣  Might have recipes, might not‣  Contain LWRPs, Definitions, or Libraries useful toother cookbook authors‣  YUM Cookbook is a perfect example▾  Contains a yum_key LWRP for adding repository keys to yum▾  Contains a yum_repository LWRP for adding repositories to yum▾  Contains some recipes to configure commonly used public yumrepositories on your machine
  58. 58. COOKBOOK WRAPPERCOOKBOOKPATTERN
  59. 59. WRAPPER COOKBOOKS‣  Super lightweight‣  Contain recipes‣  Contain attribute overrides‣  Riot-Java is a perfect example▾  Contains a Oracle Java 6 recipe▾  Contains a Oracle Java 7 recipe▾  An abstraction on top of the Java cookbook for Riot engineers▾  Overrides the default “install flavor” for Java▾  Overrides the default location of the Java artifacts to an internallocation▾  NOT A FORK OF THE JAVA COOKBOOK
  60. 60. WORK INVERTICALSFROM THEOUTSIDE-INFAVOR DATA-DRIVENCOOKBOOKSWRITE WELL-ENCAPSULATEDCOOKBOOKSHAVE SHORTITERATIONLOOPSTHEBERKSHELFWAY
  61. 61. OLD WORKFLOWEDIT  COOKBOOK  UPLOAD  COOKBOOK  PROVISION  MACHINE  BOOTSTRAP  MACHINE  RUN  CHEF  CLIENT  SSH  AND  VALIDATE  5-30minutes
  62. 62. NEW WORKFLOWEDIT  COOKBOOK  VAGRANT  UP  /  VAGRANT  PROVISION  TEST  /  SSH  AND  VALIDATE  5  seconds  –  1  minute  
  63. 63. BERKSHELF VAGRANT PLUG-IN$ vagrant plugin install vagrant-berkshelf Installing the vagrant-berkshelf plugin. This can take a few minutes... Installed the plugin vagrant-berkshelf (1.2.0)!
  64. 64. Bringing machine default up with virtualbox provider... [Berkshelf] Updating Vagrants berkshelf: /Users/reset/.berkshelf/vagrant/berkshelf-20130424-40671-t43soz’ Generating chef JSON and uploading... Running chef-solo... [2013-04-25T03:01:31+00:00] INFO: *** Chef 10.14.2 *** [2013-04-25T03:01:32+00:00] INFO: Setting the run_list to ["recipe[minitest-handler::default]", "recipe[myface::default]"] from JSON [2013-04-25T03:01:32+00:00] INFO: Starting Chef Run for myface-berkshelf #### CHEF RUN GOES HERE #### [2013-04-25T03:02:10+00:00] INFO: Chef Run complete in 38.330025825 seconds [2013-04-25T03:02:10+00:00] INFO: Running report handlers Run options: -v --seed 41224 # Running tests: minitesthandler::default#test_0001_runs_no_tests = 0.00 s = . myface::default#test_0001_runs_no_tests_by_default = 0.00 s = . Finished tests in 0.002425s, 824.7576 tests/s, 0.0000 assertions/s. 2 tests, 0 assertions, 0 failures, 0 errors, 0 skips [2013-04-25T03:02:10+00:00] INFO: Report handlers complete PROVISION NEW ENVIRONMENT INMINUTES$ vagrant up 1 minute 20 secon
  65. 65. Running chef-solo... [2013-04-25T03:01:31+00:00] INFO: *** Chef 10.14.2 *** [2013-04-25T03:01:32+00:00] INFO: Setting the run_list to ["recipe[minitest-handler::default]", "recipe[myface::default]"] from JSON [2013-04-25T03:01:32+00:00] INFO: Starting Chef Run for myface-berkshelf #### CHEF RUN GOES HERE #### [2013-04-25T03:02:10+00:00] INFO: Chef Run complete in 0.520326234 seconds [2013-04-25T03:02:10+00:00] INFO: Running report handlers Run options: -v --seed 41224 # Running tests: minitesthandler::default#test_0001_runs_no_tests = 0.00 s = . myface::default#test_0001_runs_no_tests_by_default = 0.00 s = . Finished tests in 0.002425s, 824.7576 tests/s, 0.0000 assertions/s. 2 tests, 0 assertions, 0 failures, 0 errors, 0 skips [2013-04-25T03:02:10+00:00] INFO: Report handlers complete CONVERGE IN SECONDS$ vagrant provision 7 seconds
  66. 66. WORK INVERTICALSFROM THEOUTSIDEFAVOR DATA-DRIVENCOOKBOOKSWRITE WELL-ENCAPSULATEDCOOKBOOKSHAVE SHORTITERATIONLOOPSTHEBERKSHELFWAY
  67. 67. LEARNMORE ATBERKSHELF.COM
  68. 68. CONTRIBUTEGITHUB.COM/RIOTGAMES/BERKSHELFHOW CAN YOU HELP?OR JOIN USWWW.RIOTGAMES.COM/CAREERS
  69. 69. CALL MEMAYBE?Jamie Winsor@resetexistencegithub.com/reset

×