The Berkshelf Way
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,927
On Slideshare
1,927
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
19
Comments
0
Likes
6

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Hey everyone, I’m Jamie Winsor and I love video games. I’ve been playing for a while; I’ve been enamoured with video games since I was a little older than two. I distinctly remember wanting to learn how to read because games like Zelda were too difficult and I learned how to type so I would be able to communicate with people in Everquest more easily.In an open letter to my third grade teacher I told her I’d be a “video game maker” by 2013. Here it is.*NEXT*What’s up now Mrs. Hoey? I might not have made 1 million cartridges like I promised, but I’m working on it.I also apparently didn’t know what Thanksgiving was all about when I was in kindergarten*NEXT*But you didn’t come to hear about what my League of Legends ELO is, or about the time I camped Raster of Guk for 34 hours straight to get my Monk’s epic in Everquest. You’re probably here for something more Chef related.But really, my love for games is what brought me to this stage in the first place. Specifically my passion for persistent, always-on, online games like World of Warcraft, Everquest, or League of Legends, is why I am here today. You see, persistent online games have a unique problem to solve that has a lot more in common with other online services like Amazon, Facebook, or online banking software. They need to be available 24 hours a day and 7 days a week. And these games can get big.I work for a company in located in Santa Monica called Riot Games and we are the creators of one of my all time favorite games; League of Legends.
  • LoL is one of these always-on, online games. It came out in 2009 and since then has grown quite a bit. We are one of the most played games in the world with 32 million active monthly players, 12 million of which login and play every day and at a peak time during the day there is 5 million players playing at once.And players have high expectation for the availability of these online games. If my online banking software goes down (and it does ALL the time) I’ll just come back later. If I go home, crack open a beer and try to play some a game with some of my friends and IT’S down?*NEXT**PAUSE FOR LOLS**NEXT*I get mad and, I work on, these things and understand the space. Dedicated players have high expectations and that bar is continually getting raised. It’s no longer acceptable to have “launch day” problems or minimal content for players to experience. That newness factor of simply being able to get into a game and shoot a rocket at your friend has long worn off for a lot of us. We demand not only triple A content but also a triple A online service.
  • At Riot, we aspire to be the most player focused game company in the world. Everything we do at Riot is driven by us trying to do what is best for our players.For the last 4 years I have been on a quest to help build a reliable platform that software engineers and operations people alike could use to deploy and configure their software with hopes that the gaming community, my friends, and myself could enjoy higher uptimes and more rapid delivery of new content. And by using and contributing to the open source world, my dream might actually become a reality.
  • At Riot we are using Chef to bring league of legends to players around the world and it is the very core component to our publishing platform. And I love Chef. I love almost everything about Chef; the people making it, the software, the language it's written in, the community.But for as much as I love Chef, there’s a learning curve that goes with it.
  • And that curve can be steep. This particular graph represents the difficulty level of popular massively multiplayer online games. It’s the time spent playing versus the level of skill required as you go.Now Chef definitely isn’t this bad, but it doesn’t matter if you have a background in software engineering or a background in operations, it can be rough man.It's been about four years now since the first time I used Chef. I still remember my first time, “ahh back in 2009 if I recall correctly”
  • This was me. Sitting there wondering what the in the hell is a cookbook? The clever names of the Chef primitives, and just the sheer number of them that you will learn in your first hours of working with Chef can be daunting to a new user.Once I had a handle on the terminology I was well on my way to automating my infrastructure. I was creating cookbooks, roles, I think data bags came out somewhere in there and I was exploring those as well.I had built up quite the Chef repository and I was getting shit done.
  • This is what my Chef repository looked like when I had a “strong grasp” of Chef. For anyone who might not know what a “Chef repository is”; it’s a collection of all the things that you put in your Chef server and it’s generally stored using a source control manager like Git.My cookbooks were stored in the “site-cookbooks” directory, anything that I had downloaded from the community site was in the “cookbooks” directory and then any changes I made to those cookbooks were just committed right on top of the master branch. I had a pretty large repository with a lot of cookbooks and with the tangled mess of my own cookbooks and my changes to cookbooks I was given, it was nearly impossible to receive updates from those cookbooks that I had downloaded from the community site.I did a lot of experimenting to try and figure out how I could clean this up. I was really surprised that I was having such a hard time since this was the “recommended way” of doing things among the community.The next experiment that I ran was pretty much the last before I hit a good stride. I thought, “I know…”
  • …I’ll use GitSubmodules to maintain the cookbooks in my repository and clean that right up.This was a dark time. Git can be confusing to even the beardiest of technologists and while there’s nothing complicated about submodules on the surface, they can be a beast to maintain. There’s a sort of ritual that you’ve got to remember to perform to keep everything up to date.It wasn’t until I decided to stop keeping cookbooks inside of a Chef repository until things really started to take off
  • By not storing cookbooks in the chef repository, they were just more portable.I could package one up and give it to somebody else. I could take one from somebody else and use it in our infrastructure and it would just work. It also encouraged less long lived forks. Instead of treating the community NGINX cookbook as a basis for a “Riot NGINX” cookbook by creating a long lived fork of it in our Chef repository, we instead were helping to fix or make the NGINX community cookbook more generic so it was useful to more people.There were some rules that we had to establish to make sure these cookbooks would be portable and re-usable. We also needed to create a bit of tooling to make it all possible.
  • The tool we set out to build started as a spike, or an experiment.Initially we had a knife plugin that did the job of getting all of the dependencies that a Chef repository had.Things went in a very different direction once we proved that the spike was a good idea but it needed more of the team’s attention. We started to follow README driven development: This is when you write down exactly how you want to explain to somebody how to use your software before you actually write your software.Through this exercise we realized what we really needed to build was a mix between the rubygem’s “Gem” command and Bundler, for cookbooks. We eventually landed on what we call today, Berkshelf.
  • Berkshelf is a standalone binary*NEXT*It’s a command line tool*NEXT*It’s a source code management tool similar to Mix from Elixir, Go from Go, or Leiningen from Clojure*NEXT*It’s a package manager similar to Gem, Apt, or Yum*NEXT*And it reproduces the Cookbook generation, packaging, and distribution portions of KnifeEven after I tell people what Berkshelf is, there’s still an elephant in the room, an unanswered question left on everyone’s mind, so I’ll just get that out of the way.
  • Why is it called Berkshelf?If you go through the commit history you’ll see that it wasn’t always called Berkshelf. When we finally landed on the idea to store cookbooks on your own local machine we wanted to call it Bookshelf. It fit with the the naming theme Chef already had going for it: It’s where you store cookbooks in your house, but that’s just as easy to google as Chef and Knife.My team and I sit in a jabber chat room throughout the day where we share image macros of extremely important things. One of those was this particular image:*NEXT*This picture of a little girl telling you just how much she loves Gersberms, her fravrit berks.
  • I know a lot of you here at ChefConf already know all about cookbooks, but let’s take a quick minute to explain what a cookbook is for anyone watching the VHS after the conference or anyone in the audience whose a newcomer.
  • Cookbooks contain instructions for how to install and configure a thing or components of a thing. These cookbooks are fed into Chef-Client or Chef-Solo along with some information about your computer.In this example we’re feeding the NGINX cookbook into Chef
  • Every cookbook contains at least one recipe (called default.rb), but it could contain more with more verbose names. These recipes are where the instructions are held for configuring your machine.
  • And these instructions are represented by collections of resources.Resources are an abstraction on top of your operating system to manipulate things like * services * files * directories * packages * yum repositories
  • Here is one of the resources from the default recipe in the NGINX cookbook. It’s a package resource which will install the “NGINX” package on your computer.The really cool thing here is that this resource doesn’t make any mention about WHAT operating system we’re running on. It just knows that you want to install the package called NGINX and it knows how to do that depending on which operating system this recipe is executed on.This is why I love Chef. Chef gives us the power to make configuring machines easy, regardless of what platform. In the very near future I envision a world where people use Chef to configure their machines not just because it’s a repeatable and reliable way, but just because it’s easier to do it then it is to manually do it.One of the ways that we’re going to get there is by providing Chef developers with tools that allow them to be productive as possible.
  • And we think that Berkshelf is that tool. Let’s take a quick 5 minutes to all get familiar.
  • The first thing you want to do is get Ruby on your machine.Assuming that you already have Ruby, all you need to do is install the Berkshelf gem with this command here.This will give you access to the “berks” command. With the berks command you can do things like generate a new cookbook
  • The cookbook generated will include everything that you need to get moving.And if you already have an existing cookbook that you want to add Berkshelf support to
  • You can just run the Berks init command
  • And if at any time you get stuck you can always just ask Berks for help by passing the –h flag to berks or any of it’s sub commands. Like any good unix utility, it will give you a description of all the commands and the option flags that they support.
  • To speed things up let’s look at a cookbook that already exists. The Chef-Server community cookbook. This is the cookbook which you would use to build an open source Chef Server in your infrastructure.*NEXT*Here is the directory structure of the Chef-Server cookbook. It pretty standard stuff. There are two files in the root of the cookbook which are very important to the operation of Berkshelf that we need to highlight.*NEXT*The Berksfile and metadata files
  • One of the most important yet, most neglected files in a cookbook is the metadata file. It describes a lot of very important pieces of information.
  • It contains the name of the cookbook. In the current version of Chef, this field is automatically populated by the name of the directory that the Cookbook is in.This isn’t the behavior that you want: the Chef-Server cookbook is still the Chef-Server cookbook even if it’s in a directory called Cheese-Steak. This will be fixed in a future version of Chef, but for now PLEASE explicitly set the name of your cookbook.
  • It also contains the version of the cookbook.Version numbers in Chef are represented using a versioning scheme called Semantic Versioning or SemVer. You can read the RFC at semver.org if you want to study up, but you really just need to know that there are three numbers and two dots.If the first number changes then there’s probably breaking changes between the last released version that you should be aware of and there’s probably a bunch of awesome features in it. It’s called the Major version.When the second number changes it means there’s some new features, enhancements to old features, and cumulative bug fixes in the new version. It’s called the Minor version.A cookbook whose third number has been changed indicates bug fixes only. This is called the patch version. It’s almost always a good idea to accept the latest patch version of a cookbook.None of these numbers should ever decrement unless the number to their left is incremented. For example: 1.2.3 is a lower version than 2.0.0.
  • The metadata also included is a list of supported platforms, like CentOS, Red Hat, or Ubuntu.When authoring a cookbook, be sure to include all of the platforms you know that cookbook to work on. Please don’t leave it blank or list EVERY platform if you haven’t tested the cookbook on them all. People will hate you for wasting their time.This attribute will become more important as the adoption rates for automated testing tools, like test kitchen, increase. These tools will reflect on the supported platforms to know what platforms to automatically test your cookbook against.
  • And the last important thing that you can describe in a Cookbook’s metadata is any dependencies upon other Cookbooks.Anytime you leverage an LWRP, library, definition, or recipe from another cookbook, that cookbook should be listed as a dependency in your metadata.The purpose of this is two fold:1. it will ensure that your Chef run will fail early if the Chef-Client doesn’t have all of the necessary cookbooks to successfully converge2. It will also give Berkshelf a list of Cookbook versions that it needs to satisfy and retrieve
  • This is what the actual contents of the Chef-Server metadata looks like
  • Now, the Berksfile is what is actually given to Berkshelf.It defines a list of cookbook sources that you want to retrieve. The Berksfile for the Chef-Server cookbook is pretty basic containing only about 5 lines of code.The first line says, “look on the Opscode community site for any cookbooks we define without an explicit location”.The second line instructs Berkshelf to look at the metadata file found in the current working directory. Pretty much every cookbook that contains a Berksfile will have this line in it. Berkshelf will read the metadata and retrieve any dependencies listed in the metadata and also their dependencies, and their dependencies, and so on.
  • Now let’s go and get all the cookbooks we need to operate the Chef-Server cookbook. We do that with the Berks install command.*NEXT*Run Berks install*NEXT*And then Berkshelf will reflect on the metadata of the Chef-Server cookbook and go and download all of it’s dependencies (and their dependencies) from the Opscode community site.
  • If you’ve been using Chef you might ask yourself, “where did those cookbooks go?”. Afterall, I haven’t made mention of a Chef repository with a “cookbooks” or a “site-cookbooks” directory at all.*NEXT*That’s because, just like Ruby’s Gems, they are installed into a common location on your local disk. We call this “the berkshelf”.And unlike a “cookbooks” directory in a Chef repository, you’ll notice that there can be multiple versions of a cookbook. This is because, like Rubygems, we store every version of a cookbook that you’ve installed.
  • In order to get this behavior we couldn’t just write some software and call it a day. We also needed to make some changes to the way we we’re writing cookbooks and managing our Chef repository. We call this paradigm shift “The Berkshelf Way”.It’s a growing set of principles and best practices that we’ve discovered through creating, and teaching others on how to create cookbooks. Our goal for Berkshelf and The Berkshelf Way is to get an engineer managing their application in a production environment using Chef as soon as possible.
  • One of the first things we stress is to work in verticals from the outside-in by starting on the surface. I love the idea of behavior driven development. I think that things are easier to accomplish if you have a clear and attainable goal.Think about what our ACTUAL goal is when we implement Chef in our organizations? You wouldn’t be wrong if you said it’s to configure a massive amount of computers on your network, but let’s pull back even further.What is it exactly that your organization is providing to it’s customers?
  • Let's take Riot Games for example. What do we provide to our players?Is it NGINX?Is it CentOS?No, it’s League of Legends.
  • So create a cookbook for that!Every application or service that you are a provider for should have it's own cookbook.In the case of League of Legends we have a lot of different services which make up what our players perceive as the game. Each one of these services has it’s own cookbook which is developed and maintained by the engineering team who wrote the service. And this cookbook lives not in a Chef repository, but next to the service’s codebase and in a repository of it’s own.
  • And each one of these services is made up of components.This is a common list of components that you might see. And when a software engineer has a commonly occurring problem he can often remedy it by applying something called a Design pattern.
  • A design pattern is a solution to a commonly occurring problem within a given context in the software world.
  • We can map a service in our infrastructure to a cookbook of it’s own which will allow us to work in verticals and from the outside-in. We call this the Application Cookbook Pattern.This idea isn't something that comes with the Chefdocumentation. There's no choice to generate or designate a new cookbook as an “Application Cookbook”. It’s just a normal cookbook which follows a simple pattern that deploys and configures one of your services.In Riot’s case this is League of Legends or the services surrounding it.
  • Let's take a look at an application cookbook for my fictional social networking application "Myface”.Let's imagine that this is the only product that my company provides and it's a large app that rivals Myspace and Facebook. It's comprised of a couple of different components and it's distributed across a number of machines in a datacenter.
  • After mapping the components to their own recipe, we end up with these 7Anyone of these recipes can be put in the run list of a node and it should converge successfully.Each recipe configures/installs only that one component and nothing more.
  • Let’s open up the database server recipe.In it we are setting the mysql port to listen on 3309 instead of 3306 and the max connections to 1000 instead of 800.We’re also including a few other recipes, common system from the myface cookbook and the server recipe from the mysql cookbook.Look at what we were able to do here with just 4 lines of code. We hid the implementation detail that myface uses mysql by wrapping it in this database server recipe. We made things easier for the operator or another engineer by just having this neat little package that can be included in the run list of a machine to get a database server for myface.Now what about that _common_system recipe. That wasn’t even in the list of 7 recipes that I showed you earlier and it doesn’t map to any of myface’s components.
  • That’s because _common_system is what I like to call a “private recipe”.These are never exposed to the end user. They’re never even put in the run list of a node. They’re always included in other recipes to get some sort of shared and encapsulated behavior between recipes. These should be well documented in the code for other developers to take advantage of.Public recipes are what you actually put in the run list of a node and they are well documented in the README and metadata.
  • And if we look into common system this is what you see
  • By exposing the ‘max_connections’ and ‘port’ tunables in the mysql cookbook we were able to customize what a MyFACE database server looked like without ever opening up the MySQL cookbook and editing it ourselves.This is because the MySQL cookbook is a data driven cookbook.
  • Writing data-driven cookbook’s allows us to change it’s behavior at runtime. That dream of having a single, high quality, cookbook for any major service that can be installed on a machine is both real and possible.
  • We can also make our applications easier to manage by exposing a set of tunables.Let’s go back to our MyFACE cookbook. Operators could make the application:Listen on a different port or bind addressChange the memory usage of workersAny application configuration, reallyEven set services to started or stopped states
  • The way a cookbook behaves can be changed without ever forking and changing the actual cookbook with these 3 primitives. They act as a sort of parameter list to your cookbook.By reflecting on attribute, or data bag data, we can create high quality, portable, re-usable cookbooks. This should be the goal of any of the cookbooks that you seen on the community site.Really, there shouldn't be a reason that we all have our own MySQL or NGINX cookbook.
  • Out of the three given primitives my first choice is always attributes.They often require no additional work on the cookbook consumer because you can provide sensible default values within an attribute file.As my cookbook grows I like to create an attribute file for each recipe.Attributes are also pretty bad-ass because you can configure components of your application on an environment level like, “all of production”.All of production has this feature disabled or this rate limit set or this max connections set.
  • Data bags have their place, but you put a bit more burden on the consumer of your cookbook when you require the use of a data bag because it, and any items need to be created and keyed into the server without any sort of validation.It also requires a bit more finesse as a cookbook author. What happens if the data bag hasn’t been created and a chef run is executed? Well, the user gets this cryptic 404 message unless you handled the loading and validations in your recipe.However, they are particularly useful if you want to set stuff on an organization level. You can do things like:Setup a set of users to be put on every machineOr groupsYour organization specific yum or apt repositoriesThings configured by your “base” cookbook (if you have one)
  • They are also useful if you have any kind of sensitive data to store. Right now there is no built-in primitive found in Chef to store sensitive data as an attribute so you need to rely on encrypted data bags or hack it in yourself.If you have a password or a private key that you need to store, put that in an encrypted data bag.
  • The community site is an amazing ideal. Cookbooks which perform a purpose, only that purpose, that can be used in anyone’s infrastructure. A big collection of ready to use easy bake oven cookbooks.The problem is that a lot of these cookbooks are tied to the Chef Repository they were written around. If you package up a cookbook and send it over to me, I shouldn’t need parts of, or all of, your Chef Repository.This is a big reason why we wanted cookbooks to be developed in seclusion away from a Chef Repository. Writing cookbooks away from everything else in a project of their own produces well encapsulated and portable cookbooks.
  • You’re probably asking yourself them, “how do I use roles if they aren’t packaged with my cookbooks”?
  • Stahp it. Don’t. Don’t use em.
  • Roles aren’tVersionedThey can’t be packaged and distributedThey aren’t namespaced. We’re called Riot Game(s). Think of how awesome it is when somebody names a role “game”They are organization level data. They aren’t tied to an environment, they are tied to an organization. If I have a role called “game server” and it includes two recipes in a run list and this role is used in production and also another QA environment which tests the latest version of the game. What do you think happens when we change that role for the QA environment? Well it changes what the signature of a game server looks like in production. We could be breaking something or releasing something when we aren’t ready!They add more onto the cookbook user who now also not only needs to understand the primitive, but they also need to create that role on their Chef server.
  • Ok so you might say, “whoa… what about data bags?”They’re organization wide data too. Not only that, they’ve got a structure too them so they are even easier to break.
  • Well… use them where they make sense.I mentioned that they hold organization level data. If you need to store organization-wide data, then use one.If you need to store sensitive data, then use an encrypted one.But because data bags aren't packaged with cookbooks, I try my best to avoid using them in favor of attributes.
  • You should always validate that any data bag your cookbook requires has been created on the Chef server and is in the expected format as early as possible.
  • Another key to writing well encapsulated and easy to use cookbooks is to have an entry point, a sort of main() function.The default recipe is special. It’s that main() function for a cookbook. Even if you have never read the README of a cookbook, you should still be able to put the default recipe in the run list of a node and have a node become a thing that the cookbook represents.And this will work because all we are doing is including the recipe of each component; in the proper order; that we provide, in the default recipe.
  • Encapsulation and re-usability doesn’t only apply to recipes in a cookbook. A cookbook can provide more than that.A cookbook can also provide:Lightweight resources and providersLibrariesAnd DefinitionsCookbooks don’t even NEED to have recipes. They can just exist and contain these other things.
  • We call these cookbooks, “Library Cookbooks”.
  • These cookbooks might have recipes – but often times they don’t.They exist for the sole purpose to make your life easier when writing Chef recipes.The Database Cookbook is a perfect example
  • And there’s one last pattern that I want to talk about: the Wrapper Cookbook Pattern.
  • These are super lightweight cookbooks that provide an abstraction on top of another cookbook.They usually contains recipes and attribute overrides and nothing more.We have an internal cookbook called Riot-Java which is a perfect example of this pattern. It contains two recipes. One for Oracle Java 6 and one for Oracle Java 7.It’s an abstraction on top of the community Java cookbook for Riot Engineers. It automatically sets the install flavor to Riot’s taste and overrides the default location of the Java artifacts to an internal location.It is NOT a Fork of the Java cookbook. It’s just a thin wrapper around the Java cookbook so these things common to nearly all projects at Riot Games don’t need to be duplicated.
  • Are you still with me? There’s just one last principle that we have to cover.Even if Chef was so easy that if everyone in the world picked up Chef and immediately became a master, all of this still wouldn’t matter.It wouldn’t because we have essentially been working developing in the dark for the last four years.
  • Testing minor changes: 7 seconds
  • Testing minor changes: 7 seconds
  • It was a slow start when we set out to teach the other engineers at Riot Games how to use and develop for Chef.With Berkshelf and The Berkshelf Way, we were able to train over 30 Riot engineers how to effectively author cookbooks of production environment quality in just four weeks. These classes were held in house and taught by one member of my team. Each class was three days long and included more detailed explanations of the material that I’m going to show you today.

Transcript

  • 1. JAMIE WINSOR| SOFTWARE ENGINEERTHE BERKSHELF WAY
  • 2. LEAGUE OF LEGENDS
  • 3. ME IN2009200 201 201 201
  • 4. 200 201 201 201MY CHEFREPO IN2010
  • 5. 200 201 201 201AND IN2011…
  • 6. 200 201 201 201…FROM2012AND ON
  • 7. WHAT’SBERKSHELF?‣ A COMMAND LINETOOL‣ A SOURCE CODEMANAGEMENT TOOL▾ Similar to „Mix‟, „Go‟, or„Leiningen‟‣ A PACKAGE MANAGER▾ Similar to „Gem‟, „Apt‟, or „Yum‟‣ REPLACES PORTIONSOF KNIFE▾ Cookbook Generator, CookbookUploader, Cookbook Downloader
  • 8. WHY IS IT CALLED BERKSHELF?
  • 9. INTRO TOCOOKBOOKS:60
  • 10. COOKBOOKS ARE FED INTO CHEFNGINXCOOKBOOK1.2.3Chef-Client /Chef-Solo
  • 11. RECIPES GO IN COOKBOOKSNGINXCOOKBOOK1.2.3RECIPERECIPERECIPE
  • 12. RECIPES CONTAIN RESOURCECOLLECTIONSDEFAULTPACKAGE NGINXSERVICE NGINXREPOYUM_REPOSITORYNGINXYUM_KEY NGINXNGINXCOOKBOOK1.2.3
  • 13. EXAMPLE RESOURCE123package “nginx” doaction :installendPACKAGE NGINX
  • 14. INTRO TOBERKSHELF5min
  • 15. INSTALL BERKSHELF$ gem install berkshelfSuccessfully installed berkshelf-1.4.01 gem installed
  • 16. CREATE A NEW COOKBOOK$ berks cookbook myfacecreate myface/files/defaultcreate myface/templates/defaultcreate myface/attributescreate myface/definitionscreate myface/librariescreate myface/providerscreate myface/recipescreate myface/resourcescreate myface/recipes/default.rbcreate myface/metadata.rbcreate myface/LICENSEcreate myface/README.mdcreate myface/Berksfilecreate myface/chefignorecreate myface/.gitignorerun git init from "./myface"create myface/Gemfilecreate myface/VagrantfileUsing myface (0.1.0) at path: /Users/reset/code/riot-cookbooks/myface
  • 17. CONVERT AN EXISTING COOKBOOK$ berks initcreate Berksfilecreate chefignorecreate .gitignorerun git init from "."create Gemfilecreate VagrantfileUsing old_cookbook (0.1.0) at path: /Users/reset/code/riot-cookbooks/old_cookbookSuccessfully initialized
  • 18. STUCK? ASK FOR HELP!$ berks cookbook –hUsage:berks cookbook NAMEOptions:[--foodcritic] # Creates a Thorfile with Foodcritic support to lint test your cookbook[--chef-minitest] # Creates chef-minitest support files and directories, adds minitest-handler …[--scmversion] # Creates a Thorfile with SCMVersion support to manage versions for continuous integration-L, [--license=LICENSE] # License for cookbook (apachev2, gplv2, gplv3, mit, reserved)# Default: reserved-m, [--maintainer=MAINTAINER] # Name of cookbook maintainer-e, [--maintainer-email=MAINTAINER_EMAIL] # Email address of cookbook maintainer[--no-bundler] # Skips generation of a Gemfile and other Bundler specific support[--skip-vagrant] # Skips adding a Vagrantfile and adding supporting gems to the Gemfile[--skip-git] # Skips adding a .gitignore and running git init in the cookbook directory-c, [--config=PATH] # Path to Berkshelf configuration to use.-F, [--format=FORMAT] # Output format to use.# Default: human-q, [--quiet] # Silence all informational output.-d, [--debug] # Output debug informationCreate a skeleton for a new cookbook
  • 19. A COMPLETE EXAMPLECHEF-SERVERCOOKBOOK1.2.3├── Berksfile├── CHANGELOG.md├── CONTRIBUTING.md├── Gemfile├── LICENSE├── README.md├── Thorfile├── Vagrantfile├── attributes│ └── default.rb├── chefignore├── libraries│ ├── dev_helper.rb│ └── omnitruck_client.rb├── metadata.rb├── recipes│ ├── default.rb│ └── dev.rb└── templates└── default└── chef-server.rb.erbBerksfilemetadata.rb
  • 20. METADATA IS IMPORTANTCOOKBOOK
  • 21. COOKBOOKMETADATA IS IMPORTANTCHEF-SERVERTHE NAME OF THECOOKBOOK
  • 22. COOKBOOKCHEF-SERVERMETADATA IS IMPORTANTV2.0.0 THE VERSION OF THECOOKBOOKFOLLOW SEMVER: HTTP://SEMVER.O
  • 23. METADATA IS IMPORTANTCOOKBOOKYUMV1.2.3COOKBOOKCHEF-SERVERV1.2.3COOKBOOKAPTV1.2.3Red Hat CentOS UbuntuSET YOUR SUPPORTED PLATFORMSCentOS UbuntuRed Hat
  • 24. METADATA IS IMPORTANTCOOKBOOKYUMV1.2.3SET WHAT COOKBOOKS YOU DEPEND ONCOOKBOOKCHEF-SERVERV1.2.3COOKBOOKAPTV1.2.3SUPPORTEDPLATFORMSSUPPORTEDPLATFORMSSUPPORTEDPLATFORMS
  • 25. CHEF-SERVER COOKBOOK’S METADATAname "chef-server"maintainer "Opscode, Inc."maintainer_email "cookbooks@opscode.com"license "Apache 2.0"description "Installs and configures Chef Server"long_description IO.read(File.join(File.dirname(__FILE__), README.md))version "2.0.0"depends "apt", ">= 1.7.0"depends "yum", ">= 0.5.0"%w{ ubuntu redhat centos fedora amazon scientific oracle }.each do |os|supports osend
  • 26. IT STARTS WITH A BERKSFILE├── Berksfile├── CHANGELOG.md├── CONTRIBUTING.md├── Gemfile├── LICENSE├── README.md├── Thorfile├── Vagrantfile├── attributes│ └── default.rb├── chefignore├── libraries│ ├── dev_helper.rb│ └── omnitruck_client.rb├── metadata.rb├── recipes│ ├── default.rb│ └── dev.rb└── templates└── default└── chef-server.rb.erbBerksfilesite :opscodemetadatagroup :dev docookbook gitendBerksfile
  • 27. INSTALLING COOKBOOKSUsing chef-server (2.0.0) at path: /Users/reset/code/riot-cookbooks/chef-serverInstalling git (2.5.0) from site: http://cookbooks.opscode.com/api/v1/cookbooksInstalling apt (1.9.2) from site: http://cookbooks.opscode.com/api/v1/cookbooksInstalling yum (2.2.0) from site: http://cookbooks.opscode.com/api/v1/cookbooksInstalling dmg (1.1.0) from site: http://cookbooks.opscode.com/api/v1/cookbooksInstalling build-essential (1.4.0) from site: http://cookbooks.opscode.com/api/v1/cookbooksInstalling windows (1.8.8) from site: http://cookbooks.opscode.com/api/v1/cookbooksInstalling chef_handler (1.1.4) from site: http://cookbooks.opscode.com/api/v1/cookbooksInstalling runit (1.1.2) from site: http://cookbooks.opscode.com/api/v1/cookbooks$ berks install
  • 28. INSTALLED TO WHERE?$ ls ~/.berkshelf/cookbooksapache2-1.6.0 chef_handler-1.1.4 known_host-0.1.3 nginx-1.3.0 riot_chef-client-2.0.7 tsf4g-0.0.14apt-1.9.0 couchdb-2.4.0 liquibase-0.0.1 not_system_ruby-0.1.7 riot_coherence-0.10.12 ucspi-tcp-1.0.0ark-0.0.11 cron-1.0.4 logrotate-0.8.2 ntp-1.1.8 riot_ejabberd-0.0.152 ulimit-0.1.3artifact-0.10.10 daemontools-1.0.0 logrotate-1.1.0 ohai-1.1.6 riot_ejabberd-0.0.188 users-1.1.4artifact-0.10.2 database-1.3.12 lvm-0.8.6 openssl-1.0.0 riot_geoip-0.1.0 vim-1.0.2artifact-1.2.0 database-1.3.4 magic_shell-0.2.0 postgresql-1.0.0 riot_java-0.1.4 windows-1.8.2artifact-1.3.1 dmg-1.1.0 man-0.7.0 pvpnet_core-2.3.123 riot_java-0.1.8 xfs-1.0.0artifact-1.4.0 editor-0.2.0 mysql-1.2.4 pvpnet_core-2.3.131 riot_mysql-2.0.1 xml-1.1.2aws-0.100.0 erlang-1.2.0 mysql-1.3.0 python-1.2.2 riot_rtview-1.0.1 yum-2.1.0bluepill-1.1.0 gecode-2.0.0 mysql-2.1.0 rbenv-1.3.2 riot_team-0.9.41 zlib-2.0.0build-essential-1.1.2 git-2.2.0 mysql-2.1.2 riot_activemq-0.9.28 riot_tomcat-1.0.18build-essential-1.3.4 hornetq-0.0.27 networking_basic-0.0.5 riot_base-2.7.32 runit-0.16.2chef-client-2.1.4 java-1.5.0 nexus-1.1.0 riot_base-2.7.37 runit-1.0.4chef-server-1.1.0 java-1.9.0 nginx-1.2.0 riot_base-2.7.49 sudo-1.2.0
  • 29. UPLOADING COOKBOOKSUsing chef-server (2.0.0) at path: /Users/reset/code/riot-cookbooks/chef-serverUsing git (2.2.0)Using apt (1.9.0)Using yum (2.1.0)Using dmg (1.1.0)Using build-essential (1.3.4)Using windows (1.8.2)Using chef_handler (1.1.4)Using runit (0.16.2)Uploading chef-server (2.0.0) to: https://api.opscode.com:443/organizations/vialstudiosUploading git (2.2.0) to: https://api.opscode.com:443/organizations/vialstudiosUploading apt (1.9.0) to: https://api.opscode.com:443/organizations/vialstudiosUploading yum (2.1.0) to: https://api.opscode.com:443/organizations/vialstudiosUploading dmg (1.1.0) to: https://api.opscode.com:443/organizations/vialstudiosUploading build-essential (1.3.4) to: https://api.opscode.com:443/organizations/vialstudiosUploading windows (1.8.2) to: https://api.opscode.com:443/organizations/vialstudiosUploading chef_handler (1.1.4) to: https://api.opscode.com:443/organizations/vialstudiosUploading runit (0.16.2) to: https://api.opscode.com:443/organizations/vialstudios$ berks upload
  • 30. THEBERKSHELF WAY
  • 31. WORK INVERTICALSFROM THEOUTSIDE-INTHEBERKSHELFWAY
  • 32. CREATE ACOOKBOOKFOR YOURAPPLICATIONCOOKBOOK
  • 33. SERVICES ARE MADE UP OFCOMPONENTS1 app servers2 cache servers3 database servers4 background workers5 load balancers6 etc
  • 34. APPLY A DESIGN PATTERN
  • 35. APPLICATIONCOOKBOOKPATTERNCOOKBOOK
  • 36. COOKBOOKMYFACE
  • 37. COMPONENTS OF MYFACE1 load balancer2 application server proxy3 caching service4 application server5 background worker6 database server
  • 38. MAPCOMPONENTSTO RECIPESRECIPERECIPERECIPERECIPE
  • 39. RECIPES OF MYFACEmyface::defaultmyface::load_balancermyface::app_proxymyface::app_servermyface::cache_servermyface::worker_poolmyface::database_server
  • 40. CRACK OPENMYFACE::DATABASE_SERVERnode.set[:mysql][:port] = 3309node.set[:mysql][:tunable][:max_connections] = 1000include_recipe "myface::_common_system"include_recipe "mysql::server"
  • 41. PUBLIC/PRIVATE RECIPESPUBLIC RECIPE‣ What you put in therun_list of a node‣ Documented in theREADME‣ Documented in themetadataPRIVATE RECIPE‣ Not exposed to end user‣ Should never be put inthe run_list of a node‣ Always included intoother recipes‣ Documented in the codefor other developers
  • 42. WHAT’S INMYFACE::_COMMON_SYSTEM?include_recipe "ntp::default"# . . . MAYBE SETUP SOME USERS . . .# . . . MAYBE SETUP SOME ADDITIONAL REPOS . . .
  • 43. WORK INVERTICALSFROM THEOUTSIDE-INFAVOR DATA-DRIVENCOOKBOOKSTHEBERKSHELFWAY
  • 44. COOKBOOKCHANGE ACOOKBOOKSBEHAVIOR ATRUNTIME
  • 45. TUNABLES OF MYFACE1 listen on a different port or bind address2 set memory usage of workers3 any app config4 set services to started or stopped states
  • 46. 3 WAYS TO CONFIGUREATTRIBUTESDATA BAGSENCRYPTED DATA BAGS
  • 47. FAVOR CONFIGURING WITH ATTRIBUTES‣ Path of least resistance for cookbook consumers‣ Provide sensible defaults in the Attribute files▾ Consider creating an attribute file for each recipe‣ Can configure your application on an environmentlevel
  • 48. WHEN TO USE DATA BAGSUSERSGROUPSYUM/APTTHINGS CONFIGURED BY "BASE"COOKBOOK (IF YOU HAVE ONE)ORGANIZATION LEVELCONFIGURATIONS
  • 49. USE ENCRYPTED DATA BAGSWHEN STORING SENSITIVEDATA
  • 50. WORK INVERTICALSFROM THEOUTSIDE-INFAVOR DATA-DRIVENCOOKBOOKSWRITE WELL-ENCAPSULATEDCOOKBOOKSTHEBERKSHELFWAY
  • 51. NOT VERSIONEDCAN’T BE PACKAGED / DISTRIBUTEDNOT NAMESPACEDSTAHP USING ROLESTHEY ARE ORGANIZATION LEVEL DATAADDITIONAL SETUP COMPLEXITY FORTHE USER
  • 52. USE DATA BAGS WHERE THEY MAKESENSEDO YOU NEED TO STOREORGANIZATION-WIDE DATA?DO YOU NEED TO STORESENSITIVE DATA?Use oneUse an encrypted one
  • 53. USE DATA BAGS WHERE THEY MAKESENSEMOST IMPORTANTLY…VALIDATE YOUR DATABAGS!
  • 54. 5 include_recipe "myface::database”6 include_recipe "myface::app_server”7 include_recipe "myface::worker_pool”8 include_recipe "myface::cache_server”9 include_recipe "myface::app_proxy"DEFAULTS RECIPE IS SPECIAL
  • 55. LIGHTWEIGHT RESOURCES ANDPROVIDERS (LWRP)LIBRARIESDEFINITIONSCOOKBOOKS ARE MORE THAN RECIPESCOOKBOOKS DON’TNEED TO HAVE RECIPES
  • 56. COOKBOOK LIBRARYCOOKBOOKPATTERN
  • 57. LIBRARY COOKBOOKS‣ Probably doesn‟t have recipes‣ Contain LWRPs, Definitions, or Libraries useful toother cookbook authors‣ Database Cookbook is a perfect example. Has HWRP‟s for▾ Managing mysql, postgres, and sql server databases▾ Managing database users in mysql, postgres, and sql server
  • 58. COOKBOOK WRAPPERCOOKBOOKPATTERN
  • 59. WRAPPER COOKBOOKS‣ Super lightweight‣ Contain recipes‣ Contain attribute overrides‣ Riot-Java is a perfect example▾ Contains a Oracle Java 6 recipe▾ Contains a Oracle Java 7 recipe▾ An abstraction on top of the Java cookbook for Riot engineers▾ Overrides the default “install flavor” for Java▾ Overrides the default location of the Java artifacts to an internallocation▾ NOT A FORK OF THE JAVA COOKBOOK
  • 60. WORK INVERTICALSFROM THEOUTSIDE-INFAVOR DATA-DRIVENCOOKBOOKSWRITE WELL-ENCAPSULATEDCOOKBOOKSHAVE SHORTITERATIONLOOPSTHEBERKSHELFWAY
  • 61. OLD WORKFLOWEDITCOOKBOOKUPLOADCOOKBOOKPROVISIONMACHINEBOOTSTRAPMACHINERUN CHEFCLIENTSSH ANDVALIDATE5-30
  • 62. NEW WORKFLOWEDITCOOKBOOKVAGRANT UP/ VAGRANTPROVISIONTEST / SSHANDVALIDATE5 seconds – 1 minute
  • 63. BERKSHELF VAGRANT PLUG-IN$ vagrant plugin install vagrant-berkshelfInstalling the vagrant-berkshelf plugin. This can take a few minutes...Installed the plugin vagrant-berkshelf (1.2.0)!
  • 64. Bringing machine default up with virtualbox provider...[Berkshelf] Updating Vagrants berkshelf: /Users/reset/.berkshelf/vagrant/berkshelf-20130424-40671-t43soz’Generating chef JSON and uploading...Running chef-solo...[2013-04-25T03:01:31+00:00] INFO: *** Chef 10.14.2 ***[2013-04-25T03:01:32+00:00] INFO: Setting the run_list to ["recipe[minitest-handler::default]", "recipe[myface::default]"] from JSON[2013-04-25T03:01:32+00:00] INFO: Starting Chef Run for myface-berkshelf#### CHEF RUN GOES HERE ####[2013-04-25T03:02:10+00:00] INFO: Chef Run complete in 38.330025825 seconds[2013-04-25T03:02:10+00:00] INFO: Running report handlersRun options: -v --seed 41224# Running tests:minitesthandler::default#test_0001_runs_no_tests = 0.00 s = .myface::default#test_0001_runs_no_tests_by_default =0.00 s = .Finished tests in 0.002425s, 824.7576 tests/s, 0.0000 assertions/s.2 tests, 0 assertions, 0 failures, 0 errors, 0 skips[2013-04-25T03:02:10+00:00] INFO: Report handlers completePROVISION NEW ENVIRONMENT INMINUTES$ vagrant up1 minute 20 sec
  • 65. Running chef-solo...[2013-04-25T03:01:31+00:00] INFO: *** Chef 10.14.2 ***[2013-04-25T03:01:32+00:00] INFO: Setting the run_list to ["recipe[minitest-handler::default]", "recipe[myface::default]"] from JSON[2013-04-25T03:01:32+00:00] INFO: Starting Chef Run for myface-berkshelf#### CHEF RUN GOES HERE ####[2013-04-25T03:02:10+00:00] INFO: Chef Run complete in 0.520326234 seconds[2013-04-25T03:02:10+00:00] INFO: Running report handlersRun options: -v --seed 41224# Running tests:minitesthandler::default#test_0001_runs_no_tests = 0.00 s = .myface::default#test_0001_runs_no_tests_by_default =0.00 s = .Finished tests in 0.002425s, 824.7576 tests/s, 0.0000 assertions/s.2 tests, 0 assertions, 0 failures, 0 errors, 0 skips[2013-04-25T03:02:10+00:00] INFO: Report handlers completeCONVERGE IN SECONDS$ vagrant provision7 seconds
  • 66. WORK INVERTICALSFROM THEOUTSIDEFAVOR DATA-DRIVENCOOKBOOKSWRITE WELL-ENCAPSULATEDCOOKBOOKSHAVE SHORTITERATIONLOOPSTHEBERKSHELFWAY
  • 67. LEARNMORE ATBERKSHELF.COM
  • 68. CONTRIBUTEGITHUB.COM/RIOTGAMES/BERKSHELFHOW CAN YOU HELP?OR JOIN USWWW.RIOTGAMES.COM/CAREERS
  • 69. CALL MEMAYBE?Jamie Winsor@resetexistencegithub.com/reset