Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
@seanwalberg
BREAKING TECHNOLOGY
SILOS WITH CHEF
Sean Walberg <sean@ertw.com>
Infrastructure guy
National Football League
@seanwalberg
These are silos
(Every DevOps presentation needs a picture of a silo)
@seanwalberg
Yup
@seanwalberg
Tech view
@seanwalberg
People view
@seanwalberg
How do you change a
culture?
@seanwalberg
BLUF
Optimize for conversations – automate
the bad ones away
Make a menu – Do one thing, and do it
well.
Don’...
@seanwalberg
Start Development
Go faster!
STOP!
@seanwalberg
OFF SEASON, 2014
@seanwalberg
@seanwalberg
LET’S FIX SOME PROBLEMS
• Environmental drift
• Configuration files
• Developer access to servers
@seanwalberg
INTER/INTRA ENVIRONMENT DRIFT
@seanwalberg
AUTOMATE ALL THE THINGS!
@seanwalberg
NFL NOW ENVIRONMENT
• About a dozen services and ~100 servers by the
end in production, ~200 in total
• Cookb...
@seanwalberg
CONFIGURATION FILES
http://www.mommysbusy.com/wp-content/uploads/2013/10/Messy-Desk.jpg
@seanwalberg
CAN WE FIX IT? YES WE CAN!
Developer modifies config on dev server
Infrastructure team diffs and templates
Ot...
@seanwalberg
http://6iee.com/372028.html
DEVELOPER ACCESS TO SERVERS
@seanwalberg
• Development only
• Except when we’re in a crunch
• And then we turn off Chef
• Then I spend hours cleaning ...
@seanwalberg
KNIFE-VSPHERE
• Clone + Bootstrap VM
• Take snapshots
• Resize/add disk
• Change settings
• Run commands
@seanwalberg
DURING THE SEASON
@seanwalberg
NODE, ADD THYSELF TO A POOL!
f5_pool ”pool_name" do
host node['fqdn']
ip node['ipaddress']
port app_port
not_...
@seanwalberg
START ROLLING OUT CHEF
• role[base] – everything new
• LDAP
• Access control
• Base packages
• role[minimal] ...
@seanwalberg
RETROSPECTIVE
• Chef + knife-vsphere + f5 worked great
• Still many manual steps though
• Why do developers s...
@seanwalberg
OFF SEASON, 2015
Microservices, yay!
@seanwalberg
I REALLY WANT TO FIX THIS STUFF
• Developers on servers
• Standardization of frameworks
• Config files
@seanwalberg
LET’S HAVE SOME CONVERSATIONS
• “Is the config on the box what it should be?”
• “I need to see the logs!”
• “...
@seanwalberg
THEY’RE ALL RELATED!
@seanwalberg
WE’RE A RESTAURANT AND WE
HAVE ONE DISH
@seanwalberg
LE MENU
• Everything is built around the name of the app
• Start with a consistent build pipeline
• Start pro...
@seanwalberg
CONSISTENT NAMING
Tags: [app:sso]
Runlist: recipe[nfl-apps]
Server: locenvssozz
Service: sso
Logs: /var/log/s...
@seanwalberg
LOGGING
template "/opt/nfl/…/logback.xml” do
source 'logback.xml.erb'
owner 'root'
mode '755'
variables(
app_...
@seanwalberg
ADD SOME CONTEXT TO LOGS
<staticAdditionalField>_app:<%= @app_service %></staticAdditionalField>
<staticAddit...
@seanwalberg
EASY CHANGING OF LOGGING
<!-- Shortcut to debug -->
<% node['apps']['debug_packages'].each do |package| %>
<l...
@seanwalberg
APPDYNAMICS (APM)
Install agents
Configure agents based on attributes
No knowledge of app
Include_recipe “nfl...
@seanwalberg
LET’S FIX CONFIGURATION
• Chef drops a consul agent on each server
• Joins it to the cluster for that environ...
@seanwalberg
SAME PATTERN AS BEFORE
Install consul
Join to #{node.chef_environment}
No knowledge of app
Include_recipe “nf...
@seanwalberg
USING CONFIG
• Developers commit to Consulation (YAML)
• Peer review + linting/smoketest in Phabricator
• Sec...
@seanwalberg
ATTRIBUTES ARE A GOOD
INTERFACE
default['apps']['os_memory'] = '1024’
if jvm_memory is not overridden
system_...
@seanwalberg
OR USE A ROLE
$ knife role show os1536_memory_adjustment
chef_type role
name: os1536_memory_adjustment
overri...
@seanwalberg
DO YOU SERVE GLUTEN FREE?
Yes, you can order off the menu.
But we need to talk, and it’ll take longer.
How ma...
@seanwalberg
RETROSPECTIVE
• Config evolved, but worked great
• Standardized apps meant no need for dev box
• Devs on serv...
@seanwalberg
WHAT ELSE CAN WE AUTOMATE?
• DNS
• AppD configuration
• Builds and deploys
• Fastly (CDN) configuration
@seanwalberg
The code needs to be the
source of truth
Configuration Transformation
API driven deploy
@seanwalberg
This becomes VCL
This is a functional
test
CHEF TAUGHT US THAT CONFIGURATION
DOESN’T HAVE TO SUCK
@seanwalberg
This is so
much better
than that
@seanwalberg
DON’T AUTOMATE THE
CONVERSATIONS AWAY
@seanwalberg
NOW, LET’S GET BETTER AT
USING CHEF
Again, conversations will help us.
@seanwalberg
CONTINUOUS INTEGRATION
“Does it converge?”
@seanwalberg
PEER REVIEW
@seanwalberg
AUTO LINTING
@seanwalberg
ESTABLISH A WORKFLOW
@seanwalberg
ALSO REMEMBER
You’re coding for the most
inexperienced member of your team
AKA the “the new person”
@seanwalberg
LET’S WRAP
• Think about conversations
• Chef is more than just configs on servers
• Reduce complexity with a...
@seanwalberg
THANKS!
Sean Walberg <sean@ertw.com>
@seanwalberg
Upcoming SlideShare
Loading in …5
×

Breaking Technology Silos with Chef

281 views

Published on

Chef is an amazing tool but to really unlock its potential you need to look at how it integrates with the rest of your technology. This presentation is the story of how the NFL used Chef to transform its siloed infrastructure and practices into something more agile, automated, and reliable. This presentation will talk about the last 2 years of Chef at the NFL, including how we integrated it with our virtualization infrastructure, load balancers, storage, and application performance monitoring. We'll talk about some things that Chef taught us about infrastructure as code that we were able to apply to other areas, and things we learned to make our cookbooks easier to manage across groups.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Breaking Technology Silos with Chef

  1. 1. @seanwalberg BREAKING TECHNOLOGY SILOS WITH CHEF Sean Walberg <sean@ertw.com> Infrastructure guy National Football League
  2. 2. @seanwalberg These are silos (Every DevOps presentation needs a picture of a silo)
  3. 3. @seanwalberg Yup
  4. 4. @seanwalberg Tech view
  5. 5. @seanwalberg People view
  6. 6. @seanwalberg How do you change a culture?
  7. 7. @seanwalberg BLUF Optimize for conversations – automate the bad ones away Make a menu – Do one thing, and do it well. Don’t neglect your team
  8. 8. @seanwalberg Start Development Go faster! STOP!
  9. 9. @seanwalberg OFF SEASON, 2014
  10. 10. @seanwalberg
  11. 11. @seanwalberg LET’S FIX SOME PROBLEMS • Environmental drift • Configuration files • Developer access to servers
  12. 12. @seanwalberg INTER/INTRA ENVIRONMENT DRIFT
  13. 13. @seanwalberg AUTOMATE ALL THE THINGS!
  14. 14. @seanwalberg NFL NOW ENVIRONMENT • About a dozen services and ~100 servers by the end in production, ~200 in total • Cookbook per app • Environment settings in Chef environment
  15. 15. @seanwalberg CONFIGURATION FILES http://www.mommysbusy.com/wp-content/uploads/2013/10/Messy-Desk.jpg
  16. 16. @seanwalberg CAN WE FIX IT? YES WE CAN! Developer modifies config on dev server Infrastructure team diffs and templates Other environment settings are added to cookbook or environment Deploy to staging/preview/production
  17. 17. @seanwalberg http://6iee.com/372028.html DEVELOPER ACCESS TO SERVERS
  18. 18. @seanwalberg • Development only • Except when we’re in a crunch • And then we turn off Chef • Then I spend hours cleaning it up DEVELOPER ACCESS TO SERVERS
  19. 19. @seanwalberg KNIFE-VSPHERE • Clone + Bootstrap VM • Take snapshots • Resize/add disk • Change settings • Run commands
  20. 20. @seanwalberg DURING THE SEASON
  21. 21. @seanwalberg NODE, ADD THYSELF TO A POOL! f5_pool ”pool_name" do host node['fqdn'] ip node['ipaddress'] port app_port not_if { node['apps']['skip_load_balancer’]} end
  22. 22. @seanwalberg START ROLLING OUT CHEF • role[base] – everything new • LDAP • Access control • Base packages • role[minimal] – just get Chef on it • Base packages • SSH keys
  23. 23. @seanwalberg RETROSPECTIVE • Chef + knife-vsphere + f5 worked great • Still many manual steps though • Why do developers still need access to servers? • Can we manage config files asynchronously? • No standardization in frameworks.
  24. 24. @seanwalberg OFF SEASON, 2015 Microservices, yay!
  25. 25. @seanwalberg I REALLY WANT TO FIX THIS STUFF • Developers on servers • Standardization of frameworks • Config files
  26. 26. @seanwalberg LET’S HAVE SOME CONVERSATIONS • “Is the config on the box what it should be?” • “I need to see the logs!” • “This framework looks cool, maybe I’ll try it.” • “I don’t trust what you’re telling me.” • “I didn’t know that makes your life harder.”
  27. 27. @seanwalberg THEY’RE ALL RELATED!
  28. 28. @seanwalberg WE’RE A RESTAURANT AND WE HAVE ONE DISH
  29. 29. @seanwalberg LE MENU • Everything is built around the name of the app • Start with a consistent build pipeline • Start projects from a template • Log with slf4j, we’ll config graylog for you • Deployable fat JAR (for Java stuff) • Instrumentation is added on the server • One chef recipe
  30. 30. @seanwalberg CONSISTENT NAMING Tags: [app:sso] Runlist: recipe[nfl-apps] Server: locenvssozz Service: sso Logs: /var/log/sso/ Deployable: /opt/nfl/sso/sso.jar Graylog: tag logs with app:sso AppD Tier: sso Repo: sso
  31. 31. @seanwalberg LOGGING template "/opt/nfl/…/logback.xml” do source 'logback.xml.erb' owner 'root' mode '755' variables( app_service: app, instance_id: instance_id, environment: node.chef_environment ) end
  32. 32. @seanwalberg ADD SOME CONTEXT TO LOGS <staticAdditionalField>_app:<%= @app_service %></staticAdditionalField> <staticAdditionalField>_env:<%= @environment %></staticAdditionalField> <staticAdditionalField>_instance:<%= @instance_id %></staticAdditionalField>
  33. 33. @seanwalberg EASY CHANGING OF LOGGING <!-- Shortcut to debug --> <% node['apps']['debug_packages'].each do |package| %> <logger name="<%= package %>" level="debug" /> <% end %> <!-- Fine tuning --> <% if node['apps'].key? 'loglevel’ node['apps']['loglevel'].each do |package, level| %> <logger name="<%= package %>" level="<%= level %>" /> <% end; end %> <!-- Default logging based on tag --> <logger name="com.nfl.dm.<%= @app_service %>" level="info" />
  34. 34. @seanwalberg APPDYNAMICS (APM) Install agents Configure agents based on attributes No knowledge of app Include_recipe “nfl-appdynamics” Adjust startup scripts Handle custom AppD config Legacy stuff recipe[nfl-appdynamics::agent] Fix your own startup scripts and custom config
  35. 35. @seanwalberg LET’S FIX CONFIGURATION • Chef drops a consul agent on each server • Joins it to the cluster for that environment • Sets startup scripts for the app to tell the app where to find consul, and the configs within the KV store • Starter template provides a module that reads Consul on startup and configs Spring Boot • Config is in a repo that anyone can use I would be lying if I said we got this right on the first try!
  36. 36. @seanwalberg SAME PATTERN AS BEFORE Install consul Join to #{node.chef_environment} No knowledge of app Include_recipe “nfl-consul” Adjust startup scripts Populate service discovery values Legacy stuff Recipe[nfl-consul] Fix your own startup scripts
  37. 37. @seanwalberg USING CONFIG • Developers commit to Consulation (YAML) • Peer review + linting/smoketest in Phabricator • Secrets in Vault • Auto deploy to environments on merge The Rules 1. There is no other config but Consulation 2. If it changes meaning, change the key name 3. If you don’t know what it is, ignore it
  38. 38. @seanwalberg ATTRIBUTES ARE A GOOD INTERFACE default['apps']['os_memory'] = '1024’ if jvm_memory is not overridden system_memory = memory_from_ohai jvm_memory = system_memory – os_memory end # Wrapper cookbook can tweak either jvm memory or os_memory
  39. 39. @seanwalberg OR USE A ROLE $ knife role show os1536_memory_adjustment chef_type role name: os1536_memory_adjustment override_attributes: apps: os_memory: 1536
  40. 40. @seanwalberg DO YOU SERVE GLUTEN FREE? Yes, you can order off the menu. But we need to talk, and it’ll take longer. How many conventions are we changing?
  41. 41. @seanwalberg RETROSPECTIVE • Config evolved, but worked great • Standardized apps meant no need for dev box • Devs on servers virtually eliminated • Time for a new service is ~30min • Deploy anytime! • Conversations became higher value
  42. 42. @seanwalberg WHAT ELSE CAN WE AUTOMATE? • DNS • AppD configuration • Builds and deploys • Fastly (CDN) configuration
  43. 43. @seanwalberg The code needs to be the source of truth Configuration Transformation API driven deploy
  44. 44. @seanwalberg This becomes VCL This is a functional test CHEF TAUGHT US THAT CONFIGURATION DOESN’T HAVE TO SUCK
  45. 45. @seanwalberg This is so much better than that
  46. 46. @seanwalberg DON’T AUTOMATE THE CONVERSATIONS AWAY
  47. 47. @seanwalberg NOW, LET’S GET BETTER AT USING CHEF Again, conversations will help us.
  48. 48. @seanwalberg CONTINUOUS INTEGRATION “Does it converge?”
  49. 49. @seanwalberg PEER REVIEW
  50. 50. @seanwalberg AUTO LINTING
  51. 51. @seanwalberg ESTABLISH A WORKFLOW
  52. 52. @seanwalberg ALSO REMEMBER You’re coding for the most inexperienced member of your team AKA the “the new person”
  53. 53. @seanwalberg LET’S WRAP • Think about conversations • Chef is more than just configs on servers • Reduce complexity with a menu • Apply the “Chef Way” elsewhere • Keep improving!
  54. 54. @seanwalberg THANKS! Sean Walberg <sean@ertw.com> @seanwalberg

×