More Related Content


Breaking Technology Silos with Chef

  1. @seanwalberg BREAKING TECHNOLOGY SILOS WITH CHEF Sean Walberg <> Infrastructure guy National Football League
  2. @seanwalberg These are silos (Every DevOps presentation needs a picture of a silo)
  3. @seanwalberg Yup
  4. @seanwalberg Tech view
  5. @seanwalberg People view
  6. @seanwalberg How do you change a culture?
  7. @seanwalberg BLUF Optimize for conversations – automate the bad ones away Make a menu – Do one thing, and do it well. Don’t neglect your team
  8. @seanwalberg Start Development Go faster! STOP!
  9. @seanwalberg OFF SEASON, 2014
  10. @seanwalberg
  11. @seanwalberg LET’S FIX SOME PROBLEMS • Environmental drift • Configuration files • Developer access to servers
  13. @seanwalberg AUTOMATE ALL THE THINGS!
  14. @seanwalberg NFL NOW ENVIRONMENT • About a dozen services and ~100 servers by the end in production, ~200 in total • Cookbook per app • Environment settings in Chef environment
  15. @seanwalberg CONFIGURATION FILES
  16. @seanwalberg CAN WE FIX IT? YES WE CAN! Developer modifies config on dev server Infrastructure team diffs and templates Other environment settings are added to cookbook or environment Deploy to staging/preview/production
  18. @seanwalberg • Development only • Except when we’re in a crunch • And then we turn off Chef • Then I spend hours cleaning it up DEVELOPER ACCESS TO SERVERS
  19. @seanwalberg KNIFE-VSPHERE • Clone + Bootstrap VM • Take snapshots • Resize/add disk • Change settings • Run commands
  20. @seanwalberg DURING THE SEASON
  21. @seanwalberg NODE, ADD THYSELF TO A POOL! f5_pool ”pool_name" do host node['fqdn'] ip node['ipaddress'] port app_port not_if { node['apps']['skip_load_balancer’]} end
  22. @seanwalberg START ROLLING OUT CHEF • role[base] – everything new • LDAP • Access control • Base packages • role[minimal] – just get Chef on it • Base packages • SSH keys
  23. @seanwalberg RETROSPECTIVE • Chef + knife-vsphere + f5 worked great • Still many manual steps though • Why do developers still need access to servers? • Can we manage config files asynchronously? • No standardization in frameworks.
  24. @seanwalberg OFF SEASON, 2015 Microservices, yay!
  25. @seanwalberg I REALLY WANT TO FIX THIS STUFF • Developers on servers • Standardization of frameworks • Config files
  26. @seanwalberg LET’S HAVE SOME CONVERSATIONS • “Is the config on the box what it should be?” • “I need to see the logs!” • “This framework looks cool, maybe I’ll try it.” • “I don’t trust what you’re telling me.” • “I didn’t know that makes your life harder.”
  27. @seanwalberg THEY’RE ALL RELATED!
  29. @seanwalberg LE MENU • Everything is built around the name of the app • Start with a consistent build pipeline • Start projects from a template • Log with slf4j, we’ll config graylog for you • Deployable fat JAR (for Java stuff) • Instrumentation is added on the server • One chef recipe
  30. @seanwalberg CONSISTENT NAMING Tags: [app:sso] Runlist: recipe[nfl-apps] Server: locenvssozz Service: sso Logs: /var/log/sso/ Deployable: /opt/nfl/sso/sso.jar Graylog: tag logs with app:sso AppD Tier: sso Repo: sso
  31. @seanwalberg LOGGING template "/opt/nfl/…/logback.xml” do source 'logback.xml.erb' owner 'root' mode '755' variables( app_service: app, instance_id: instance_id, environment: node.chef_environment ) end
  32. @seanwalberg ADD SOME CONTEXT TO LOGS <staticAdditionalField>_app:<%= @app_service %></staticAdditionalField> <staticAdditionalField>_env:<%= @environment %></staticAdditionalField> <staticAdditionalField>_instance:<%= @instance_id %></staticAdditionalField>
  33. @seanwalberg EASY CHANGING OF LOGGING <!-- Shortcut to debug --> <% node['apps']['debug_packages'].each do |package| %> <logger name="<%= package %>" level="debug" /> <% end %> <!-- Fine tuning --> <% if node['apps'].key? 'loglevel’ node['apps']['loglevel'].each do |package, level| %> <logger name="<%= package %>" level="<%= level %>" /> <% end; end %> <!-- Default logging based on tag --> <logger name="<%= @app_service %>" level="info" />
  34. @seanwalberg APPDYNAMICS (APM) Install agents Configure agents based on attributes No knowledge of app Include_recipe “nfl-appdynamics” Adjust startup scripts Handle custom AppD config Legacy stuff recipe[nfl-appdynamics::agent] Fix your own startup scripts and custom config
  35. @seanwalberg LET’S FIX CONFIGURATION • Chef drops a consul agent on each server • Joins it to the cluster for that environment • Sets startup scripts for the app to tell the app where to find consul, and the configs within the KV store • Starter template provides a module that reads Consul on startup and configs Spring Boot • Config is in a repo that anyone can use I would be lying if I said we got this right on the first try!
  36. @seanwalberg SAME PATTERN AS BEFORE Install consul Join to #{node.chef_environment} No knowledge of app Include_recipe “nfl-consul” Adjust startup scripts Populate service discovery values Legacy stuff Recipe[nfl-consul] Fix your own startup scripts
  37. @seanwalberg USING CONFIG • Developers commit to Consulation (YAML) • Peer review + linting/smoketest in Phabricator • Secrets in Vault • Auto deploy to environments on merge The Rules 1. There is no other config but Consulation 2. If it changes meaning, change the key name 3. If you don’t know what it is, ignore it
  38. @seanwalberg ATTRIBUTES ARE A GOOD INTERFACE default['apps']['os_memory'] = '1024’ if jvm_memory is not overridden system_memory = memory_from_ohai jvm_memory = system_memory – os_memory end # Wrapper cookbook can tweak either jvm memory or os_memory
  39. @seanwalberg OR USE A ROLE $ knife role show os1536_memory_adjustment chef_type role name: os1536_memory_adjustment override_attributes: apps: os_memory: 1536
  40. @seanwalberg DO YOU SERVE GLUTEN FREE? Yes, you can order off the menu. But we need to talk, and it’ll take longer. How many conventions are we changing?
  41. @seanwalberg RETROSPECTIVE • Config evolved, but worked great • Standardized apps meant no need for dev box • Devs on servers virtually eliminated • Time for a new service is ~30min • Deploy anytime! • Conversations became higher value
  42. @seanwalberg WHAT ELSE CAN WE AUTOMATE? • DNS • AppD configuration • Builds and deploys • Fastly (CDN) configuration
  43. @seanwalberg The code needs to be the source of truth Configuration Transformation API driven deploy
  44. @seanwalberg This becomes VCL This is a functional test CHEF TAUGHT US THAT CONFIGURATION DOESN’T HAVE TO SUCK
  45. @seanwalberg This is so much better than that
  47. @seanwalberg NOW, LET’S GET BETTER AT USING CHEF Again, conversations will help us.
  48. @seanwalberg CONTINUOUS INTEGRATION “Does it converge?”
  49. @seanwalberg PEER REVIEW
  50. @seanwalberg AUTO LINTING
  51. @seanwalberg ESTABLISH A WORKFLOW
  52. @seanwalberg ALSO REMEMBER You’re coding for the most inexperienced member of your team AKA the “the new person”
  53. @seanwalberg LET’S WRAP • Think about conversations • Chef is more than just configs on servers • Reduce complexity with a menu • Apply the “Chef Way” elsewhere • Keep improving!
  54. @seanwalberg THANKS! Sean Walberg <> @seanwalberg

Editor's Notes

  1. Different approaches to config
  2. Chef, ACLs, disabling Chef, environment drift
  3. Devs on servers virtually eliminated because of standardization and trust/conversations)
  4. Tempting to make everything automated Add artificial constraints to force conversations Don’t build environments/servers automatically otherwise you miss that conversation about what it does and how we’re going to run it
  5. Peer review for knowledge sharing and improving each other’s skills Automatic running of tests where possible and