@seanwalberg
BREAKING TECHNOLOGY
SILOS WITH CHEF
Sean Walberg <sean@ertw.com>
Infrastructure guy
National Football League

@seanwalberg
These are silos
(Every DevOps presentation needs a picture of a silo)

@seanwalberg
Yup

@seanwalberg
Tech view

@seanwalberg
People view

@seanwalberg
How do you change a
culture?

@seanwalberg
BLUF
Optimize for conversations – automate
the bad ones away
Make a menu – Do one thing, and do it
well.
Don’t neglect your team

@seanwalberg
Start Development
Go faster!
STOP!

@seanwalberg
OFF SEASON, 2014

@seanwalberg

@seanwalberg
LET’S FIX SOME PROBLEMS
• Environmental drift
• Configuration files
• Developer access to servers

@seanwalberg
INTER/INTRA ENVIRONMENT DRIFT

@seanwalberg
AUTOMATE ALL THE THINGS!

@seanwalberg
NFL NOW ENVIRONMENT
• About a dozen services and ~100 servers by the
end in production, ~200 in total
• Cookbook per app
• Environment settings in Chef environment

@seanwalberg
CONFIGURATION FILES
http://www.mommysbusy.com/wp-content/uploads/2013/10/Messy-Desk.jpg

@seanwalberg
CAN WE FIX IT? YES WE CAN!
Developer modifies config on dev server
Infrastructure team diffs and templates
Other environment settings are added to cookbook or environment
Deploy to staging/preview/production

@seanwalberg
http://6iee.com/372028.html
DEVELOPER ACCESS TO SERVERS

@seanwalberg
• Development only
• Except when we’re in a crunch
• And then we turn off Chef
• Then I spend hours cleaning it up
DEVELOPER ACCESS TO SERVERS

@seanwalberg
KNIFE-VSPHERE
• Clone + Bootstrap VM
• Take snapshots
• Resize/add disk
• Change settings
• Run commands

@seanwalberg
DURING THE SEASON

@seanwalberg
NODE, ADD THYSELF TO A POOL!
f5_pool ”pool_name" do
host node['fqdn']
ip node['ipaddress']
port app_port
not_if { node['apps']['skip_load_balancer’]}
end

@seanwalberg
START ROLLING OUT CHEF
• role[base] – everything new
• LDAP
• Access control
• Base packages
• role[minimal] – just get Chef on it
• Base packages
• SSH keys

@seanwalberg
RETROSPECTIVE
• Chef + knife-vsphere + f5 worked great
• Still many manual steps though
• Why do developers still need access to servers?
• Can we manage config files asynchronously?
• No standardization in frameworks.

@seanwalberg
OFF SEASON, 2015
Microservices, yay!

@seanwalberg
I REALLY WANT TO FIX THIS STUFF
• Developers on servers
• Standardization of frameworks
• Config files

@seanwalberg
LET’S HAVE SOME CONVERSATIONS
• “Is the config on the box what it should be?”
• “I need to see the logs!”
• “This framework looks cool, maybe I’ll try it.”
• “I don’t trust what you’re telling me.”
• “I didn’t know that makes your life harder.”

@seanwalberg
THEY’RE ALL RELATED!

@seanwalberg
WE’RE A RESTAURANT AND WE
HAVE ONE DISH

@seanwalberg
LE MENU
• Everything is built around the name of the app
• Start with a consistent build pipeline
• Start projects from a template
• Log with slf4j, we’ll config graylog for you
• Deployable fat JAR (for Java stuff)
• Instrumentation is added on the server
• One chef recipe

@seanwalberg
CONSISTENT NAMING
Tags: [app:sso]
Runlist: recipe[nfl-apps]
Server: locenvssozz
Service: sso
Logs: /var/log/sso/
Deployable: /opt/nfl/sso/sso.jar
Graylog: tag logs with app:sso
AppD Tier: sso
Repo: sso

@seanwalberg
LOGGING
template "/opt/nfl/…/logback.xml” do
source 'logback.xml.erb'
owner 'root'
mode '755'
variables(
app_service: app,
instance_id: instance_id,
environment: node.chef_environment
)
end

@seanwalberg
ADD SOME CONTEXT TO LOGS
<staticAdditionalField>_app:<%= @app_service %></staticAdditionalField>
<staticAdditionalField>_env:<%= @environment %></staticAdditionalField>
<staticAdditionalField>_instance:<%= @instance_id %></staticAdditionalField>

@seanwalberg
EASY CHANGING OF LOGGING
<!-- Shortcut to debug -->
<% node['apps']['debug_packages'].each do |package| %>
<logger name="<%= package %>" level="debug" />
<% end %>
<!-- Fine tuning -->
<% if node['apps'].key? 'loglevel’
node['apps']['loglevel'].each do |package, level| %>
<logger name="<%= package %>" level="<%= level %>" />
<% end; end %>
<!-- Default logging based on tag -->
<logger name="com.nfl.dm.<%= @app_service %>" level="info" />

@seanwalberg
APPDYNAMICS (APM)
Install agents
Configure agents based on attributes
No knowledge of app
Include_recipe “nfl-appdynamics”
Adjust startup scripts
Handle custom AppD config
Legacy stuff
recipe[nfl-appdynamics::agent]
Fix your own startup scripts and custom config

@seanwalberg
LET’S FIX CONFIGURATION
• Chef drops a consul agent on each server
• Joins it to the cluster for that environment
• Sets startup scripts for the app to tell the app where to
find consul, and the configs within the KV store
• Starter template provides a module that reads Consul on
startup and configs Spring Boot
• Config is in a repo that anyone can use
I would be lying if I said we got this
right on the first try!

@seanwalberg
SAME PATTERN AS BEFORE
Install consul
Join to #{node.chef_environment}
No knowledge of app
Include_recipe “nfl-consul”
Adjust startup scripts
Populate service discovery values
Legacy stuff
Recipe[nfl-consul]
Fix your own startup scripts

@seanwalberg
USING CONFIG
• Developers commit to Consulation (YAML)
• Peer review + linting/smoketest in Phabricator
• Secrets in Vault
• Auto deploy to environments on merge
The Rules
1. There is no other config but Consulation
2. If it changes meaning, change the key name
3. If you don’t know what it is, ignore it

@seanwalberg
ATTRIBUTES ARE A GOOD
INTERFACE
default['apps']['os_memory'] = '1024’
if jvm_memory is not overridden
system_memory = memory_from_ohai
jvm_memory = system_memory – os_memory
end
# Wrapper cookbook can tweak either jvm memory or os_memory

@seanwalberg
OR USE A ROLE
$ knife role show os1536_memory_adjustment
chef_type role
name: os1536_memory_adjustment
override_attributes:
apps:
os_memory: 1536

@seanwalberg
DO YOU SERVE GLUTEN FREE?
Yes, you can order off the menu.
But we need to talk, and it’ll take longer.
How many conventions are we changing?

@seanwalberg
RETROSPECTIVE
• Config evolved, but worked great
• Standardized apps meant no need for dev box
• Devs on servers virtually eliminated
• Time for a new service is ~30min
• Deploy anytime!
• Conversations became higher value

@seanwalberg
WHAT ELSE CAN WE AUTOMATE?
• DNS
• AppD configuration
• Builds and deploys
• Fastly (CDN) configuration

@seanwalberg
The code needs to be the
source of truth
Configuration Transformation
API driven deploy

@seanwalberg
This becomes VCL
This is a functional
test
CHEF TAUGHT US THAT CONFIGURATION
DOESN’T HAVE TO SUCK

@seanwalberg
This is so
much better
than that

@seanwalberg
DON’T AUTOMATE THE
CONVERSATIONS AWAY

@seanwalberg
NOW, LET’S GET BETTER AT
USING CHEF
Again, conversations will help us.

@seanwalberg
CONTINUOUS INTEGRATION
“Does it converge?”

@seanwalberg
PEER REVIEW

@seanwalberg
AUTO LINTING

@seanwalberg
ESTABLISH A WORKFLOW

@seanwalberg
ALSO REMEMBER
You’re coding for the most
inexperienced member of your team
AKA the “the new person”

@seanwalberg
LET’S WRAP
• Think about conversations
• Chef is more than just configs on servers
• Reduce complexity with a menu
• Apply the “Chef Way” elsewhere
• Keep improving!

@seanwalberg
THANKS!
Sean Walberg <sean@ertw.com>
@seanwalberg