SlideShare a Scribd company logo
1 of 54
Download to read offline
@seanwalberg
BREAKING TECHNOLOGY
SILOS WITH CHEF
Sean Walberg <sean@ertw.com>
Infrastructure guy
National Football League
@seanwalberg
These are silos
(Every DevOps presentation needs a picture of a silo)
@seanwalberg
Yup
@seanwalberg
Tech view
@seanwalberg
People view
@seanwalberg
How do you change a
culture?
@seanwalberg
BLUF
Optimize for conversations – automate
the bad ones away
Make a menu – Do one thing, and do it
well.
Don’t neglect your team
@seanwalberg
Start Development
Go faster!
STOP!
@seanwalberg
OFF SEASON, 2014
@seanwalberg
@seanwalberg
LET’S FIX SOME PROBLEMS
• Environmental drift
• Configuration files
• Developer access to servers
@seanwalberg
INTER/INTRA ENVIRONMENT DRIFT
@seanwalberg
AUTOMATE ALL THE THINGS!
@seanwalberg
NFL NOW ENVIRONMENT
• About a dozen services and ~100 servers by the
end in production, ~200 in total
• Cookbook per app
• Environment settings in Chef environment
@seanwalberg
CONFIGURATION FILES
http://www.mommysbusy.com/wp-content/uploads/2013/10/Messy-Desk.jpg
@seanwalberg
CAN WE FIX IT? YES WE CAN!
Developer modifies config on dev server
Infrastructure team diffs and templates
Other environment settings are added to cookbook or environment
Deploy to staging/preview/production
@seanwalberg
http://6iee.com/372028.html
DEVELOPER ACCESS TO SERVERS
@seanwalberg
• Development only
• Except when we’re in a crunch
• And then we turn off Chef
• Then I spend hours cleaning it up
DEVELOPER ACCESS TO SERVERS
@seanwalberg
KNIFE-VSPHERE
• Clone + Bootstrap VM
• Take snapshots
• Resize/add disk
• Change settings
• Run commands
@seanwalberg
DURING THE SEASON
@seanwalberg
NODE, ADD THYSELF TO A POOL!
f5_pool ”pool_name" do
host node['fqdn']
ip node['ipaddress']
port app_port
not_if { node['apps']['skip_load_balancer’]}
end
@seanwalberg
START ROLLING OUT CHEF
• role[base] – everything new
• LDAP
• Access control
• Base packages
• role[minimal] – just get Chef on it
• Base packages
• SSH keys
@seanwalberg
RETROSPECTIVE
• Chef + knife-vsphere + f5 worked great
• Still many manual steps though
• Why do developers still need access to servers?
• Can we manage config files asynchronously?
• No standardization in frameworks.
@seanwalberg
OFF SEASON, 2015
Microservices, yay!
@seanwalberg
I REALLY WANT TO FIX THIS STUFF
• Developers on servers
• Standardization of frameworks
• Config files
@seanwalberg
LET’S HAVE SOME CONVERSATIONS
• “Is the config on the box what it should be?”
• “I need to see the logs!”
• “This framework looks cool, maybe I’ll try it.”
• “I don’t trust what you’re telling me.”
• “I didn’t know that makes your life harder.”
@seanwalberg
THEY’RE ALL RELATED!
@seanwalberg
WE’RE A RESTAURANT AND WE
HAVE ONE DISH
@seanwalberg
LE MENU
• Everything is built around the name of the app
• Start with a consistent build pipeline
• Start projects from a template
• Log with slf4j, we’ll config graylog for you
• Deployable fat JAR (for Java stuff)
• Instrumentation is added on the server
• One chef recipe
@seanwalberg
CONSISTENT NAMING
Tags: [app:sso]
Runlist: recipe[nfl-apps]
Server: locenvssozz
Service: sso
Logs: /var/log/sso/
Deployable: /opt/nfl/sso/sso.jar
Graylog: tag logs with app:sso
AppD Tier: sso
Repo: sso
@seanwalberg
LOGGING
template "/opt/nfl/…/logback.xml” do
source 'logback.xml.erb'
owner 'root'
mode '755'
variables(
app_service: app,
instance_id: instance_id,
environment: node.chef_environment
)
end
@seanwalberg
ADD SOME CONTEXT TO LOGS
<staticAdditionalField>_app:<%= @app_service %></staticAdditionalField>
<staticAdditionalField>_env:<%= @environment %></staticAdditionalField>
<staticAdditionalField>_instance:<%= @instance_id %></staticAdditionalField>
@seanwalberg
EASY CHANGING OF LOGGING
<!-- Shortcut to debug -->
<% node['apps']['debug_packages'].each do |package| %>
<logger name="<%= package %>" level="debug" />
<% end %>
<!-- Fine tuning -->
<% if node['apps'].key? 'loglevel’
node['apps']['loglevel'].each do |package, level| %>
<logger name="<%= package %>" level="<%= level %>" />
<% end; end %>
<!-- Default logging based on tag -->
<logger name="com.nfl.dm.<%= @app_service %>" level="info" />
@seanwalberg
APPDYNAMICS (APM)
Install agents
Configure agents based on attributes
No knowledge of app
Include_recipe “nfl-appdynamics”
Adjust startup scripts
Handle custom AppD config
Legacy stuff
recipe[nfl-appdynamics::agent]
Fix your own startup scripts and custom config
@seanwalberg
LET’S FIX CONFIGURATION
• Chef drops a consul agent on each server
• Joins it to the cluster for that environment
• Sets startup scripts for the app to tell the app where to
find consul, and the configs within the KV store
• Starter template provides a module that reads Consul on
startup and configs Spring Boot
• Config is in a repo that anyone can use
I would be lying if I said we got this
right on the first try!
@seanwalberg
SAME PATTERN AS BEFORE
Install consul
Join to #{node.chef_environment}
No knowledge of app
Include_recipe “nfl-consul”
Adjust startup scripts
Populate service discovery values
Legacy stuff
Recipe[nfl-consul]
Fix your own startup scripts
@seanwalberg
USING CONFIG
• Developers commit to Consulation (YAML)
• Peer review + linting/smoketest in Phabricator
• Secrets in Vault
• Auto deploy to environments on merge
The Rules
1. There is no other config but Consulation
2. If it changes meaning, change the key name
3. If you don’t know what it is, ignore it
@seanwalberg
ATTRIBUTES ARE A GOOD
INTERFACE
default['apps']['os_memory'] = '1024’
if jvm_memory is not overridden
system_memory = memory_from_ohai
jvm_memory = system_memory – os_memory
end
# Wrapper cookbook can tweak either jvm memory or os_memory
@seanwalberg
OR USE A ROLE
$ knife role show os1536_memory_adjustment
chef_type role
name: os1536_memory_adjustment
override_attributes:
apps:
os_memory: 1536
@seanwalberg
DO YOU SERVE GLUTEN FREE?
Yes, you can order off the menu.
But we need to talk, and it’ll take longer.
How many conventions are we changing?
@seanwalberg
RETROSPECTIVE
• Config evolved, but worked great
• Standardized apps meant no need for dev box
• Devs on servers virtually eliminated
• Time for a new service is ~30min
• Deploy anytime!
• Conversations became higher value
@seanwalberg
WHAT ELSE CAN WE AUTOMATE?
• DNS
• AppD configuration
• Builds and deploys
• Fastly (CDN) configuration
@seanwalberg
The code needs to be the
source of truth
Configuration Transformation
API driven deploy
@seanwalberg
This becomes VCL
This is a functional
test
CHEF TAUGHT US THAT CONFIGURATION
DOESN’T HAVE TO SUCK
@seanwalberg
This is so
much better
than that
@seanwalberg
DON’T AUTOMATE THE
CONVERSATIONS AWAY
@seanwalberg
NOW, LET’S GET BETTER AT
USING CHEF
Again, conversations will help us.
@seanwalberg
CONTINUOUS INTEGRATION
“Does it converge?”
@seanwalberg
PEER REVIEW
@seanwalberg
AUTO LINTING
@seanwalberg
ESTABLISH A WORKFLOW
@seanwalberg
ALSO REMEMBER
You’re coding for the most
inexperienced member of your team
AKA the “the new person”
@seanwalberg
LET’S WRAP
• Think about conversations
• Chef is more than just configs on servers
• Reduce complexity with a menu
• Apply the “Chef Way” elsewhere
• Keep improving!
@seanwalberg
THANKS!
Sean Walberg <sean@ertw.com>
@seanwalberg

More Related Content

What's hot

What Makes a Good Cookbook?
What Makes a Good Cookbook?What Makes a Good Cookbook?
What Makes a Good Cookbook?Julian Dunn
 
Managing Servers with Chef
Managing Servers with ChefManaging Servers with Chef
Managing Servers with ChefJoe Kepley
 
A Simple 8-Step Guide to Setting Up a Dev Shop
A Simple 8-Step Guide to Setting Up a Dev ShopA Simple 8-Step Guide to Setting Up a Dev Shop
A Simple 8-Step Guide to Setting Up a Dev ShopScott Porad
 
Avoiding integration hell
Avoiding integration hellAvoiding integration hell
Avoiding integration hellaaronbassett
 
Premature optimisation: The Root of All Evil
Premature optimisation: The Root of All EvilPremature optimisation: The Root of All Evil
Premature optimisation: The Root of All EvilFabio Akita
 
Customizing Chef for Fun and Profit
Customizing Chef for Fun and ProfitCustomizing Chef for Fun and Profit
Customizing Chef for Fun and ProfitJon Cowie
 
Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...
Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...
Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...Simplilearn
 
Large web applications development
Large web applications developmentLarge web applications development
Large web applications developmentTaras Omelianenko
 
ITB2016 - Mixing up the front end with ColdBox elixir
ITB2016 - Mixing up the front end with ColdBox elixirITB2016 - Mixing up the front end with ColdBox elixir
ITB2016 - Mixing up the front end with ColdBox elixirOrtus Solutions, Corp
 
Chef Actions: Delightful near real-time activity tracking!
Chef Actions: Delightful near real-time activity tracking!Chef Actions: Delightful near real-time activity tracking!
Chef Actions: Delightful near real-time activity tracking!James Casey
 
Putting the 'ctions' in Azure Fun-ctions
Putting the 'ctions' in Azure Fun-ctionsPutting the 'ctions' in Azure Fun-ctions
Putting the 'ctions' in Azure Fun-ctionsNullOps
 
Chef Intro @ SF Bay Area LSPE meetup
Chef Intro @ SF Bay Area LSPE meetupChef Intro @ SF Bay Area LSPE meetup
Chef Intro @ SF Bay Area LSPE meetupmdxp
 
Testing CSS - Front end ops by Arpit Maheshwari
Testing CSS - Front end ops   by Arpit MaheshwariTesting CSS - Front end ops   by Arpit Maheshwari
Testing CSS - Front end ops by Arpit MaheshwariArpit Maheshwari
 
Chef Tutorial for DEVOPS Newbies
Chef Tutorial for DEVOPS NewbiesChef Tutorial for DEVOPS Newbies
Chef Tutorial for DEVOPS NewbiesMamun Rashid, CCDH
 
Performance Tuning Azure SQL Database
Performance Tuning Azure SQL DatabasePerformance Tuning Azure SQL Database
Performance Tuning Azure SQL DatabaseGrant Fritchey
 
CI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page AppsCI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page AppsMike North
 
WTF is Sensu and Monitoring
WTF is Sensu and MonitoringWTF is Sensu and Monitoring
WTF is Sensu and MonitoringToby Jackson
 

What's hot (20)

What Makes a Good Cookbook?
What Makes a Good Cookbook?What Makes a Good Cookbook?
What Makes a Good Cookbook?
 
Managing Servers with Chef
Managing Servers with ChefManaging Servers with Chef
Managing Servers with Chef
 
A Simple 8-Step Guide to Setting Up a Dev Shop
A Simple 8-Step Guide to Setting Up a Dev ShopA Simple 8-Step Guide to Setting Up a Dev Shop
A Simple 8-Step Guide to Setting Up a Dev Shop
 
Avoiding integration hell
Avoiding integration hellAvoiding integration hell
Avoiding integration hell
 
Premature optimisation: The Root of All Evil
Premature optimisation: The Root of All EvilPremature optimisation: The Root of All Evil
Premature optimisation: The Root of All Evil
 
Customizing Chef for Fun and Profit
Customizing Chef for Fun and ProfitCustomizing Chef for Fun and Profit
Customizing Chef for Fun and Profit
 
Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...
Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...
Chef Tutorial | Chef Tutorial For Beginners | DevOps Chef Tutorial | DevOps T...
 
Large web applications development
Large web applications developmentLarge web applications development
Large web applications development
 
ITB2016 - Mixing up the front end with ColdBox elixir
ITB2016 - Mixing up the front end with ColdBox elixirITB2016 - Mixing up the front end with ColdBox elixir
ITB2016 - Mixing up the front end with ColdBox elixir
 
Ansible ALLTHETHINGS
Ansible ALLTHETHINGSAnsible ALLTHETHINGS
Ansible ALLTHETHINGS
 
Testing puppet
Testing puppetTesting puppet
Testing puppet
 
DevOp with Me!
DevOp with Me!DevOp with Me!
DevOp with Me!
 
Chef Actions: Delightful near real-time activity tracking!
Chef Actions: Delightful near real-time activity tracking!Chef Actions: Delightful near real-time activity tracking!
Chef Actions: Delightful near real-time activity tracking!
 
Putting the 'ctions' in Azure Fun-ctions
Putting the 'ctions' in Azure Fun-ctionsPutting the 'ctions' in Azure Fun-ctions
Putting the 'ctions' in Azure Fun-ctions
 
Chef Intro @ SF Bay Area LSPE meetup
Chef Intro @ SF Bay Area LSPE meetupChef Intro @ SF Bay Area LSPE meetup
Chef Intro @ SF Bay Area LSPE meetup
 
Testing CSS - Front end ops by Arpit Maheshwari
Testing CSS - Front end ops   by Arpit MaheshwariTesting CSS - Front end ops   by Arpit Maheshwari
Testing CSS - Front end ops by Arpit Maheshwari
 
Chef Tutorial for DEVOPS Newbies
Chef Tutorial for DEVOPS NewbiesChef Tutorial for DEVOPS Newbies
Chef Tutorial for DEVOPS Newbies
 
Performance Tuning Azure SQL Database
Performance Tuning Azure SQL DatabasePerformance Tuning Azure SQL Database
Performance Tuning Azure SQL Database
 
CI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page AppsCI/CD and Asset Serving for Single Page Apps
CI/CD and Asset Serving for Single Page Apps
 
WTF is Sensu and Monitoring
WTF is Sensu and MonitoringWTF is Sensu and Monitoring
WTF is Sensu and Monitoring
 

Similar to Breaking Technology Silos with Chef

Building a PWA - For Everyone Who Is Scared To
Building a PWA - For Everyone Who Is Scared ToBuilding a PWA - For Everyone Who Is Scared To
Building a PWA - For Everyone Who Is Scared ToRaymond Camden
 
Cookbook Reusability @ Chef Community summit 2014
Cookbook Reusability @ Chef Community summit 2014Cookbook Reusability @ Chef Community summit 2014
Cookbook Reusability @ Chef Community summit 2014Sean OMeara
 
Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...
Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...
Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...Amazon Web Services
 
Environments - Fundamentals Webinar Series Week 5
Environments - Fundamentals Webinar Series Week 5Environments - Fundamentals Webinar Series Week 5
Environments - Fundamentals Webinar Series Week 5Chef
 
Scaling and Distributing
Scaling and DistributingScaling and Distributing
Scaling and DistributingDima Malenko
 
Chef Cookbook Design Patterns
Chef Cookbook Design PatternsChef Cookbook Design Patterns
Chef Cookbook Design PatternsEric Krupnik
 
Chef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Software, Inc.
 
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...Chef Software, Inc.
 
Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable productJulian Simpson
 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupDATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupSaewoong Lee
 
Introduction to Chef: Automate Your Infrastructure by Modeling It In Code
Introduction to Chef: Automate Your Infrastructure by Modeling It In CodeIntroduction to Chef: Automate Your Infrastructure by Modeling It In Code
Introduction to Chef: Automate Your Infrastructure by Modeling It In CodeJosh Padnick
 
03 ansible towerbestpractices-nicholas
03 ansible towerbestpractices-nicholas03 ansible towerbestpractices-nicholas
03 ansible towerbestpractices-nicholasKhairul Zebua
 
AWS Webcast - AWS OpsWorks Continuous Integration Demo
AWS Webcast - AWS OpsWorks Continuous Integration Demo  AWS Webcast - AWS OpsWorks Continuous Integration Demo
AWS Webcast - AWS OpsWorks Continuous Integration Demo Amazon Web Services
 
Using Chef and Vagrant at Gengo
Using Chef and Vagrant at GengoUsing Chef and Vagrant at Gengo
Using Chef and Vagrant at GengoGengo
 

Similar to Breaking Technology Silos with Chef (20)

Full-Stack CakePHP Deployment
Full-Stack CakePHP DeploymentFull-Stack CakePHP Deployment
Full-Stack CakePHP Deployment
 
Building a PWA - For Everyone Who Is Scared To
Building a PWA - For Everyone Who Is Scared ToBuilding a PWA - For Everyone Who Is Scared To
Building a PWA - For Everyone Who Is Scared To
 
Into the Box 2018 Building a PWA
Into the Box 2018 Building a PWA Into the Box 2018 Building a PWA
Into the Box 2018 Building a PWA
 
Cookbook Reusability @ Chef Community summit 2014
Cookbook Reusability @ Chef Community summit 2014Cookbook Reusability @ Chef Community summit 2014
Cookbook Reusability @ Chef Community summit 2014
 
Chef Jumpstart
Chef JumpstartChef Jumpstart
Chef Jumpstart
 
Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...
Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...
Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:I...
 
Environments - Fundamentals Webinar Series Week 5
Environments - Fundamentals Webinar Series Week 5Environments - Fundamentals Webinar Series Week 5
Environments - Fundamentals Webinar Series Week 5
 
Scaling and Distributing
Scaling and DistributingScaling and Distributing
Scaling and Distributing
 
Chef Cookbook Design Patterns
Chef Cookbook Design PatternsChef Cookbook Design Patterns
Chef Cookbook Design Patterns
 
Chef Cookbook Workflow
Chef Cookbook WorkflowChef Cookbook Workflow
Chef Cookbook Workflow
 
Chef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of Chef
 
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
 
Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable product
 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupDATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backup
 
Introduction to Chef: Automate Your Infrastructure by Modeling It In Code
Introduction to Chef: Automate Your Infrastructure by Modeling It In CodeIntroduction to Chef: Automate Your Infrastructure by Modeling It In Code
Introduction to Chef: Automate Your Infrastructure by Modeling It In Code
 
Continuous database deployment
Continuous database deploymentContinuous database deployment
Continuous database deployment
 
03 ansible towerbestpractices-nicholas
03 ansible towerbestpractices-nicholas03 ansible towerbestpractices-nicholas
03 ansible towerbestpractices-nicholas
 
Django Girls Tutorial
Django Girls TutorialDjango Girls Tutorial
Django Girls Tutorial
 
AWS Webcast - AWS OpsWorks Continuous Integration Demo
AWS Webcast - AWS OpsWorks Continuous Integration Demo  AWS Webcast - AWS OpsWorks Continuous Integration Demo
AWS Webcast - AWS OpsWorks Continuous Integration Demo
 
Using Chef and Vagrant at Gengo
Using Chef and Vagrant at GengoUsing Chef and Vagrant at Gengo
Using Chef and Vagrant at Gengo
 

Breaking Technology Silos with Chef

Editor's Notes

  1. Different approaches to config
  2. Chef, ACLs, disabling Chef, environment drift
  3. Devs on servers virtually eliminated because of standardization and trust/conversations)
  4. Tempting to make everything automated Add artificial constraints to force conversations Don’t build environments/servers automatically otherwise you miss that conversation about what it does and how we’re going to run it
  5. Peer review for knowledge sharing and improving each other’s skills Automatic running of tests where possible and