Workflow Design - Extracting Signal from the Noise

Workflow Design
Extracting Signal from the Noise

@jonlives
Jon Cowie
Staff Operations Engineer

@jonlives
There is no correct way to
use Chef.
workﬂow_design

@jonlives
…but I just want to 
get stuff done!

@jonlives
Absorb what is
useful.
Discard what is
useless.

@jonlives
“I am not smart enough to
build an ontology … that
can encompass all the
variations in infrastructure.
Nobody is, the world
moves too fast.”

@jonlives
There is no
magic pill.

@jonlives
Beware Argument Culture!
Kate Heddleston: bit.ly/argument-culture

@jonlives
[An] argument culture urges us to
approach the world—and the
people in it—in an adversarial
frame of mind
Deborah Tannen “The Argument Culture”

@jonlives
Argument Culture
• Useful in “regulated” environments like law
• Engineering is not one of those environments
• Weed out weak logic, keep strong ideas
• Everybody is on one side or the other
• We attack their position and defend our own
• We can win or lose arguments—just like battles

@jonlives
Argument Culture - Effects
• Emotional interactions
• Winning becomes paramount
• The loud dominate
• People are afraid to speak
• Best practice == repeating the loudest arguments
• Sucks for diversity in cultures and orgs

@jonlives
What a Workflow is…
workflow
noun ( also work-flow) /ˈwɜːkfləʊ/
the way that a particular type of work is
organised, or the order of the stages in a
particular work process
Cambridge Business English Dictionary

@jonlives
What a Workflow is not…
• “The Chef Way”
• “We need to use X tool”
• “We need to be more DevOps”
• “We need our process to be more like Y”
• “Z in Chef is bad. Never do Z.”

@jonlives
A workflow is the key to
solving a problem.

@jonlives
There is no correct way to
use Chef…

@jonlives
There are only lessons
learned by others in the
process of solving their
specific problems.

@jonlives
Case Studies
• Anonymized
• Companies of varying sizes
• First, a quick summary
• Then we look at their motivations
• Finally we look at their solutions

@jonlives
Case Study #1 - Summary
• Small internet video company
• Small Ops team
• EC2 based infrastructure
• Semi-immutable infrastructure
• Packer used to build AMI containers
• Chef run on instances to correct config & service status
• Previously required rolling a new AMI

@jonlives
Case Study #1 - Motivation
• They’re trying to make it as easy to get started as
possible
• Chat or comments in PRs since everybody is remote
• They aren't interested in letting some third party
change stuff unexpectedly
• No auto-deploying master - they had an accidental
push

@jonlives
Case Study #1 - Solution
• Cookbooks live in chef repo
• Dependencies in Berksfile
• “Pull request” development model
• Merge to master & Jenkins runs foodcritic & rubocop
• Merge into “jenkins” branch, triggers upload to server

@jonlives
• One project team inside extremely large company
• On-premise cloudstack infrastructure
• VMs created, then bootstrapped with Chef
• Dev, QA, Staging, Prod environments
• Legacy cookbooks maintained alongside new
• 1 workflow for new cookbooks, one for legacy

@jonlives
• Too easy to make an innocuous change that broke
several features without knowing about it
• Desire to catch syntax errors or undeclared attributes
• Complete test coverage with minitest, kitchen-ci for
easier PR validation, and local cookbook development

@jonlives
• 1 monolithic project cookbook
• Add dependencies to Berksfile & “berks install”
• Create recipes & write tests
• “kitchen converge” until it works
• Push to git & chef server
• Test in dev, then on to prod

@jonlives
• Mid-sized online marketplace
• Mainly on-premise infrastructure
• Servers often long-lived & not immutable
• Large number of Dev & Ops Chef users
• One main environment

@jonlives
• Many Chefs
• Many small changes by often inexperienced users
• Historical cookbooks make using community
cookbooks difficult & time consuming
• App deployed using separate deploy tool
• Continuous delivery in “DNA”

@jonlives
• Cookbooks live in monolithic repository
• Roles == attribs, environments == constraints
• Tooling geared for efficiency & iteration vs testing
• Testing done locally with chef-zero & why-run
• …or in prod with whitelists
• Versions always pinned in prod environment

@jonlives
• Mid-sized ecommerce service provider
• Mainly on-premise infrastructure
• Multiple product-based teams
• Multiple environments to support those teams

@jonlives
• Teams based around the Products they build and run
• Different opinions & requirements for workflow
• Most effort was spent on new greenfield projects
• Trying to avoid “veneering tech debt”
• Chef was new to most people
• Pressure to produce visible & demonstrable results

@jonlives
• 85% of cookbooks in monolithic repo
• Start with community cookbooks
• Wrap as needed
• Team or Product specific environments
• Roles used to set permissions etc
• Optional cookbook pinning

@jonlives
So, how do we create a
Workflow?

@jonlives
Workflow Design
• What problem are you solving?
• Who are you solving it for?
• How does your ideal solution look?
• Choose your Tooling
• Implement your workflow

@jonlives
What Problem are you Solving?
• Write down a simple problem
statement
• “I want to automate app deploys”
• “I want to automate server builds”
• “I want us to test our cookbooks
more”
• Break that down into simple pieces
What
Who
How
Tooling
Implement

@jonlives
Who are you solving it for?
• Understand your stakeholders
• Understand your business priorities
• What do *they* care about most?
• How often will they work with your
solution?
• Conway’s law
What
Who
How
Tooling
Implement

@jonlives
Conway’s Law
“Organizations which design systems ...
are constrained to produce designs which
are copies of the communication
structures of these organizations”
Melvin Conway, 1968

@jonlives
How does your ideal solution look?
• Have an ideal “end goal” in mind
• Agree with stakeholders
• You won’t get there in one go
• But you’ll know what you’re
working towards
What
Who
How
Tooling
Implement

@jonlives
Choose your Weapons!
• Pick candidate tools
• Probe strengths and weaknesses
• Understand their motivations
• Extensibility if tweaking needed?
• Is anybody already using it in the
same way?
What
Who
How
Tooling
Implement

@jonlives
…or make your own Weapons
• Sometimes, you have to invent
• Legitimate approach
• Many popular workflow tools
started this way
• Be sure you have time / resources
• Clear mission statement is vital
What
Who
How
Tooling
Implement

@jonlives
Implementation
• Have realistic expectations
• Start slowly
• Efficiency-Thoroughness Trade-Off
(ETTO)
• Iterate often
• Learn from failures. They will happen.
• Improve
What
Who
How
Tooling
Implement

@jonlives
Don’t Fear the Code!

@jonlives
Don’t Fear the Code!
• Understanding == Effectiveness
• If you know how it works, you can figure out
why it breaks.
• Research known bugs & limitations
• Be prepared to report or fix issues

@jonlives
It’s OK to try stuff!

@jonlives
http://jonliv.es/book
Discount Code: AUTHD
40% off Print
50% off Digital

@jonlives
Thanks!
Questions?
@jonlives / http://jonliv.es / jon@etsy.com

Workflow Design - Extracting Signal from the Noise

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (11)

More from Jon Cowie

More from Jon Cowie (7)

Recently uploaded

Recently uploaded (20)

Workflow Design - Extracting Signal from the Noise