Talk at ChefConf 2015 on techniques for template-driven deployment of topologies composed of Chef nodes, by John Sweitzer and Christine Draper. Video: https://www.youtube.com/watch?v=hoXf0Uo5bCo

1. 1
Serve Meals,
Not Ingredients
John Sweitzer
john_sweitzer@thirdwaveinsights.com
@tkdsweitz
Christine Draper
christine_draper@thirdwaveinsights.com
@CristineDraper
@3rdwaveinsights
http://www.thirdwaveinsights.com

2. 2
Topology
Meals are topologies, nodes are key ingredients
Nodes
Software on a
Node
Connections
Meal
Ingredients

3. 3
Emulating topologies in Chef starts with grouping nodes
Topology JSON
Minimize collisions
Topology-as-code
Optimized for Chef
Faster iterations
{
"name": "inttest1",
“provisioning” : {
},
"services": [
{
"name": "appserver-elb",
"type": "load_balancer"
}
],
"nodes": [
{
"name": "appserver"
},
{
"name": "dbserver"
}
]}
topo.json
Chef Server
Environments
Data Bags
Cookbooks
Nodes
Chef Server
Nodes Node
Environments
Data Bags
Cookbooks
● Ways to emulate:
– Environments
– Tags
– Node attributes
– Data bags
knife-topo

4. 4
Multi-* Enterprise - This may not be you, exactly, but...
● Multiple lines of business
running business systems
● Multiple deployment across
application pipeline
● Technology Adoption Model
First team First wave Broad adoption Full adoption
Asset
Tracker
Inventory
Customer
Orders
Human
Resources
Application pipeline
Test Stage DeliveryDevelop

5. 5
Jake's charter
More meetings
Less autonomy/control
Endless debates/escalations
Tedious data gathering
Reduce cost
Share skills
Share content
Why increase
consistency?
INCREASE CONSISTENCY ACROSS TEAMS
WITHOUT DECREASING THEIR AGILITY

6. 6
The WORK (and it is work) to increase consistency...
Gather
Organize
Decide
Change
Monitor
Waste Innovation

7. 7
Organize: Topology-Blueprint-Inventory Pattern
BlueprintsPipelineTopologies
Test Stage DeliverDevelop
Inventory
Optimized
Arrangements
Particular
Deployment
Automate.Insights
Organizational
Flow
Bill of
Materials
An IT business system is a collection of IT
resources delivering a business function
● Blueprints evolve
● Topologies are specialized
● Visibility encourages collaboration

8. 8
Decide: What is expected to be common in all deployments?
Software Version Default recipe
mongodb 2.6.5 coolco_mongodb::default
nodeapp 1.0 coolco_nodeapp::deploy_from_
package
nodejs 0.10.29 coolco_nodejs::default
Recipe Inventory
apt
chef_client::config
coolco_handlers::default
coolco_mongodb::default
coolco_nodeapp::deploy_from_package
coolco_nodejs::default
Software Attribute Value
mongodb xxx yyy
xxx yyy
xxx yyy
Software Attribute Value
nodeapp nodeapp.name asset_tracker
nodeapp.user jake
nodeapp.dir /home/automation
RunlistNode
attributes

9. 9
Decide: Look across blueprints...
Automate.
Insights
● Node Types
● Software and Version
● Attribute Values
● Recipes
● Cookbooks
Software attribute
value by blueprint

10. 10
Change – All does not need to change at once
BlueprintsPipelineTopologies
Test Stage DeliverDevelop
Inventory
Optimized
Arrangements
Particular
Deployment
Organizational
Flow
Bill of
Materials
Chef
Server
Actual IT
Resources
Environments
Data Bags
Cookbooks Nodes
Roles
● Prime
● Synchronize
● Local Override
StagedChanges
nodejs 0.10.28 to 0.10.30

11. 11
Gathering and Changing details in Chef Server
knife topo
export YourTopology
node1 node2 node3 …
> YourTopology.json
Topology
JSONChef Server
knife-
topo
Automate.Insights
knife topo
import YourTopology
knife topo
create YourTopology
Create blueprints from nodes
Align topologies with blueprint
Propagate changes to Chef
server
Gather
Organize
Decide
Change
Monitor

12. 12
Fred's Scenario
Meanwhile Shadow IT has sprung up in various
lines of business, using a range of providers
Fred's one of the technical leads who's been
successful bringing consistency across CoolCo's
core business systems...
CoolCo wants Fred to help the Lines of Business
adopt Chef and become more consistent in their
delivery, without impacting their agility

13. 13
Fred meets with one of the lines of
business to understand what they have...
appserver
AMI
appserver
auto scaling groupElastic Load
Balancer
(appserver-
elb)
dbserver
AWS cloud
Launch
Configuration
db url
protocol
& port
instance
protocol
& port
Resources": {
"elbappserverelb": {
"Type": "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties": {
"AvailabilityZones": [
"us-west-2c", "us-west-2b", "us-west-2a"
],
"Listeners": [
{
"InstancePort": "3001",
"LoadBalancerPort": "3001",
"Protocol": "HTTP",
"InstanceProtocol": "HTTP"
}
]}
},
...
Create multiple test topologies
rapidly & consistently
Delete topologies when done
CloudFormation
Template
CloudFormation
Stacks
CloudFormation
AWS Icons: http://aws.amazon.com/architecture/icons/
Hook-Em... a recently-
developed customer
conversion system

14. 14
… and what their challenges are
app
+
nodejs
mongo
db
application &
middleware
"DBServer": {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bashn",
"yum update -y aws-cfn-bootstrapn",
<snip 20 lines>
"yum -y install mongo-10gen-server > /tmp/yum-mongo.log 2>&1n",
<snip 50 lines>
"/etc/init.d/mongod start > /tmp/mongod-start.log 2>&1n",
<snip 20 lines>
"/opt/aws/bin/cfn-signal -e 0 -r "Mongo setup complete" '", { "Ref" :
"WaitHandleMongo" }, "'n"
] ] } },
...
}
...
}}}
It's hard to configure &
update the software
Our admin who created our
templates has left...

15. 15
A shared goal emerges...
Deployer Interface
that makes it easy to
customize and
update the topology
Topology-as-code
Deployment lifecycle
actions
Full stack provisioning
Reuse organization
cookbooks & practices
Chef Server
Git
Topology
JSON
chef-
provisioning
...Now to make it happen!
Provisioning
Recipes
Convert from existing
templates

16. 16
Chef Server
What are the steps?
For the Hook'Em business system:
1) Set up Hook'Em System Inventory in Automate.Insights
2) Create a Blueprint based on the existing CloudFormation template
3) Create Topologies and provision them using generated recipes
app +
nodejs
mongo
dbapplication &
middleware
infrastructure
Topology
JSON
chef-
provisioning
Automate.
Insights
Provisioning
recipes
● System inventory
● Existing template

17. 17
1 – Setup Hook'Em System Inventory
Software Version Default recipe
mongodb 2.6.5 coolco_mongodb::default
nodeapp 1.0 coolco_nodeapp::deploy_from_
package
nodejs 0.10.29 coolco_nodejs::default
Recipe Inventory
apt
chef_client::config
coolco_handlers::default
coolco_mongodb::default
coolco_nodeapp::deploy_from_package
coolco_nodejs::default
Software Attribute Value
mongodb xxx yyy
xxx yyy
xxx yyy
Software Attribute Value
nodeapp nodeapp.name hookem
nodeapp.user automation
nodeapp.dir /home/automation
RunlistNode
attributes
Fred primes the System Inventory in Automate.Insights with CoolCo's standard recipes and
recommended attribute values for the software in the HookEm System.
System Inventory
Automate.
Insights
mongodb
nodeapp
nodejsBlueprint

18. 18
2 - Create a Blueprint for Hook'Em
Topology
JSON Automate.
Insights
The Hook'Em team converts the existing CloudFormation template using topo-provision, and uses it
and the System Inventory to create a Blueprint and Topologies in Automate.Insights
topo-
provision
CloudFormation
template
Software
inventory
Full-Stack
Topology JSON

19. 19
3 - Generate recipes & provision the full stack topology
chef-provisioning can handle network resources too!
app +
nodejs
mongo
dbapplication &
middleware
infrastructure
Full Stack
Topology JSON
topo-
provision
Chef-Provisioning recipes
chef-
provisioning
with_driver “aws”
load_balancer ... do
...
end
machine … do
…
end
aws_launch_config … do
...
end
aws_autoscaling_group … do
...
end
● Deploy
● Undeploy
load_balancer "appserver-elb" do
load_balancer_options({
:listeners=> [{
:port=>3001, :protocol=>"HTTP",
:instance_port=>3001, :instance_protocol=>"HTTP"
}],
:availability_zones=>
["us-west-2c", "us-west-2a", "us-west-2b"]})
end
machine "dbserver" do
run_list ["recipe[coolco::db]"]
attribute 'topo', {"node_type"=>"dbserver", "name"=>"test"}
end
deploy.rb:
Supports deployment
lifecycle actions
Sensitive to resource
relationships
Using topo-provision, the Hook'Em team generate topology provisioning recipes that enable chef-
provisioning to provision the full stack.

20. 20
Summary of transformation
Infrastructure templates
Stack create and delete
Customization of parameters
Full stack Topology JSON
Conversion from existing templates
Topology deploy and un-deploy
Integrated provisioning and configuration
management
Customization of attributes & run list
Line-of-sight across topologies
Tool assistance for change propagation
Topology
Template
Deployment
Deployer
Interface
As-Is To-Be
Practices Reuse organization cookbooks
Benefit from Chef test tools
CloudFormation
Template
CloudFormation
Stacks
CloudFormation
Topology JSON
ChefAutomate.Insights

21. 21
A Cloud-neutral topology template has wider value to the
Chef community
Topology
JSON
chef-
provisioning
Provisioning
Recipes
Convert from other
formats
Increase portability
CloudFormation
Terraform
OASIS TOSCA
Home-grown
Integrate with multiple
tools, from Chef and
others
Adoption of a Chef-oriented topology template that is not tied to one Cloud Provider helps increase
portability and tool integration going forward.
Deploy to multiple
providers
This is a vision with proof points for CloudFormation, AWS and Vagrant.
Test
Kitchen
kitchen.
yml
Automate.
Insights

22. 22
Thank you!
Tools:
http://www.thirdwaveinsights.com/automateinsights
https://github.com/christinedraper/topo-provision
https://github.com/christinedraper/knife-topo
Find Out More
Speakers:
john_sweitzer@thirdwaveinsights.com
@tkdsweitz
christine_draper@thirdwaveinsights.com
@CristineDraper