1. How$We$Use$Chef$(currently)
By#Dan#Ivovich
January'21,'2015
LAX$Internal$Conf

2. Chef%can%be%hard%to%wrap%your%head%around

3. Automa'ons*can*go*horribly*wrong

4. But$without$Chef
This%is%your%server%infrastructure

5. Chef%has%a%learning%curve

6. I've%given%talks%about%what%Chef%is%and%how%Chef%
works
But$it$is$hard$to$cover$all$of$Chef$thoroughly

7. The$Good$News$is$Chef%works!

8. So#Let's#Get#Prac%cal
How$do$we$use$Chef$today$at$SmartLogic?

9. Consider)this)app)Wuphf
If#you#send#a#Wuphf,#the#message#goes#to#the#
recipients'#home#phone,#cell#phone,#email,#Facebook,#
twi<er,#fax#and#homescreen#at#the#same#>me.

10. Wuphf&Tech&Stack
1. Rails(App((Unicorn)
2. Nginx
3. Amazon(S3,(Twi=er,(Facebook,(and(Twilio(tokens
4. Redis
5. Sidekiq
6. Postgresql

11. How$do$we$use$the$tool$to$minimize$
differences$between$environments?
Your%server%cluster%is%all%about%coordina1on

12. Infrastructure+Considera0ons
1. Staging)is)a)single)VM
2. Beta)is)a)web/worker)VM)and)a)DB)VM
3. Produc=on)has)web,)worker,)and)DB)VMs

13. Chef%Setup%(Berksfile)
source 'https://supermarket.getchef.com'
cookbook 'annoyances', '~> 0.0.2'
cookbook 'build-essential'
cookbook 'firewall', '>= 0.8'
cookbook 'git', '~> 3.0'
cookbook 'hostname', '~> 0.0.4'
cookbook 'hostsfile', '~> 1.0'
cookbook 'imagemagick', '~> 0.2'
cookbook 'monit', '~> 0.7'
cookbook 'mysql'
cookbook 'newrelic'
cookbook 'nginx', '~> 2.7'
cookbook 'openssl', '~> 1.1'
cookbook 'openssh', '~> 1.2'
cookbook 'postgresql', '~> 3.4'
cookbook 'rbenv'
cookbook 'redisio', '~> 1.4'
cookbook 'sudo', '~> 1.1'
cookbook 'ufw'
cookbook 'users_solo', '~> 1.0'
cookbook 'xml'
cookbook 'github_keys', github: 'smartlogic/github_keys-cookbook'
cookbook 'wuphf', path: './site-cookbooks/wuphf'

14. One$Chef$Cookbook$to$RULE$THEM$ALL
cookbook 'wuphf', path: './site-cookbooks/wuphf'
Place&all&the&work&our&app&needs&done&in&a&single&
cookbook&with&recipies&for&each&major&component
1. wuphf::setup
2. wuphf::db
3. wuphf::redis
4. wuphf::web
5. wuphf::app
6. wuphf::worker
We#can#use#these#recipes#in#each#environment#as#we#
need#to.

15. The$recipe$for$your$app$can$be$very$simple
What%do%those%recipes%look%like?

16. wuphf::setup
Do#the#work#that#needs#to#happen#on#every#server#we#
set#up

17. wuphf::setup
Hostname)and)hosts)file
include_recipe 'hostname'
hostsfile_entry '127.0.0.1' do
hostname "#{node['fqdn']} #{node['hostname']} localhost"
end
(node['wuphf']['hosts'] || {}).each_pair do |name, ip|
hostsfile_entry ip do
hostname name
end
end

18. wuphf::setup
Compile(dependencies,(ssl,(ssh,(sudo(setup
include_recipe 'build-essential'
include_recipe 'openssl'
include_recipe 'openssh'
include_recipe 'sudo'
Admin&accounts,&ssh&keys,&firewall&rules
include_recipe 'users_solo::admins'
include_recipe 'github_keys'
include_recipe 'ufw'

19. wuphf::db
Setup&and&configure&a&database&server

20. wuphf::db
Make%sure%setup%has%run
include_recipe 'wuphf::setup'
Install'the'db'server'so/ware,'add2ons,'and'tuning
include_recipe 'postgresql::server'
include_recipe 'postgresql::contrib'
include_recipe 'postgresql::ruby'
include_recipe 'postgresql::config_pgtune'

21. wuphf::db
Make%sure%the%applica/on%database%user%and%schema%
exists
connection_info = {
:host => "127.0.0.1",
:port => 5432,
:username => 'postgres',
:password => node['postgresql']['password']['postgres']
}
postgresql_database_user node['wuphf']['database_user'] do
connection connection_info
password node['wuphf']['database_password']
action :create
end
postgresql_database node['wuphf']['database'] do
connection connection_info
encoding 'UTF8'
owner node['wuphf']['database_user']
action :create
end

22. wuphf::redis
Setup&a&redis&server

23. wuphf::redis
Make%sure%setup%has%run
include_recipe 'wuphf::setup'
Install'the'redis'and'enable'it,'install'monit'for'
monitoring
include_recipe 'redisio::install'
include_recipe 'redisio::enable'
include_recipe 'monit'

24. wuphf::web
Setup&Nginx&web&server&components

25. wuphf::web
Make%sure%setup%has%run
include_recipe 'wuphf::setup'
Install'Nginx'from'the'repository,'and'monit'so'we'
can'monitor'it
include_recipe 'nginx::repo'
include_recipe 'nginx'
include_recipe 'monit'

26. wuphf::app
Setup&the&applica,on&core

27. wuphf::app
Make%sure%setup%has%run
include_recipe 'wuphf::setup'
Install'client'libraries'needed'and'monitoring
include_recipe 'postgresql::client'
include_recipe 'redisio::install'
include_recipe 'monit'

28. wuphf::app
Install'applica+on'ruby'and'setup'folder'structure
include_recipe 'wuphf::ruby'
include_recipe 'wuphf::app_env'
Install'applica+on'dependencies
include_recipe 'xml'
include_recipe 'imagemagick'
include_recipe 'nodejs'

29. wuphf::app
Rotate&your&logs
package "logrotate"
template "/etc/logrotate.d/wuphf_unicorn" do
source "unicorn_logrotate.erb"
mode "0644"
owner 'root'
group 'root'
variables(
:log_paths => [
"#{node['wuphf']['deploy_to']}/shared/log/#{node['wuphf']['environment']}.log",
"#{node['wuphf']['deploy_to']}/shared/log/unicorn.log"
],
:pid_path => "#{node['wuphf']['deploy_to']}/shared/tmp/pids/unicorn.pid"
)
end

30. WOAH!&Two&new&recipes!
1. wuphf::ruby
2. wuphf::app_env

31. wuphf::ruby
Setup&Ruby&and&Install&Bundler
include_recipe 'rbenv'
include_recipe 'rbenv::ruby_build'
rbenv_ruby node['wuphf']['ruby_version']
rbenv_gem "bundler" do
ruby_version node['wuphf']['ruby_version']
end

32. wuphf::app_env
Setup&the&folder&structure&for&the&app&to&deploy&to
directory "#{node['wuphf']['cap_base']}" do
action :create
owner 'deploy'
group 'deploy'
mode '0755'
end
directory "#{node['wuphf']['deploy_to']}" do
action :create
owner 'deploy'
group 'deploy'
mode '0755'
end
directory "#{node['wuphf']['deploy_to']}/shared" do
action :create
owner 'deploy'
group 'deploy'
mode '0755'
end
directory "#{node['wuphf']['deploy_to']}/shared/config" do
action :create
owner 'deploy'
group 'deploy'
mode '0755'
end

33. wuphf::app_env
Create&the&database.yml&and&file&for&dotenv
template "#{node['wuphf']['deploy_to']}/shared/config/database.yml" do
source 'database.yml.erb'
owner 'deploy'
group 'deploy'
mode '0644'
end
template "#{node['wuphf']['deploy_to']}/shared/wuphf_env" do
source 'wuphf_env.erb'
owner 'deploy'
group 'deploy'
mode '0644'
end

34. wuphf::worker
Setup&background&worker&node

35. wuphf::worker
Make%sure%setup%has%run
include_recipe 'wuphf::setup'
Install'client'libraries'needed'and'monitoring
include_recipe 'postgresql::client'
include_recipe 'redisio::install'
include_recipe 'monit'

36. wuphf::worker
Install'applica+on'ruby'and'setup'folder'structure
include_recipe 'wuphf::ruby'
include_recipe 'wuphf::app_env'
Install'applica+on'dependencies
include_recipe 'xml'
include_recipe 'imagemagick'
include_recipe 'nodejs'

37. Make%sure%your%new%cookbook%has%a0ributes
a"ributes/default.rb
default['wuphf']['cap_base'] = '/home/deploy/apps'
default['wuphf']['deploy_to'] = '/home/deploy/apps/wuphf'
default['wuphf']['environment'] = 'production'
default['wuphf']['database'] = 'wuphf'
default['wuphf']['adapter'] = 'postgresql'
default['wuphf']['database_user'] = 'postgres'
default['wuphf']['database_password'] = (node['postgresql']['password']['postgres'] rescue nil)
default['wuphf']['database_host'] = 'localhost'
default['wuphf']['database_pool'] = 25
default['wuphf']['ruby_version'] = '2.1.3'
default['wuphf']['env_settings'] = {}

38. Review&cookbook&architecture

39. Now$to$configure$our$nodes

40. The$wuphf$config$key$and$the$Run$List$vary$
significantly
Firewall(and(Postgres(keys(also(vary(from(node(to(node

41. First&let's&consider&the&firewall&rules

42. Review&network&architecture

43. Staging'(10.0.0.1)
"firewall" : {
"rules" : [
{ "http" : { "port" : "80" } }
]
},
The$cookbook$keeps$SSH$port$22$open$for$us

44. Beta%Web/App/Worker%(10.0.0.2)
"firewall" : {
"rules" : [
{ "http" : { "port" : "80" } }
]
},
Beta%DB/Redis%(10.0.0.3)
"firewall" : {
"rules" : [
{ "psql" : { "port" : "5432", "source": "10.0.0.2/32" } },
{ "redis" : { "port" : "6379", "source": "10.0.0.2/32" } }
]
},

45. Prod%Web%(10.0.0.5)
"firewall" : {
"rules" : [
{ "http" : { "port" : "80" } }
]
},
Prod%Worker%(10.0.0.6)
"firewall" : {
"rules" : [ ]
},

46. Prod%DB/Redis%(10.0.0.7)
"firewall" : {
"rules" : [
{ "psql" : { "port" : "5432", "source": "10.0.0.5/32" } },
{ "psql" : { "port" : "5432", "source": "10.0.0.6/32" } },
{ "redis" : { "port" : "6379", "source": "10.0.0.5/32" } },
{ "redis" : { "port" : "6379", "source": "10.0.0.6/32" } },
]
},

47. Postgres(Setup
Staging'(10.0.0.1)
"postgresql" : {
"password" : {
"postgres" : "super-secret-password"
},
"enable_pgdg_apt" : true,
"version" : "9.3",
"config_pgtune" : {
"max_connections" : "100",
"db_type" : "desktop"
}
},

48. Beta%DB/Redis%(10.0.0.3)%(Addi3ons)
"postgresql" : {
"config" : {
"listen_addresses" : "localhost, 10.0.0.3"
},
"pg_hba" : [
{ "type" : "local", "db" : "all", "user" : "postgres", "addr" : null, "method" : "ident"},
{ "type" : "local", "db" : "all", "user" : "all", "addr" : null, "method" : "ident"},
{ "type" : "host", "db" : "all", "user" : "all", "addr" : "127.0.0.1/32", "method" : "md5"},
{ "type" : "hostssl", "db" : "all", "user" : "wuphf", "addr" : "10.0.0.2/32", "method" : "md5"},
{ "type" : "host", "db" : "all", "user" : "all", "addr" : "::1/128", "method" : "md5"}
],
"config_pgtune" : {
"db_type" : "web",
"max_connections" : "400"
}
},

49. Prod%DB/Redis%(10.0.0.7)%(Addi4ons)
"postgresql" : {
"pg_hba" : [
{ "type" : "hostssl", "db" : "all", "user" : "wuphf", "addr" : "10.0.0.5/32", "method" : "md5"},
{ "type" : "hostssl", "db" : "all", "user" : "wuphf", "addr" : "10.0.0.6/32", "method" : "md5"},
],
},

50. Wuphf
Staging'(10.0.0.1)
"wuphf" : {
"environment" : "staging",
"database" : "wuphf_staging",
"env_settings" : {
"AWS_ACCESS_KEY" : "AWS_ACCESS_KEY",
"AWS_SECRET_ACCESS_KEY" : "AWS_SECRET_ACCESS_KEY",
"FACEBOOK_APP_ID" : "FACEBOOK_APP_ID",
"FACEBOOK_APP_SECRET" : "FACEBOOK_APP_SECRET",
"NEW_RELIC_LICENSE_KEY" : "NEW_RELIC_LICENSE_KEY",
"TWITTER_CONSUMER_KEY" : "TWITTER_KEY",
"TWITTER_CONSUMER_SECRET" : "TWITTER_SECRET",
"TWILIO_API_KEY" : "TWILIO_API_KEY",
"REDIS_PROVIDER" : "REDIS_URL",
"REDIS_URL" : "redis://localhost:6379",
"RACK_ENV" : "staging",
"RAILS_ENV" : "staging"
}
},

51. Beta%Web/App/Worker%(10.0.0.2)
"wuphf" : {
"environment" : "beta",
"database" : "wuphf_beta",
"hosts" : {
"db1" : "10.0.0.3"
},
"env_settings" : {
"AWS_ACCESS_KEY" : "AWS_ACCESS_KEY",
"AWS_SECRET_ACCESS_KEY" : "AWS_SECRET_ACCESS_KEY",
"FACEBOOK_APP_ID" : "FACEBOOK_APP_ID",
"FACEBOOK_APP_SECRET" : "FACEBOOK_APP_SECRET",
"NEW_RELIC_LICENSE_KEY" : "NEW_RELIC_LICENSE_KEY",
"TWITTER_CONSUMER_KEY" : "TWITTER_KEY",
"TWITTER_CONSUMER_SECRET" : "TWITTER_SECRET",
"TWILIO_API_KEY" : "TWILIO_API_KEY",
"REDIS_PROVIDER" : "REDIS_URL",
"REDIS_URL" : "redis://10.0.0.3:6379",
"RACK_ENV" : "beta",
"RAILS_ENV" : "beta"
}
},

52. Prod%Web/App%(10.0.0.5)
"wuphf" : {
"environment" : "production",
"database" : "wuphf_production",
"hosts" : {
"db1" : "10.0.0.7",
"worker1" : "10.0.0.6"
},
"env_settings" : {
"AWS_ACCESS_KEY" : "AWS_ACCESS_KEY",
"AWS_SECRET_ACCESS_KEY" : "AWS_SECRET_ACCESS_KEY",
"FACEBOOK_APP_ID" : "FACEBOOK_APP_ID",
"FACEBOOK_APP_SECRET" : "FACEBOOK_APP_SECRET",
"NEW_RELIC_LICENSE_KEY" : "NEW_RELIC_LICENSE_KEY",
"TWITTER_CONSUMER_KEY" : "TWITTER_KEY",
"TWITTER_CONSUMER_SECRET" : "TWITTER_SECRET",
"TWILIO_API_KEY" : "TWILIO_API_KEY",
"REDIS_PROVIDER" : "REDIS_URL",
"REDIS_URL" : "redis://10.0.0.7:6379",
"RACK_ENV" : "production",
"RAILS_ENV" : "production"
}
},

53. Prod%Worker%(10.0.0.6)
"wuphf" : {
"environment" : "production",
"database" : "wuphf_production",
"hosts" : {
"db1" : "10.0.0.7",
"web1" : "10.0.0.5",
},
"env_settings" : {
"AWS_ACCESS_KEY" : "AWS_ACCESS_KEY",
"AWS_SECRET_ACCESS_KEY" : "AWS_SECRET_ACCESS_KEY",
"FACEBOOK_APP_ID" : "FACEBOOK_APP_ID",
"FACEBOOK_APP_SECRET" : "FACEBOOK_APP_SECRET",
"NEW_RELIC_LICENSE_KEY" : "NEW_RELIC_LICENSE_KEY",
"TWITTER_CONSUMER_KEY" : "TWITTER_KEY",
"TWITTER_CONSUMER_SECRET" : "TWITTER_SECRET",
"TWILIO_API_KEY" : "TWILIO_API_KEY",
"REDIS_PROVIDER" : "REDIS_URL",
"REDIS_URL" : "redis://10.0.0.7:6379",
"RACK_ENV" : "production",
"RAILS_ENV" : "production"
}
},

54. App#Run#Lists

55. Staging'(10.0.0.1)
"run_list": [
"recipe[wuphf::db]",
"recipe[wuphf::redis]",
"recipe[wuphf::web]",
"recipe[wuphf::app]",
"recipe[wuphf::worker]",
"recipe[newrelic]"
]

56. Beta%Web/App/Worker%(10.0.0.2)
"run_list": [
"recipe[wuphf::web]",
"recipe[wuphf::app]",
"recipe[wuphf::worker]",
"recipe[newrelic]"
]
Beta%DB/Redis%(10.0.0.3)
"run_list": [
"recipe[wuphf::db]",
"recipe[wuphf::redis]",
"recipe[newrelic]"
]

57. Prod%Web%(10.0.0.5)
"run_list": [
"recipe[wuphf::web]",
"recipe[wuphf::app]",
"recipe[newrelic]"
]
Prod%Worker%(10.0.0.6)
"run_list": [
"recipe[wuphf::worker]",
"recipe[newrelic]"
]

58. Prod%DB/Redis%(10.0.0.7)
"run_list": [
"recipe[wuphf::db]",
"recipe[wuphf::redis]",
"recipe[newrelic]"
]

59. Review&system&architecture

60. Confused?*Amazed?

61. Don't&worry
It#is#a#process#of#trial#and#error

62. You've'got'this

63. You$can$look$awesome

64. Any$ques)ons?

65. Thank&You!
smartlogic.io
@smartlogic
danivovich.com
@danivovich

66. Image&Cita*ons
h"p:/
/devopsreac.ons.tumblr.com/post/
104064772852/understanding@git
h"p:/
/collec.onofawesome.com/2012/10/18/
anchorman@60@of@the@.me@gif/
h"p:/
/devopsreac.ons.tumblr.com/post/
106500035590/building@a@new@server@and@adding@
it@to@the@live
h"p:/
/devopsreac.ons.tumblr.com/post/
105951734875/describing@our@network@
infrastructure
h"p:/
/theoffice.wikia.com/wiki/WUPHF.com

67. Image&Cita*ons&(con*nued)
h"p:/
/devopsreac.ons.tumblr.com/post/
77162384009/se?ngAupAnewAhostAwithApuppetAorA
chef
h"p:/
/devopsreac.ons.tumblr.com/post/
87186564388/kspliceAkpatch
h"p:/
/devopsreac.ons.tumblr.com/post/
97207834532/fixingAsystemsAinternals
h"p:/
/giphy.com/gifs/keanuAreevesAuPnKU86sFa2fm
h"p:/
/devopsreac.ons.tumblr.com/post/
107492619940/deployingAaAcluster
h"p:/
/devopsreac.ons.tumblr.com/post/

68. Image&Cita*ons&(con*nued)
h"p:/
/i.imgur.com/dJ5jnB0.gif
h"p:/
/devopsreac:ons.tumblr.com/post/
61010641998/someChos:ngCcompaniesConC
availability
h"p:/
/i.imgur.com/QdVD216.gif
h"ps:/
/i.imgur.com/CXNOCBb.gif
h"p:/
/media.tumblr.com/
fe40a2da7c8e594479f48fd8450817d5/
tumblr_inline_nczuo9C9ov1raprkq.gif
h"p:/
/devopsreac:ons.tumblr.com/post/
72072493400/srmdsCserverCroomCmadnessC

69. Image&Cita*ons&(concluded)
h"p:/
/devopsreac.ons.tumblr.com/post/
79349198148/vendors<benchmarks
h"p:/
/devopsreac.ons.tumblr.com/post/
86295268875/cleaning<up<openssl<code
h"p:/
/devopsreac.ons.tumblr.com/post/
93197357391/security<hole
h"p:/
/devopsreac.ons.tumblr.com/post/
87581910878/trying<to<make<progress<on<monday
h"p:/
/devopsreac.ons.tumblr.com/post/
91731143332/doing<trial<and<error<with<chef