4. What?
• Chef at Etsy
• Familiarity and Understanding
Wednesday, June 27, 12
5. What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
Wednesday, June 27, 12
6. What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
Wednesday, June 27, 12
7. What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
• A liberal sprinkling of screwups
Wednesday, June 27, 12
8. What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
• A liberal sprinkling of screwups
• Open Sourced Goodness - We’re all here!
Wednesday, June 27, 12
9. What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
• A liberal sprinkling of screwups
• Open Sourced Goodness - We’re all here!
• [x] = http://tiny.cc/velocity2012
Wednesday, June 27, 12
10. Opscode is Orange,
Velocity is Blue.
In Soviet Russia,
Cookbook writes you.
Wednesday, June 27, 12
13. Our Setup
• Open Source chef server 0.10.4
Wednesday, June 27, 12
14. Our Setup
• Open Source chef server 0.10.4
• Backup to Opscode Platform
Wednesday, June 27, 12
15. Our Setup
• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS,
some RHEL & mac)
Wednesday, June 27, 12
16. Our Setup
• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS,
some RHEL & mac)
• KVM & lxc virts, self hosted too.
Wednesday, June 27, 12
17. Our Setup
• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS,
some RHEL & mac)
• KVM & lxc virts, self hosted too.
• Never test in production!
Wednesday, June 27, 12
18. Our Setup
• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS,
some RHEL & mac)
• KVM & lxc virts, self hosted too.
• Never test in production!
• Many chefs don’t spoil our soup
Wednesday, June 27, 12
19. Our Setup
• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS,
some RHEL & mac)
• KVM & lxc virts, self hosted too.
• Never test in production!
• Many chefs don’t spoil our soup
Wednesday, June 27, 12
32. Chef Dashboard
• Chef handler sends metrics to graphite [4]
Wednesday, June 27, 12
33. Chef Dashboard
• Chef handler sends metrics to graphite [4]
• git clone git://github.com/etsy/chef-handlers.git
Wednesday, June 27, 12
34. Chef Dashboard
• Chef handler sends metrics to graphite [4]
• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in
graphite.rb
Wednesday, June 27, 12
35. Chef Dashboard
• Chef handler sends metrics to graphite [4]
• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in
graphite.rb
• Add the following to client.rb
Wednesday, June 27, 12
36. Chef Dashboard
• Chef handler sends metrics to graphite [4]
• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in
graphite.rb
• Add the following to client.rb
• require "<clonedir>/graphite.rb"
graphite_handler = GraphiteReporting.new
report_handlers << graphite_handler
exception_handlers << graphite_handler
Wednesday, June 27, 12
37. Chef Dashboard
• Chef handler sends metrics to graphite [4]
• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in
graphite.rb
• Add the following to client.rb
• require "<clonedir>/graphite.rb"
graphite_handler = GraphiteReporting.new
report_handlers << graphite_handler
exception_handlers << graphite_handler
• Etsy dashboards framework [5]
Wednesday, June 27, 12
38. [12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com
[12:54:02] <irccat> https://github.etsycorp.com/gist/384228
[12:54:02] <irccat>
[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com
[12:54:07] <irccat> https://github.etsycorp.com/gist/384227
[12:54:07] <irccat>
Wednesday, June 27, 12
40. Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]
Wednesday, June 27, 12
41. Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]
• git clone git://github.com/etsy/chef-handlers.git
Wednesday, June 27, 12
42. Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]
• git clone git://github.com/etsy/chef-handlers.git
• Set your irccat[6] server’s URL in
logtoirc.rb
Wednesday, June 27, 12
43. Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]
• git clone git://github.com/etsy/chef-handlers.git
• Set your irccat[6] server’s URL in
logtoirc.rb
• Add the following to client.rb
Wednesday, June 27, 12
44. Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]
• git clone git://github.com/etsy/chef-handlers.git
• Set your irccat[6] server’s URL in
logtoirc.rb
• Add the following to client.rb
• require "<clonedir>/logtoirc.rb"
exception_handlers << Etsy::LogToIRC.new
Wednesday, June 27, 12
45. ~ > knife node lastrun buildtest11.ny4dev.etsy.com
Status failed
Elapsed Time 4.628171438
Start Time 2012-06-18 10:06:28 +0000
End Time 2012-06-18 10:06:32 +0000
Recipe Action Resource Type Resource
Backtrace
<snip>
Exception
Chef::Exceptions::Package: package[php] (php::buildtest line 20) had
an error: Version 5.3.10-1.el5 of php not found. Did you specify both
version and release? (version-release, e.g. 1.84-10.fc6)
Wednesday, June 27, 12
46. ~ > knife search node 'lastrun_debug_formatted_exception:Chef:
:Exceptions::Package*' -a lastrun.debug.formatted_exception
5 items found
id: masterrestore.ny4.etsy.com
lastrun.debug.formatted_exception: Chef::Exceptions::Package:
package[postgresql-server] (postgresql::server-8.3 line 1) had an
error: Installed package postgresql-server-8.3.16-1PGDG_id is newer
than candidate package postgresql-server-8.3.11-1PGDG_id.rhel5
id: buildtest11.ny4dev.etsy.com
lastrun.debug.formatted_exception: Chef::Exceptions::Package:
package[php] (php::buildtest line 20) had an error: Version
5.3.10-1.el5 of php not found. Did you specify both version and
release? (version-release, e.g. 1.84-10.fc6)
<snip>
Wednesday, June 27, 12
48. Chef lastrun Info
• Chef handler and knife plugin [7]
Wednesday, June 27, 12
49. Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
Wednesday, June 27, 12
50. Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
• Add the following to client.rb
Wednesday, June 27, 12
51. Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
• Add the following to client.rb
• require "lastrun_update"
handler = LastRunUpdateHandler.new
report_handlers << handler
exception_handlers << handler
Wednesday, June 27, 12
52. Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
• Add the following to client.rb
• require "lastrun_update"
handler = LastRunUpdateHandler.new
report_handlers << handler
exception_handlers << handler
• knife node lastrun <nodename>
Wednesday, June 27, 12
54. Simplicity
• Think of yourself at 3AM!
Wednesday, June 27, 12
55. Simplicity
• Think of yourself at 3AM!
• Please, won’t you think of the new guy?
Wednesday, June 27, 12
56. Simplicity
• Think of yourself at 3AM!
• Please, won’t you think of the new guy?
• Minimize the logics!
Wednesday, June 27, 12
57. Simplicity
• Think of yourself at 3AM!
• Please, won’t you think of the new guy?
• Minimize the logics!
• As few logical steps from start to finish
as possible.
Wednesday, June 27, 12
58. Simplicity - Not!
Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)
Subject: so close to death
# Don't install v2 on search or Cent 5.6 nodes
-if node[:fqdn] !~ /b(^(preprod-)?search[0-9]{2}|ny4dev.etsy.com|^(preprod-)?
giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|
^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|
^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|
devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|^imgcache[0-9]{2}|
imgconvert[0-9]{2}|^imgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|
^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|
deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})b/ and node.role?("Web56") ==
false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false
+if node[:fqdn] !~ /b(^(preprod-)?search[0-9]{2}|ny4dev.etsy.com|^(preprod-)?
giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|
^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|
^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|
devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|^imgcache[0-9]{2}|
imgconvert[0-9]{2}|^imgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|
^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|
deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})b/ and node.role?("Web56") ==
false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false and
node.role?("Auth") == false
Wednesday, June 27, 12
59. Simplicity - Better!
if node.chef_environment == "libmemcached_upgrade"
package "libmemcached" do
version "1.0.4-1"
action :install
end
<snip>
else
package "libmemcached" do
version "0.53-1.1"
action :install
end
<snip>
end
Wednesday, June 27, 12
62. Simplicity - Complexity
• Sometimes you need complex behaviour
• Don’t fight it, try to abstract it.
Wednesday, June 27, 12
63. Simplicity - Complexity
• Sometimes you need complex behaviour
• Don’t fight it, try to abstract it.
• Case in point: Syslog-ng refactor
Wednesday, June 27, 12
66. Case Study: Syslog-ng
• 36 recipes
• 30 versions of syslog-ng.conf
Wednesday, June 27, 12
67. Case Study: Syslog-ng
• 36 recipes
• 30 versions of syslog-ng.conf
• 27 manually configured files in /etc/syslog-
ng.d on central server
Wednesday, June 27, 12
68. Case Study: Syslog-ng
• 36 recipes
• 30 versions of syslog-ng.conf
• 27 manually configured files in /etc/syslog-
ng.d on central server
• Edge cases and exceptions galore
Wednesday, June 27, 12
71. Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
Wednesday, June 27, 12
72. Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
• 2 templates (one for syslog-ng.conf, one
for stuff in /etc/syslog-ng.d)
Wednesday, June 27, 12
73. Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
• 2 templates (one for syslog-ng.conf, one
for stuff in /etc/syslog-ng.d)
• Attributes in roles
Wednesday, June 27, 12
74. Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
• 2 templates (one for syslog-ng.conf, one
for stuff in /etc/syslog-ng.d)
• Attributes in roles
• Not open sourced yet, sorry :(
Wednesday, June 27, 12
77. Remember, No
Panacea!
• A new package hits the repo.
Wednesday, June 27, 12
78. Remember, No
Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
Wednesday, June 27, 12
79. Remember, No
Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
• Memcached Outage
Wednesday, June 27, 12
80. Remember, No
Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
• Memcached Outage
• Do you know what services are going to
restart and when?
Wednesday, June 27, 12
81. Remember, No
Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
• Memcached Outage
• Do you know what services are going to
restart and when?
• Image Service Outage
Wednesday, June 27, 12
86. Standards - No Time!
• I won’t say “Make Time”, but you should...
Wednesday, June 27, 12
87. Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
Wednesday, June 27, 12
88. Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
Wednesday, June 27, 12
89. Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
• Jenkins integration in seconds
Wednesday, June 27, 12
90. Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
• Jenkins integration in seconds
• Supports custom rules
Wednesday, June 27, 12
91. Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
• Jenkins integration in seconds
• Supports custom rules
• Plays well with others
Wednesday, June 27, 12
98. Standards at Etsy
• “style” not “correctness”[9]
Wednesday, June 27, 12
99. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
Wednesday, June 27, 12
100. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
Wednesday, June 27, 12
101. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
Wednesday, June 27, 12
102. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or
action :nothing
Wednesday, June 27, 12
103. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or
action :nothing
• ETSY005 - Action :restart sent to a core service
Wednesday, June 27, 12
104. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or
action :nothing
• ETSY005 - Action :restart sent to a core service
• ETSY006 - Execute resource used to run chef-provided command
Wednesday, June 27, 12
105. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or
action :nothing
• ETSY005 - Action :restart sent to a core service
• ETSY006 - Execute resource used to run chef-provided command
• ETSY007 - Package or yum_package resource used to install core
package without specific version number
Wednesday, June 27, 12
106. Standards at Etsy
• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or
action :nothing
• ETSY005 - Action :restart sent to a core service
• ETSY006 - Execute resource used to run chef-provided command
• ETSY007 - Package or yum_package resource used to install core
package without specific version number
Wednesday, June 27, 12
108. Standards at Etsy
• ETSY001 - Written after Memcached
Outage
Wednesday, June 27, 12
109. Standards at Etsy
• ETSY001 - Written after Memcached
Outage
• Package or yum_package resource used
with :upgrade action
Wednesday, June 27, 12
110. Standards at Etsy
• ETSY001 - Written after Memcached
Outage
• Package or yum_package resource used
with :upgrade action
• package "memcached" do
action :upgrade
end
Wednesday, June 27, 12
112. Standards at Etsy
• ETSY005 - Written after a total Image
service outage
Wednesday, June 27, 12
113. Standards at Etsy
• ETSY005 - Written after a total Image
service outage
• Action :restart sent to a core service
Wednesday, June 27, 12
114. Standards at Etsy
• ETSY005 - Written after a total Image
service outage
• Action :restart sent to a core service
• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do
source "myvhost.conf"
notifies :restart, resources(:service => "httpd")
end
Wednesday, June 27, 12
117. CA&E
• Chef is by necessity generic
Wednesday, June 27, 12
118. CA&E
• Chef is by necessity generic
• ...so don’t take Opscode’s word for it.
Wednesday, June 27, 12
119. CA&E
• Chef is by necessity generic
• ...so don’t take Opscode’s word for it.
• If it doesn’t work well for you, change it!
Wednesday, June 27, 12
120. CA&E
• Chef is by necessity generic
• ...so don’t take Opscode’s word for it.
• If it doesn’t work well for you, change it!
• Case Study - Etsy Environments rollout
Wednesday, June 27, 12
127. Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
Wednesday, June 27, 12
128. Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
Wednesday, June 27, 12
129. Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
• Commit and push changes to git
Wednesday, June 27, 12
130. Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
• Commit and push changes to git
• knife cookbook upload php --freeze
Wednesday, June 27, 12
131. Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
• Commit and push changes to git
• knife cookbook upload php --freeze
• knife environment from file foo.json
Wednesday, June 27, 12
133. Env: Our Issues
• 41 people with Chef repo access
Wednesday, June 27, 12
134. Env: Our Issues
• 41 people with Chef repo access
• Most with knife keys
Wednesday, June 27, 12
135. Env: Our Issues
• 41 people with Chef repo access
• Most with knife keys
• All editing the same 2 files with every
change...
Wednesday, June 27, 12
137. Env: Solutions?
• Go with it and hope for the best?
Wednesday, June 27, 12
138. Env: Solutions?
• Go with it and hope for the best?
• Don’t use environments?
Wednesday, June 27, 12
139. Env: Solutions?
• Go with it and hope for the best?
• Don’t use environments?
• Write a totally new workflow?
Wednesday, June 27, 12
140. Env: Solutions?
• Go with it and hope for the best?
• Don’t use environments?
• Write a totally new workflow?
• Tweak the existing one with some
tooling?
Wednesday, June 27, 12
145. Spork: Workflow
• Wrapper around standard environments
workflow
Wednesday, June 27, 12
146. Spork: Workflow
• Wrapper around standard environments
workflow
• check - cookbook versioning
Wednesday, June 27, 12
147. Spork: Workflow
• Wrapper around standard environments
workflow
• check - cookbook versioning
• bump - increment version component
Wednesday, June 27, 12
148. Spork: Workflow
• Wrapper around standard environments
workflow
• check - cookbook versioning
• bump - increment version component
• upload - upload and freeze
Wednesday, June 27, 12
149. Spork: Workflow
• Wrapper around standard environments
workflow
• check - cookbook versioning
• bump - increment version component
• upload - upload and freeze
• promote - set env constraints
Wednesday, June 27, 12
150. Spork: Check
$> knife spork check apache2
Checking versions for cookbook apache2...
Current local version: 1.0.6
Remote versions (Max. 5 most recent only):
*1.0.6, frozen
1.0.5, frozen
<snip>
DANGER: Your local cookbook has same version number as
the starred version above!
Please bump your local version or you won't be able to
upload.
Wednesday, June 27, 12
151. Spork: Bump
$> knife spork bump apache2 <major|minor|patch|manual>
Bumping patch level of the apache2 cookbook from 1.0.6
to 1.0.7
Wednesday, June 27, 12
152. Spork: Upload
$> knife spork upload apache2
Uploading and freezing apache2 [1.0.7]
upload complete
Wednesday, June 27, 12
153. Spork: Promote
$> knife spork promote foo php
Adding version constraint php = 1.0.6
Saving changes into foo.json
Promotion complete! Please remember to upload your
changed Environment file to the Chef Server.
---
$> knife spork promote foo php --remote
Adding version constraint php = 0.1.0
Saving changes into foo.json
Uploading foo to server
Wednesday, June 27, 12
155. Spork
• Worked well, avoided the issues it was
designed for
Wednesday, June 27, 12
156. Spork
• Worked well, avoided the issues it was
designed for
• Subsequently evolved
Wednesday, June 27, 12
157. Spork
• Worked well, avoided the issues it was
designed for
• Subsequently evolved
• A lot of input & work came from Devs
Wednesday, June 27, 12
158. Spork
• Worked well, avoided the issues it was
designed for
• Subsequently evolved
• A lot of input & work came from Devs
• Organic evolution
Wednesday, June 27, 12
159. Spork
• Worked well, avoided the issues it was
designed for
• Subsequently evolved
• A lot of input & work came from Devs
• Organic evolution
• Open sourced, of course
Wednesday, June 27, 12
164. Spork: Safety Checks
• Before promoting, check version is
uploaded...
Wednesday, June 27, 12
165. Spork: Safety Checks
$> knife spork promote php --remote
<snip>
WARNING: It looks like you have multiple cookbook paths
defined so I can't tell if you're running inside a git
repo.
Checking that php version 0.1.93 exists on the server
before promoting (any error means it hasn't been
uploaded yet)...
ERROR: The object you are looking for could not be found
Response: Cannot find a cookbook named php with version
0.1.93
Wednesday, June 27, 12
166. Spork: Safety Checks
• Before promoting, check version is
uploaded...
• Check if you’re promoting changes to more
than you thought....
Wednesday, June 27, 12
167. Spork: Safety Checks
WARNING: You're about to promote changes to several
cookbooks:
WARNING:
ganglia: = 0.1.26 changed to = 0.1.25
installerz: = 0.1.66 changed to = 0.1.65
php: = 0.1.92 changed to = 0.1.93
Are you sure you want to continue? (Y/N) N
You said no, so I'm done here.
Would you like to reset your local development.json to
match the server?? (Y/N) Y
<snip>
development.json reset.
Wednesday, June 27, 12