This document discusses using Chef to automate server maintenance including patching and rebooting. It outlines: - Current pain points with patching including unclear responsibilities and risk of simultaneous reboots. - Proposed solution of using Chef to schedule patching and reboots on a periodic basis (e.g. weekends), with options to customize the schedule and opt out. - Recommendations for testing the Chef cookbook gradually and allowing time for customization before full rollout.

2. Maintenance Mode
Gilbert Sanchez
Señor Systems Engineer
Leveraging Chef to Schedule Patching, Reboot, etc.

3. • Me:
• GitHub: https://github.com/HeyItsGilbert
• Twitter @HeyItsGilbertS
• Facebook:
• Prod: https://github.com/facebook/chef-cookbooks
• CPE: https://github.com/facebook/IT-CPE
Who?

4. • Manage Corporate Windows Servers
• Automate patch deployment
• Image builds
• Collaborate with Client Platform Engineering team
What is it that you do here?

5. A Few Facebook-isms

6. • Initialization Cookbooks (See fb_init, cpe_init)
• API cookbooks
• Sharding
It's Not You, Its Me
Facebook Approach

7. Where did it all go wrong?

8. Classic Recipe for Disaster
• Small change
• Delete deprecated code
• Friday night
• Looming deadlines

9. # In cookbook A
reboot "PostInstallReboot" do
action :nothing
end
----
# In cookbook B
windows_package "WMF5" do
notify "reboot[PostInstallReboot]"
end
Danger Zone
Reboot Resources

11. Phase 1: Planning

12. • Operational tasks to patch machines that failed
vulnerability scans
• Deploying software that required reboots
• Unknown responsibility throughout the patching
process
Current State
What are the current pain points?

13. • Opt Out - Some teams want/need to be able to
patch on their own schedule
• High Availability - Don't reboot all the machines at
the same time
• PST? EST? - How to handle multi region services
Requirements
What will our end users need from us?

14. • Who doesn't love calculating time!
• Offer a UTC solution
Timezones?
Source: https://en.wikipedia.org/wiki/Time_zone

15. Phase 2: Execution

16. • New apps installed?
• New patches installed?
• This would have prevented the previous issue.
• Leverage reboot_pending?
Pending Reboots

17. 1. Read your
attributes
2. Drop your scripts/
modules/etc.
3. Setup your
scheduler
Chef Custom Resource
Run time vs Compile time is critical
Source: https://docs.chef.io/attributes.html

18. • Chef is config management, don't use as a
scheduler
• Leverage the appropriate scheduler on your OS
• Calculating time is hard for you, and harder for your
users
• Leave a file on disk for them to check
Warnings
Classic Pitfalls

19. default['your_cookbook'] = {
'auto_reboot' => true,
'day' => 'Saturday',
'duration' => 5, # 5hrs
'opt_out' => false,
'start_hour' => 1, # 1am
'trusted_tester' => false,
'utc' => false,
}
Default Options
Possible Attributes

20. # User has their own special process? FINE!
node.default['your_cookbook']['opt_out'] = True
# Midweek reboots at noon but 1hr window
node.default['your_cookbook']['day'] = 'Wednesday'
node.default['your_cookbook']['start_hour'] = 12
node.default['your_cookbook']['duration'] = 1
Cookbook for my set of machines
Customizing

21. Phase 3: More Work

22. • Roll out to your test boxes
• Don't have any? Get some now.
• Have your owners set their attributes even just to
verify
Ready to Roll
Code is ready. What's next?

23. • See cpe_utils cookbook in supermarket
• Leverage shard_over_a_week_*
Slow Roll
Catch possible errors with sharing

24. Whats next from us?

25. • Working on creating a custom shutdown option
• Useful for larger services
• Open Source!
More ToDo's
Yay!

26. • Thanks to the team!
• Specifically Korey Shirey and Kamber Borders!
Shout Outs

27. Questions?