Talk at DevOpsDC on 11/18/14 discussing how WebMD started using Chef, why it didn't work "out of the box" for us, and how we changed that.

1. Introducing and Extending Chef
at WebMD
Adam Leff
Platform Technologies

2. I hate making slides.
Powerpoint is garbage. Keynote is okay. Deckset rules.
http://www.decksetapp.com

3. It all started with a
(seemingly) simple
task...

4. "Give me a few linux boxes, I'll
take care of it."
-- Adam Leff, before he regretted every choice he made in his
life leading up to this point

5. "Here you go, just use
Opsware to manage and
automate them."

6. The supposed process...
• Create packages
• Upload packages
• Create software policies
• Attach software policies to hosts
• Remediate hosts

7. The actual process...
• Grumble about creating packages while relearning .spec
files
• Create packages
• Upload packages
• Upload again because you don't know where they went
• Create software policies and attach

8. The actual process...
• Curse openly about how much faster I could've done this
with bash
• Remediate
• Don't get any good feedback on remediation progress
• Wait forever for remediation
• End up with inconsistent hosts
• (°□°︵

9. Enough drama... why did it suck?
• Remediation was slow (5+ minutes sometimes)
• No easy config file management
• No easy way to control the order of software policies

10. Here, have some java...

12. A Better Way

13. Post-Chef Metrics
• Cookbooks written: 1
• Recipes written: 6
• Hosts converged: 7
• Average converge time: 17 sec
• Average happiness level: limitless

14. So long, Opsware...

15. Hey, I noticed your hosts
aren't registered with
Opsware anymore. Want
me to fix it for you?

16. But while you're here, want to see something cool?

17. We're not a startup.

18. Paralyzing
Realizations
• Anyone with knife access can modify
my cookbooks
• Nothing (except cookbooks) are
versioned in the chef server

19. I don't trust my co-workers.

20. I don't shouldn't trust my co-workers.

21. Ops Come-to-$RELIGIOUS_FIGURE
• You're paid to be paranoid.
• You're paid to plan for the worst.

22. If you build it, they will come.
If you don't, they'll set up their own chef server.

23. Barriers to Upload
• Enforce some standards
• Do some basic testing
• Eliminate need for users to have knife to upload cookbooks

24. Chefkins
• Chef + Jenkins
• Sets up a new cookbook for the user
• User clones the git repo
• User commits and pushes
• Jenkins build is triggered
• Jenkins tests the cookbook
• Jenkins uploads the cookbook

25. But we want/need knife access,
too.
Well, shit.

26. Throw money at the problem
But Private Chef and multi-tenancy didn't solve anything for
us.

27. Rub some code on it
Knife talks to chef via HTTP, so what if we wrote a proxy?

28. And so it was written...

29. Line Cook
• Mimics a chef server, but still requires one
• Provides AuthN and AuthZ
• Versions everything
• Blocks cookbook uploads
To our users, it just feels like knife against a chef
server.

30. Live Demo
God, I hope this works.

31. A (related) tangent...
How do we track all our stuff?

32. CMDB

33. CMDB
Asset and Inventory
Management

34. CMDB
Asset and Inventory
Management
A Source of Truth

35. The Saurus of Truth

36. Saurus Objectives
• Inventory everything
• WebMD has over 300 applications!
• Develop a hierarchy
• Document owners
• Document relationships
• Arbitrary key/value data (a.k.a. extended attributes)

37. The Hierarchy
• Product Collection
• Product
• Application
• Component
• Instance

38. The Hierarchy
• Product Collection (Consumer)
• Product (WebMD)
• Application (Runtime)
• Component (IIS)
• Instance (server1:80)

39. Other Relationships
• Instance -- Environment
• Instance -- Host
• Host -- Datacenter

43. Back to Chef...

44. Attribute Inflexibility
• Environment attributes were getting messy
• Too many people had to have access to make changes
• Too many cookbook wrappers
• Even with debug_value, very difficult to troubleshoot.

45. Sensitive Data
Chef Vault works but regenerating the keys file can be painful
and slow.

46. Wait a sec...
What about those Saurus Extended Attributes?

47. Saurus + Chef Wishlist
• Let me mark an EA as sensitive so it's encrypted at rest and
not shown in the UI
• Give a host all its Extended Attributes, inheriting from the
hierarchy
• Never allow a host to get EAs that aren't its own

48. Saurus + Chef = ❤
• Saurus API endpoint for retrieving EA data
• Chef cookbook library helper for retrieving that data
• Authenticates using the Chef client key
• Returns a Chef Mash

49. {
myInstance =
{
id = 122136ce-7f84-4c22-b1db-c353abb2aa29,
name = myInstance,
created_at = 2014-09-22T00:00:00.000+00:00,
updated_at = 2014-09-22T18:00:00.000+00:00,
ip_address = 127.0.0.1,
port = 8080,
key_value_data =
{
myPassword =unbr8kable,
myValue =local_to_instance
}
}
}

50. Alright, what's the point?
• Chef server is not perfect
• Nothing is perfect
• Chef server is an awesome artifact server
• My business logic and requirements are different than yours
• Expect to invest time/resources/etc. to make something
right

51. Lawyers...

52. How to fix a technical problem
From least expensive to most expensive...
1. Throw hardware at it
2. Buy software for it
3. Write software for it
Try these in order if you can.
But it's OK if #3 is the answer.

53. Are you using Chef yet?
• Try it! -- getchef.com
• Learn it! -- learn.getchef.com
• If you don't like it, learn something else!
Life is too short to not automate.

54. Thank you!