Successfully reported this slideshow.
Your SlideShare is downloading. ×

ChefConf 2015 Cleaning up the Kitchen

More Related Content

ChefConf 2015 Cleaning up the Kitchen

  1. 1. Cleaning up the Kitchen
  2. 2. Cleaning up the Kitchen@ablythe
  3. 3. Lessons • Be Patient/Be Hungry • Look outward – Ask Community – Ask Chef • You can’t please all the people • Know your constraints and work within them @ablythe
  4. 4. Recommendations • Have a focused team - Configuration Management does not just happen • Focus on the Workflow • Incent your community to work together • Define how your collective organization should use roles/environments/data bags • Educate!! • Involve Your Friends at Chef @ablythe
  5. 5. @ablythe
  6. 6. Aaron Blythe @ablythe
  7. 7. Founded in 1979
  8. 8. 2011 2012 2014 20152013 Journey With @ablythe
  9. 9. 2011 2012 2014 20152013 @ablythe
  10. 10. 2011 2014 20152013 2012 @ablythe
  11. 11. @ablythe
  12. 12. @ablythe
  13. 13. ON @ablythe
  14. 14. January 2012 – Late March 2012 @ablythe
  15. 15. 2011 2012 2014 2015 2013 We 50+ @ablythe
  16. 16. Which Chef Server? What is the admin password? How is this being backed up? Where did this cookbook come from? Is this the original cookbook or a fork? Why is this in production, but not in Staging? Is this wiki page up to date with which Server to connect to? Can we double bootstrap it? @ablythe
  17. 17. @ablythe
  18. 18. @ablythe
  19. 19. • Value Proposition of Chef: – Infrastructure as Code • Repeatable/Testable Deployments • Consistency - Staging Like Production @ablythe
  20. 20. In Kansas City… One Night Only • Adam Jacob!! @ablythe
  21. 21. Later that week… OpsInfra @ablythe
  22. 22. Later that Month… "I have never been in a room with that many system engineers that were all smiling, almost giddy. It was as if we all knew a secret." @ablythe
  23. 23. @ablythe
  24. 24. Paul Edelhertz - Interview • Senior VP Customer Ops – Chef • Former CEO – Empath, Inc. (bought by Navigant) @ablythe
  25. 25. Paul Edelhertz - Interview • Ability to persuade through writing is becoming a lost art – Make the message short – State exactly how I feel – Different mediums but repeat the message over and over to many people @ablythe
  26. 26. Chef – “The List” • Fuel the love of Chef • Make it easy to be successful • Give organizations reasons to buy • Be their favorite company to work with • Speed matters, be disruptive @ablythe
  27. 27. We Believe… • A minimal number of associates need to directly touch nodes (approaching zero) • These tools matter – improve the lives of our users • • Tools should be hardened - so that we can find the root cause and drive corrective action @ablythe
  28. 28. Identity • Tools Team • Culture Team @ablythe
  29. 29. @ablythe
  30. 30. @ablythe
  31. 31. @ablythe
  32. 32. Workflow Step 1: Source Code Step 2: ????? Step 3: Production (Profit!!!) @ablythe
  33. 33. @ablythe
  34. 34. @ablythe
  35. 35. @ablythe
  36. 36. RFC’s @ablythe
  37. 37. Conway’s Law – Educate who you need to 2 classes * 15 attendess * Re-delivering to 15-30/month * Over 1 year __________________ 200+ @ablythe
  38. 38. $20/Class @ablythe
  39. 39. 2011 2012 20152013 2014 Provide the Platform @ablythe
  40. 40. Migration Workshop • 1st rule of teaching: – Control all variables other than the concept you are focusing on @ablythe
  41. 41. Current State of Cookbooks • elf • 129 Cookbooks listed – 87% of cookbooks have a metadata.rb. (112/129) – 79% of cookbooks have a cookbooks folder (102/129) – 56% of cookbooks have a Rakefile. (72/129) – 33% of cookbooks have a Berksfile. (42/129) – 30% of cookbooks have a Vagrantfile. (39/129) – 24% of cookbooks have a pom.xml. (31/129) @ablythe
  42. 42. Good Migrations @ablythe
  43. 43. Success Engineer @ablythe
  44. 44. Stephen Lauck “Usually I show up and I am helping companies go from 0 to 20. You are clearly going from 40 to 60. Keep me in the loop, because I want to continue to be part of your success. We all know the companies that will be able to move fast will be the companies that remain successful in the market.” @ablythe
  45. 45. • Push Jobs • Analytics • OpenStack • SuperMarket @ablythe
  46. 46. Push Jobs • Value: Orchestration • Review: Overall positive, have-to-have feature for most of our teams • Drawbacks: Push Jobs 1.x is not very mature or feature rich. We have seen issues with Networking. @ablythe
  47. 47. Analytics • Value: Compliance/Auditing for non-coders • Review: Interesting need to see latest version. We get asked about creating this functionality regularly… Future features look promising. • Drawbacks: Early versions search slightly confusing. Filtering was not working right so very busy. @ablythe
  48. 48. OpenStack Integration • Decision (At this point): Heat Templates • Still really want to see chef-provisioning mature @ablythe
  49. 49. SuperMarket • Value: Helps lock down our pipeline • Review: Decided that we need a better pipeline first. • Drawbacks: Early on was not ready for RHEL, hacked something together, but need to revisit. Authorization model not there for an Enterprise (CCLA not applicable). @ablythe
  50. 50. 2011 2012 2014 2015 2013 March toward Continuous Integration/ Continuous Delivery@ablythe
  51. 51. Where are we now? • We are at about the 1 year mark of a focused well oiled system • Have over 3000 Chef nodes involved with 100’s of separate systems/sub-systems • Over 400 user accounts created on the server @ablythe
  52. 52. Wat ta do about CD? • Jenkins – Workflow Plugin • • Chef Delivery • All the wonderful things we have seen at the booths here @ablythe
  53. 53. Blocker – Sensitive Flag user 'zabbix' do shell '/sbin/nologin' comment 'Zabbix Monitoring System' supports manage_home: true node.run_state['zabbix_password'] = SecureRandom.base64(36) password node.run_state['zabbix_password'] gid 'zabbix' unless platform_family?('windows') sensitive true action [:create, :lock] end end @ablythe
  54. 54. Centralized vs. Decentralized @ablythe
  55. 55. Open Source • 6 associates contribute to core Chef • Another 15 or so contribute to Cookbooks in SuperMarket @ablythe
  56. 56. 2011 2012 2014 20152013 Journey With @ablythe
  57. 57. Lessons • Be Patient/Be Hungry • Look outward – Ask Community – Ask Chef • You can’t please all the people • Know your constraints and work within them @ablythe
  58. 58. Recommendations • Have a focused team - Configuration Management does not just happen • Focus on the Workflow • Incent your community to work together • Define how your collective organization should use roles/environments/data bags • Educate!! • Involve Your Friends at Chef @ablythe
  59. 59. What do you think? @ablythe Community room next session 3-4PM @ablythe

Editor's Notes

  • We work for this company.
  • Ridden the technology wave for over 35 years starting with consulting, fax machines, green screens. Our core competency is HealthCare IT

    Bring Manufacturing Automation to the Medical Lab, and over the past 30 years we’ve expanded
  • Our Data Centers now host software / services supporting 17% of US hospital Beds. (Doesn’t count clients who run their own data centers)

    This is from a smart room at a client in Florida.

    Everything is connected into a patient centric database – or what we refer to as the EMR or Electronic Medical Record.
  • We have 20,000+ Associates, 24+ countries implementing solutions across the entire continuum of care, not just in the hospital.
  • Cerner has been in the HealthCare Automation business for 35 years
    We have brought Chef along on that journey for the last 5 years.
  • 2011 only a few people at Cerner had even heard of Chef. Many of us didn’t even know about one another…
  • In 2012 we began embarking on a major project using Big Data and Mobile technologies.
    This was a fairly big departure from our usual SQL on Unix and Application on Windows stack.
    With many decisions being made quickly mistakes were made.

    There wasn’t a team dedicated to Chef, just a collective of engineer.
  • First of which was to fork Chef to get past a minor issue we were having with Red Hat at the time.
  • Since we were on a fork of the Gem we then needed to maintain our own version of abootstrap template.
  • Both Chef and RVM <click>
    said not to run on each other… But… <click>
    We Know Best… <click>
  • Remember this is around March 2012..

    Like immediately before Chef Omnibus hit this scene…
  • We finished out the 2012 year and everyone was loving Chef.
    Possibly too much love… There were over 50 Open Source Chef Servers around the company that we knew about.
    The reason for this was that the authorization model was such that it was simpler to set up another Chef Server for each cluster of applications for each environment (dev, staging, production etc.)
    Meaning non-existent
  • This lead to a mess of questions.

    Basically we had a messy kitchen…
  • Wait I didn’t say we were filthy hoarders… You know…
  • You know… messy like we had been cooking recently and took a break to play video games… I’ll clean that up later.
  • This had been in the works… but it was decided that we would build a “DevOps” team… I know that is a contraversial thing to say so I will come back to that.
    Really at this point the vision is a “tools” team that also helps educate.
  • Paul and I connected on our love of writing.
  • Paul, Adam Jacob, and a few others came together to create.

    This is not the current list but was the list at the time I spoke with Paul. You saw a more updated version this morning. This is iterative just like everything else.
  • I highly recommend that all of you in this room that you do this exercise regardless of the function of your team…
    Block off 2 hours and talk as a group and write on whiteboards simply about who you are…
    If you are the leader of your group… hell even if you are not… do some reading the day before about Brainstorming techniques. Do things to encourage the group to think together and be accepting.
    Defining your character is really freeing.
  • We learn
    We implement
    We help
    We teach
  • Consultant
  • Many (most?) associates who use Chef simply want:
    the overall experience to be simplified
    rules to be defined and followed
    so they can deploy apps and services with high quality and consistency.  The olive is a symbol of peace, wisdom, and glory.
    No matter what we land on, there will be those who want to do it slightly different.  This is healthy, and we love the users who want to make the Chef Ecosystem better for everyone other user here at Cerner.  A pimento pepper is one of the sweetest of all peppers and considered a delicacy throughout the world.
  • The Workflow
  • A lot more automated in Dev.
    Manual commands in Production
  • Simple Clean Straight Forward.

    Jenkins Cookbook with a rebrand of the looks so users were not confused with the current instance of Jenkins they were using to Integrate Java and Ruby code.
  • Funny story about the naming of Spork. I just got back from a much needed vacation, and David from our team had created this icon.
    I get it like Jon Cowie’s knife-spork – it’s a workflow thing for us. <click>
    Ok then like from the popular rails parrellizer for your tests <click>
    Basically just a food meme thing. <click>
    But we are going to use it for our workflow and run tests of our cookbooks with it right?
  • Internal RFC’s
    Goodness in involving your consumers early
    Double-edged sword as there were a few delays over the next couple months (our team was also building an Oozie workflow Kick off tool and tracking application) of you posted that months ago, don’t we have that yet?
  • Our small team was not going to change the organization structure. Nor were we going to get the budget to educate everyone.
  • As long as we are talking about being frugal/economical

    50 cents a day for the VM’s
  • 2014 -
  • we walked through how to add Berkshelf, Vagrant, ChefSpec, Test-Kitchen in a room of like 50 people.
    I would introduce the concept, explain why it was important, walk through how to do it on a cookbook I had open to show that it could be done.
    Then we had everyone in the room try it.
  • As Chef has grown we have had a few Success Engineers. Last year we were hooked up with Ryan Cragun who has been perfect for us.
  • Future features – like hitting the api from a cookbook.

  • Being in the Enterprise we are still at the point where we need to ask for a VM with a fair amount of lead time before we actually need it.

    This year our operations folks will be providing an internal cloud to us built on OpenStack. We have been able to prove out all the concepts we need using Heat templates.
  • Hoping to come home from this trip re-energized as we have the past two years. And come home with tips on how to move forward on this…
  • HealthCare Automation business for 35 years
  • We now have our head on straight, and we have eliminated a ton of the confusion.
  • After trying many different things
  • Do all of this extra work encrypting data using chef-vault or encrypted data bags, but then this is undone with what we consider to be the killer feature of Chef Reporting.
  • I have heard from some people that I respect and look up to this week that you don’t need a DevOps team. In our situation, I would argue that we have needed this and it has been beneficial. However what I struggle with repeatedly is where is the line of what should be centralized. Similar to what Jamie Windsor said yesterday we expect the team writing the application to own the cookbook currently, but continue to struggle with who should own the “Base Linux Cookbook” or policy based cookbooks that affect many teams.
  • Thank you to Chef and the Apache Foundation for Hadoop projects for helping us overcome our Not-Invented-Here Syndrome.
  • HealthCare Automation business for 35 years