Cleaning Up the Kitchen: Migrating to Enterprise Chef From Open Source - ChefConf 2015
•Download as PPTX, PDF•
0 likes•390 views
In 2013 we had over 50 open source Chef servers running at the company. Cookbooks were being used to great benefit over previous methods, however it was getting confusing. Cookbooks were being forked and changed without the names being refactored. It was sort of a messy kitchen. We decided to align with Chef and move to the Enterprise Chef offering. Join me as I walk through many lessons learned in 2014 of getting 400+ developers from 40+ teams to learn to cook together. The result has been getting thousands of nodes into a more centrally manageable location. After I share how we got our kitchen in order, I'll share our plans and progress for 2015 of providing common building blocks of continuous delivery within a large company. https://youtu.be/EHIzjrg0fFU
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (19)
Similar to Cleaning Up the Kitchen: Migrating to Enterprise Chef From Open Source - ChefConf 2015
Similar to Cleaning Up the Kitchen: Migrating to Enterprise Chef From Open Source - ChefConf 2015 (20)
More from Chef
More from Chef (20)
Recently uploaded
Recently uploaded (20)
Cleaning Up the Kitchen: Migrating to Enterprise Chef From Open Source - ChefConf 2015
- 1. Cleaning up the Kitchen
- 2. Cleaning up the Kitchen@ablythe
- 3. Lessons • Be Patient/Be Hungry • Look outward – Ask Community – Ask Chef • You can’t please all the people • Know your constraints and work within them @ablythe
- 4. Recommendations • Have a focused team - Configuration Management does not just happen • Focus on the Workflow • Incent your community to work together • Define how your collective organization should use roles/environments/data bags • Educate!! • Involve Your Friends at Chef @ablythe
- 11. 2011 2012 2014 20152013 Journey With @ablythe
- 15. @ablythe
- 16. ON @ablythe
- 17. January 2012 – Late March 2012 @ablythe
- 18. 2011 2012 2014 2015 2013 We 50+ @ablythe
- 19. Which Chef Server? What is the admin password? How is this being backed up? Where did this cookbook come from? Is this the original cookbook or a fork? Why is this in production, but not in Staging? Is this wiki page up to date with which Server to connect to? Can we double bootstrap it? @ablythe
- 20. @ablythe
- 21. @ablythe
- 22. • Value Proposition of Chef: – Infrastructure as Code • Repeatable/Testable Deployments • Consistency - Staging Like Production @ablythe
- 23. In Kansas City… One Night Only • http://www.meetup.com/DevOps-Kansas-City/ Adam Jacob!! @ablythe
- 25. Later that Month… "I have never been in a room with that many system engineers that were all smiling, almost giddy. It was as if we all knew a secret." @ablythe
- 26. @ablythe
- 27. Paul Edelhertz - Interview • Senior VP Customer Ops – Chef • Former CEO – Empath, Inc. (bought by Navigant) @ablythe
- 28. Paul Edelhertz - Interview • Ability to persuade through writing is becoming a lost art – Make the message short – State exactly how I feel – Different mediums but repeat the message over and over to many people @ablythe
- 29. Chef – “The List” • Fuel the love of Chef • Make it easy to be successful • Give organizations reasons to buy • Be their favorite company to work with • Speed matters, be disruptive @ablythe
- 30. We Believe… • A minimal number of associates need to directly touch nodes (approaching zero) • These tools matter – improve the lives of our users • • Tools should be hardened - so that we can find the root cause and drive corrective action @ablythe
- 31. Identity • Tools Team • Culture Team @ablythe
- 32. @ablythe
- 33. @ablythe
- 34. @ablythe
- 35. Workflow Step 1: Source Code Step 2: ????? Step 3: Production (Profit!!!) @ablythe
- 36. @ablythe
- 37. @ablythe
- 38. @ablythe
- 39. RFC’s @ablythe
- 40. Conway’s Law – Educate who you need to 2 classes * 15 attendess * Re-delivering to 15-30/month * Over 1 year __________________ 200+ https://learn.chef.io/index.html @ablythe
- 42. 2011 2012 20152013 2014 Provide the Platform @ablythe
- 43. Migration Workshop • 1st rule of teaching: – Control all variables other than the concept you are focusing on @ablythe
- 44. Current State of Cookbooks • https://wiki.ucern.com/display/ChefEcosystem/Booksh elf • 129 Cookbooks listed – 87% of cookbooks have a metadata.rb. (112/129) – 79% of cookbooks have a cookbooks folder (102/129) – 56% of cookbooks have a Rakefile. (72/129) – 33% of cookbooks have a Berksfile. (42/129) – 30% of cookbooks have a Vagrantfile. (39/129) – 24% of cookbooks have a pom.xml. (31/129) @ablythe
- 47. Stephen Lauck “Usually I show up and I am helping companies go from 0 to 20. You are clearly going from 40 to 60. Keep me in the loop, because I want to continue to be part of your success. We all know the companies that will be able to move fast will be the companies that remain successful in the market.” @ablythe
- 48. • Push Jobs • Analytics • OpenStack • SuperMarket @ablythe
- 49. Push Jobs • Value: Orchestration • Review: Overall positive, have-to-have feature for most of our teams • Drawbacks: Push Jobs 1.x is not very mature or feature rich. We have seen issues with Networking. @ablythe
- 50. Analytics • Value: Compliance/Auditing for non-coders • Review: Interesting need to see latest version. We get asked about creating this functionality regularly… Future features look promising. • Drawbacks: Early versions search slightly confusing. Filtering was not working right so very busy. @ablythe
- 51. OpenStack Integration • Decision (At this point): Heat Templates • Still really want to see chef-provisioning mature @ablythe
- 52. SuperMarket • Value: Helps lock down our pipeline • Review: Decided that we need a better pipeline first. • Drawbacks: Early on was not ready for RHEL, hacked something together, but need to revisit. Authorization model not there for an Enterprise (CCLA not applicable). @ablythe
- 53. 2011 2012 2014 2015 2013 March toward Continuous Integration/ Continuous Delivery@ablythe
- 54. Where are we now? • We are at about the 1 year mark of a focused well oiled system • Have over 3000 Chef nodes involved with 100’s of separate systems/sub-systems • Over 400 user accounts created on the server @ablythe
- 55. Wat ta do about CD? • Jenkins – Workflow Plugin • Go.cd • Chef Delivery • All the wonderful things we have seen at the booths here @ablythe
- 56. Blocker – Sensitive Flag user 'zabbix' do shell '/sbin/nologin' comment 'Zabbix Monitoring System' supports manage_home: true node.run_state['zabbix_password'] = SecureRandom.base64(36) password node.run_state['zabbix_password'] gid 'zabbix' unless platform_family?('windows') sensitive true action [:create, :lock] end end @ablythe
- 58. Open Source • 6 associates contribute to core Chef • Another 15 or so contribute to Cookbooks in SuperMarket @ablythe
- 59. 2011 2012 2014 20152013 Journey With @ablythe
- 60. Lessons • Be Patient/Be Hungry • Look outward – Ask Community – Ask Chef • You can’t please all the people • Know your constraints and work within them @ablythe
- 61. Recommendations • Have a focused team - Configuration Management does not just happen • Focus on the Workflow • Incent your community to work together • Define how your collective organization should use roles/environments/data bags • Educate!! • Involve Your Friends at Chef @ablythe
- 62. What do you think? @ablythe aaron.blythe@gmail.com http://www.slideshare.net/AaronBlythe/ Community room next session 3-4PM @ablythe
Editor's Notes
- We work for this company.
- Ridden the technology wave for over 35 years starting with consulting, fax machines, green screens. Our core competency is HealthCare IT Bring Manufacturing Automation to the Medical Lab, and over the past 30 years we’ve expanded
- Our Data Centers now host software / services supporting 17% of US hospital Beds. (Doesn’t count clients who run their own data centers) This is from a smart room at a client in Florida. Everything is connected into a patient centric database – or what we refer to as the EMR or Electronic Medical Record.
- We have 20,000+ Associates, 24+ countries implementing solutions across the entire continuum of care, not just in the hospital.
- Cerner has been in the HealthCare Automation business for 35 years We have brought Chef along on that journey for the last 5 years.
- 2011 only a few people at Cerner had even heard of Chef. Many of us didn’t even know about one another…
- In 2012 we began embarking on a major project using Big Data and Mobile technologies. This was a fairly big departure from our usual SQL on Unix and Application on Windows stack. With many decisions being made quickly mistakes were made. There wasn’t a team dedicated to Chef, just a collective of engineer.
- First of which was to fork Chef to get past a minor issue we were having with Red Hat at the time.
- Since we were on a fork of the Gem we then needed to maintain our own version of abootstrap template.
- Both Chef and RVM <click> said not to run on each other… But… <click> We Know Best… <click>
- Remember this is around March 2012.. Like immediately before Chef Omnibus hit this scene…
- We finished out the 2012 year and everyone was loving Chef. Possibly too much love… There were over 50 Open Source Chef Servers around the company that we knew about. The reason for this was that the authorization model was such that it was simpler to set up another Chef Server for each cluster of applications for each environment (dev, staging, production etc.) Meaning non-existent
- This lead to a mess of questions. Basically we had a messy kitchen…
- Wait I didn’t say we were filthy hoarders… You know…
- You know… messy like we had been cooking recently and took a break to play video games… I’ll clean that up later.
- This had been in the works… but it was decided that we would build a “DevOps” team… I know that is a contraversial thing to say so I will come back to that. Really at this point the vision is a “tools” team that also helps educate.
- Paul and I connected on our love of writing.
- Paul, Adam Jacob, and a few others came together to create. This is not the current list but was the list at the time I spoke with Paul. You saw a more updated version this morning. This is iterative just like everything else.
- I highly recommend that all of you in this room that you do this exercise regardless of the function of your team… Block off 2 hours and talk as a group and write on whiteboards simply about who you are… If you are the leader of your group… hell even if you are not… do some reading the day before about Brainstorming techniques. Do things to encourage the group to think together and be accepting. Defining your character is really freeing.
- We learn We implement We help We teach
- Consultant
- Many (most?) associates who use Chef simply want: the overall experience to be simplified rules to be defined and followed so they can deploy apps and services with high quality and consistency. The olive is a symbol of peace, wisdom, and glory. No matter what we land on, there will be those who want to do it slightly different. This is healthy, and we love the users who want to make the Chef Ecosystem better for everyone other user here at Cerner. A pimento pepper is one of the sweetest of all peppers and considered a delicacy throughout the world.
- The Workflow
- A lot more automated in Dev. Manual commands in Production
- Simple Clean Straight Forward. Jenkins Cookbook with a rebrand of the looks so users were not confused with the current instance of Jenkins they were using to Integrate Java and Ruby code.
- Funny story about the naming of Spork. I just got back from a much needed vacation, and David from our team had created this icon. I get it like Jon Cowie’s knife-spork – it’s a workflow thing for us. <click> Ok then like from the popular rails parrellizer for your tests <click> Basically just a food meme thing. <click> But we are going to use it for our workflow and run tests of our cookbooks with it right?
- Internal RFC’s Goodness in involving your consumers early Double-edged sword as there were a few delays over the next couple months (our team was also building an Oozie workflow Kick off tool and tracking application) of you posted that months ago, don’t we have that yet?
- Our small team was not going to change the organization structure. Nor were we going to get the budget to educate everyone.
- As long as we are talking about being frugal/economical 50 cents a day for the VM’s
- 2014 -
- we walked through how to add Berkshelf, Vagrant, ChefSpec, Test-Kitchen in a room of like 50 people. I would introduce the concept, explain why it was important, walk through how to do it on a cookbook I had open to show that it could be done. Then we had everyone in the room try it.
- As Chef has grown we have had a few Success Engineers. Last year we were hooked up with Ryan Cragun who has been perfect for us.
- Future features – like hitting the api from a cookbook.
- Being in the Enterprise we are still at the point where we need to ask for a VM with a fair amount of lead time before we actually need it. This year our operations folks will be providing an internal cloud to us built on OpenStack. We have been able to prove out all the concepts we need using Heat templates.
- Hoping to come home from this trip re-energized as we have the past two years. And come home with tips on how to move forward on this…
- HealthCare Automation business for 35 years Chef
- We now have our head on straight, and we have eliminated a ton of the confusion.
- After trying many different things
- Do all of this extra work encrypting data using chef-vault or encrypted data bags, but then this is undone with what we consider to be the killer feature of Chef Reporting.
- I have heard from some people that I respect and look up to this week that you don’t need a DevOps team. In our situation, I would argue that we have needed this and it has been beneficial. However what I struggle with repeatedly is where is the line of what should be centralized. Similar to what Jamie Windsor said yesterday we expect the team writing the application to own the cookbook currently, but continue to struggle with who should own the “Base Linux Cookbook” or policy based cookbooks that affect many teams.
- Thank you to Chef and the Apache Foundation for Hadoop projects for helping us overcome our Not-Invented-Here Syndrome.
- HealthCare Automation business for 35 years Chef