Configuration management - A "love" story

Editor's Notes

1. This talk was presented in Infracoders Melbourne on 11th March 2014 by Javier Turegano (@setoide) I tried two experiments: - Tell a parallel story (love, it could easily be horror) - Sketching all the slides in a whiteboard It is based on a real story (not 100% accurate), I took a few licences
2. First love - Perfect - Will last forever - All the things that you need now and ever - First experiences
3. A long, long time ago: small teams of devs and ops
4. Only a few servers Manual deployments Config via SSH
5. As time passes teams grow and complexity and number of servers as well
6. It was love at first sight
7. Puppet running in daemon mode in all the servers Config is stored in git and pushed to the puppet masters
8. Automation works! Hurray
9. But as the number of modules grew and dependencies started to be more complex, single errors updating a service could take completely unrelated farms down. No testing.
10. Big lost of confidence Manual test was required before deployment
11. The decision was made to stop running daemon mode
12. In order to deploy a change, an ops will write a change, push it to git and then test in the node through ssh using the noop option, if it worked as expected then they will use puppet agent to apply the change in that node.
13. But, what about the rest of the nodes... Slowly they started to be out of sync and whenever you were going to make a change you could find a bunch of unapplied changes and errors waiting for you. Confidence on doing changes continued to decrease.
14. People in a rush would do the changes manually in the servers and...
15. ...afetwards commit the changes to the repository. The first love didn't look so perfect at that stage.
16. And then someone else gets into your live... fresh, new, exciting...
17. But before that, let's expore the problem: - A single staging environment for multiple teams of development willing to test changes. Waiting and lots of work to maintain the system.
18. Dev story
19. Who could help us? Gandalf, of course. (Codename for the project to solve the problem)
20. Gandalf came with a lot of new ideas: - Use AWS cloud to avoid the limits of physical infrastructure and enable on demand resources - Chef, the blond girl of this story, as dealing with fixing the relationship with puppet was a lot of work... let's start fresh. - The gandalf tooling would allow development teams to create their own stack with the full or limited functionality to test.
21. Now a team can deploy a full end to end environment as needed
22. And then a different developer or team can create another environment with the components needed for his test.
23. The idea worked really well... but going to production was still a problem.
24. The difference between dev/test and prod could end in problems
25. After trying different experiences and when things get too complicated is time to simplify and commit.
26. As time has evolved the company has adopted lots of principle of agile and devops culture. Cross functional teams are formed and work in parallel streams in line of businesses. A central platform team takes care of the common elements of the stack: datacentres, cloud, os, common services, etc... while teams are fully reponsible for the application life cycle.
27. The platform team provides an unified platform image that can run in dev/test and staging. The image is created using puppet and includes all the common bits required (ldap, nfs, splunk forwarder, newrelic, etc)
28. New versions of the platform image are tested first in dev and staging before been promoted to production.
29. LoB teams package their application using rpms and their CIs upload them to a common repository. As the platform image provides lots of common services the application dependencies decrease enormously and simplifies the full process of deploying.
30. The combination of the rpm and the platform image can be tested.
31. When confident you can deploy to production.
32. In order to be able to easily apply different configuration to dev/test to prod environments a configuration service can provide configuration variables different to each environment. This way the combination rpm+image is the same in all environments.
33. And then you have kids... you will assume that they will follow your steps but they have their own preferences...
34. At this point, more environments end up in the cloud. One of our principles is to value autonomy/accountability over economies of scale. Teams can make their own choices as they will be the ones maintaining the systems.
35. The idea is that sharing still happens but still of imposing a technology/methodology/practice we use other mechanism to share knowledge: - Guilds: where people share their wins/loses and experiences and allow other people to pick up ideas. For example: the cloud guild, the delivery engineering guild, security guild or Ops Dojo. - Open Source model: A team creates a tool or service that other teams are interested in. A custodian is designated and guides the roadmap but other teams/individuals can contribute or even fork if they have different needs.
36. For example team 1 decides that they want to use a combination of technologies.
37. While maybe team 3 has a difference preference according with their needs/skills. Through the sharing process some ideas will be reused by other teams when they are proven sucessful.
38. Next... are you up for the cahllenge? We are hiring: http://careers.realestate.com.au