Recommended
More Related Content
What's hot
What's hot (20)
Similar to Maintainability of Configuration Management Code
Similar to Maintainability of Configuration Management Code (20)
Recently uploaded
Recently uploaded (20)
Maintainability of Configuration Management Code
- 1. / Maintainability of Config Management Code Why is Everything So Awful, and What Can We Do About it? Clinton Wolfe DevOps Practice Lead @clintoncwolfe
- 2. Part 1: A Bit About Me and My History
- 3. id clintoncwolfe • Lead DevOps Consultant at OmniTI Computer Consulting • Config Management (CM) specialist in Chef and Ansible • Software Engineering wonk • Perl web developer in a former life
- 4. Perl, huh? • mod_perl developer 1996- 2011 • Projects start green, turn brown, hard to maintain • I became a proponent of maintainability
- 6. The Push for Maintainability TMTOWTDI
- 7. The Push for Maintainability "Some ways are Better than Others"
- 8. The Push for Maintainability "STOP DOING THAT. Use this instead"
- 9. Maintainawhat? The ease with which a system can be interacted with, over time, in order to: • diagnose problems
- 10. Maintainawhat? The ease with which a system can be interacted with, over time, in order to: • diagnose problems • make repairs
- 11. Maintainawhat? The ease with which a system can be interacted with, over time, in order to: • diagnose problems • make repairs • cope with changes in requirements
- 12. Maintainawhat? The ease with which a system can be interacted with, over time, in order to: • diagnose problems • make repairs • cope with changes in requirements • maximize useful life
- 14. Part 3: Some Factors Affecting Maintainability in Application Software
- 15. Arbitrary vs Necessary Complexity Arbitrary Necessary
- 17. Entropy / Bit Rot
- 18. Write vs Read
- 19. Turnover and Skill Variance
- 20. Yadda Yadda Yadda… • lots of other things impact Application maintainability • performance optimization • rapidly changing feature requests • languages, libraries, architectures go out of style • But I won't get into that stuff • It's hard • No good answers • Not as applicable to CM
- 21. Part 4: So Now Let's Do That to the Servers
- 22. It's Easy! "You don't have to know Ruby to use Chef!"
- 23. It's Easy! "You don't have to know Ruby to use Chef!"
- 24. Consequences, Schmonsequences rm –rf –no-preserve-root {foo}/{bar}
- 25. Ops Background != Dev Background
- 26. Test Fixtures
- 27. CM isn't good for everything
- 28. Part 5: Free Advice
- 29. Comment : Why, not What
- 30. Pre-Flight
- 31. KISS on Variable Storage
- 32. Use a Linter
- 33. Write Tests. Have Tests.
- 34. Use Ephemeral Test Machines
- 35. Use a Development Kit
- 36. Use a Code Generator
- 37. Use Code Reviews to Level-Set
- 38. Burn It All Down
- 39. Really, Really Burn It All Down
- 40. Thanks! @clintoncwolfe ansibledk.com omniti.com Let's follow up in OpenSpaces!
Editor's Notes
- This is my high school senior picture, from 1995. I started programming soon after this.
- Perl culture in the 90's and early 2000's was all about "There's more than one way to do it" – creative freedom, implicit permission to leverage the full flexibility of the language to do whatever you needed to get done.
- That kind of lead to a giant mess if you ever needed anyone else to maintain your code, though. So, a movement sprang up, as in many communities, to try to find best practices.
- Today, for those still active in the Perl community, there are modules available that encapsulate some of the ideas. There's nothing stopping you from making a big horrible mess of things, of course. So that's how the Perl community independently journeyed towards maintainability. But more broadly, what do we mean, when we say "maintainability"?
- Software has a strong tendency to become more complex over time. But not all complexity is justifiable. On the right, we have a Saturn V rocket performing the Apollo 17 mission, which sent 3 people through space (where you can die in many horrible ways), landed 2 of them on the Moon, and brought them all back home safely. Also, this time they brought a little car they could drive around. On the moon. On the left, we have a device intended to open an umbrella. Take a moment and consider the sort of goals that your systems have, and whether the amount of complexity involved makes sense.
- But why would complexity go up over time? Why would you make things like the umbrella opener? Things usually start simple. But then you have to glue on a new interface, because the Product team decided on a new integration. Also, now it has to be on mobile, and make toast. No design can anticipate all of these things – nor should they, that causes other problems – so things get glued on haphazardly. There is a lot of business pressure to ADD functionality, but there is rarely pressure to reduce complexity. As time goes on, some parts of the code are used less and less, but still have to stick around for legacy integrations.
- If you don't use your code often, you may find that external elements in the environment have changed to the point that your code is no longer functional. Nothing in the IT community stands still, and so we find that all code is forced to change. If code is written with the assumption that it will never have to change, because things will naively always remain the same, maintainability will never be a consideration – and if you don't build it to be maintainable, it won't happen by accident.
- At the time that you are writing code, you head is deeply in the problem – you have a result you want to obtain, and are aware of edge cases – and also the things you don't have to worry about. There is a lot of unspoken context in your head. And, as you write, you may also be running it, gradually improving on the code until it works, or works well enough. Someone who comes along later doesn't have the context. They don't know what you tried and what you didn't, and they don't know if you wrote it that weird way out of ignorance, cleverness, or as the only way to make it work because of an unrelated issue. Have you ever tried to figure out what a piece of code is doing, only to run git blame and discover that you were the one who wrote it, six months ago?
- Of course, as time goes on, people move on. Different people have different backgrounds, experiences, and skillsets. Some aspects of the codebase may become incomprehensible to the current maintainer. In really unhealthy situations, there may be legacy codebases that have complete vacancies – there might be no one who is familiar with the code for extended periods of time. When the code is next maintained, you have to rely on docs and testing, as well as a long discovery process. We have solid docs and great tests on all the old code, though, right?
- I'm been speaking so far about factors that impact all software development, but especially focusing on things that also impact code used by Operations, like Configuration Management code. There are a lot of other factors out there, which are less applicable.
- So, let's look at how things change when we try to start managing all the servers by writing software.
- Well, the good news, is that according to the tool vendors, all of this is going to be super easy. Also, their product is easier than the others.
- A picture of a baby has never filled me with such rage. It really doesn't matter how easy the *tool* is to use when the TASK is difficult.
- Back in April of this year, there were a pair of posts to ServerFault in which people described accidentally destroying "their entire company", using a similar command line. One purported to be using Ansible to do it. One of them turned out to be a hoax, and the the other sadly didn't. I won't dive into the details here, but I just wanted to point out that while application developers are working with things like inventory counts, ecommerce, etc – the subjects of this code is typically more recoverable. An infrastructure coder, on the other hand, is literally creating and destroying servers, load balancers, etc. Not only can you destroy the data, you can destroy the server, and the backup server, too. And what does greater consequences have to do with maintainability? If code is scary to change, you'll be afraid to change it. Code you're afraid to touch is code you can't maintain. So, naturally, we'd expect the people doing this work to have special training to ensure they are able to do the work safely, right?
- The people typically assigned to do CM work – who are often championing its adoption – usually come from an ops background. That's fantastic – they are exactly the right people, with the right enthusiasm and subject-matter expertise. But as soon as you start writing infrastructure code, you're not a sysadmin anymore – you're a developer. How many year of experience do you have as a developer? In particular, how strong are your instincts around input validation, edge cases, and test fixtures? Did you know there are people who know about such things? They're called Developers and QA Engineers. You can ask them for help. You've got a great relationship with them, right?
- One of the secrets QA Engineers know is that you don't have to test things against reality. Instead, you build facilities to simulate inputs and capture outputs, and isolate the components that you need to verify. When that component is a library file or a software module, those "text fixtures" are fairly straightforward. When the object you're testing is an entire configured webserver – with provisioned hardware, installed OS, configured services, middleware, and a deployed application, it's less obvious. So, many first-time config management coders simply do the obvious thing, and test against the real thing – possibly even production.
- Once you start using CM for the obvious things – like installing and configuring a service – your imagination takes flight. What else could we do with this powerful tool? Now that I'm spending less time doing boring, low-value tasks, perhaps I can use my basic skills in CM to automate something more difficult. Maybe adding and removing people from LDAP. Or, instead of configuring machines, maybe we could create them? And the networks between them? Oh, we could create entire staging environments! And snapshot production data into the staging environments for testing! And what about scale testing? Oh, I bet we could automate database failover! Some of those are good ideas, and some of them are terrible ideas. Don't do failover with CM.
- OK, so I've been talking – a lot - about the various factors that make software maintainability a hard goal, and CM code maintainability a particularly nasty problem. My hope is that you'll be able to apply those those factors to your own situation, and come up with some ideas to improve your specific situation. That said, here are some general things you can do.
- We'll start off with a nice, easy tip: write useful comments. You can comment in nearly all formats in use these days – except, sigh, JSON. The advice here isn't just to write comments, though. People to tend to write comments that say what they're doing, which is silly – the code is the part that actually takes actions. A description of this image – skeleton, cat, french horn, spooky girl with plumes coming out of her ears – does not explain why these elements are together. Instead, try to make comments that reflect Why the task needs to be done. In six months, or in the hands of another teammate, having insight into your intent is much more valuable than insight into your implementation
- One of the most basic things you can do to increase safety (and thus make it safer to run and maintain your code) is to add validation checks. If you rely on certain variables to be set – like, which environment to be targeting – you can check for those variables and ensure they have a sane value. This is used a lot in aerospace. Above, you see a section of the Space Shuttle's control panel. When the shuttle launched, the main engines – the 3 on the back of the orbiter itself – would start up 6 seconds before the solid rocket boosters. Once the SRBs were ignited, the craft was committed to a liftoff attempt, but until then, they could run tests on the engines and abort on the pad. How many times do you think that happened? 6. Not a big deal. This is really isn't optional. When you were writing Bash scripts, validation was kind of hard, and an afterthought if anything. But you're using much more expressive languages, and so can keep you – and especially your teammates – from causing great harm. Validation can also make the code somewhat self-documenting. They key point is to perform the validations before you take any actions. If anything fails validation, you will be able to give a helpful message, and abort the run before any harm is done.
- Related to variable validation, you may be thinking about where to get those variable values. Both Ansible and Chef provide multi-tiered, override-driven defaulting systems. They are both completely infuriating, in their own ways. Chef is way, way too complex, offering too many levels and mechanisms, and a two-pass execution model that makes dynamically setting variables really dangerous. Ansible is simpler, but much buggier and poorly documented; while each of the 5 mechanisms will always have the same priority, your context constantly shifts when running a task or a play, and dynamically setting values doesn't work as expected, either. The long and short of it is that each org needs to identify the simplest possible approach to this that works for them, and then never, ever vary from that.
- So how can you make sure that – for example – someone isn't using twelve kinds of attribute definitions in a Chef cookbook? I guess you could grep for it, right? Well, it turns out there are much better tools than grep. Developers have been using specialized search tools – called "linters", because they help you pick the lint off -- that search through your code (without running it) and apply a bunch of different rules. Some rules might look for formatting, some might look for bad ideas of various kinds, but all linters also allow you to write custom rules – so you can make a rule that says where you want Chef attributes (or Ansible variables) defined.
- Did you know that testing is easy? You can use tools like inspec or Serverspec to write super-simple tests. Things like "apache should be installed" or "port 80 should be listening". The goal is to have a small set of smoke tests. If they pass, there might still be something horribly wrong, but if they fail, you know there is, and you can start addressing it. HAVING tests is more important than writing tests, though. When you have tests, your code becomes more maintainable, because you can make a change, run the tests, and verify that you didn't break any existing functionality. That's a huge confidence booster.
- So where are you going to run these tests? Please don't do it on anything you care about, like a QA environment or (shudder) production. The best workflows allow you to create temporary machines, run your new code, and destroy them, without waiting on anyone's permission. IaaS systems like AWS, GCE, or even local Vagrant/Virtualbox setups can all do this.
- So I've mentioned linters, and testing, and getting setup with an ephemeral testing system, and so one. That all sounds like a pain to set up. Well, it turns out that there are kits that contain all of that and more. ChefDK and AnsibleDK are both kits that contain all of those things. You just download them, install as an OS package, and a whole bunch of tools just magically work together. You don't have to invent that, just go download them.
- Included in both AnsibleDK and ChefDK is a code generator. A code generator is a thing that you run when you start a project, that lays out all the files, creates a bunch of small integrations (like setting up your git hooks and your IaaS credentials), and so on. They are pretty flexible, so you can customize them to your team's needs. The big advantage here is that you can use this to level-set how projects start. When you ask a junior engineer to make a new cookbook by running chef generate cookbook, and you know it will generate a project that already has the linter plugged in, and Test Kitchen setup, you know that they are going to start off having tools telling them when they are doing something wrong, regardless of their previous experience. Importantly, it also means that you projects will all at least start off looking alike, which makes it easier to transfer knowledge between projects.
- Assuming you're using version control that supports a merge-request workflow – which anything git-based certainly does – you have the opportunity to use tools like Github pull requests. If this is your first exposure to this, it may seem like it is all about control: who gets to accept things into the codebase. But more importantly in the long term is the ability to collaborate and communicate on proposed changes using comments in a web UI. This has a number of positive effects: more people on the team know what is going on; you have a record of the thought processes that went into design decicions, and junior engineers can ask questions. Some teams go so far as to have junior team members explain changes to the rest of the team. If the change isn't understandable, it needs to be simplified. Over time, the code becomes clearer, everyone understand that they are writing for people not computers, and skill levels increase.
- A key difference between application cade and configuration managemnt code is its expected lifetime. While app code may be in service for several years or even decades, CM code – even the oldest – has only been in use for 2-3 years. The code base is also typically much smaller. You have much more ability tto simply start over from scratch. Your first CM project may be a disaster; but you can scrap it and start over with lessons learned pretty easily.
- Increasingly, though, the need for true configuration management is starting to go away. The era of long-lived maches that we carefully manage through a long life of incremental changes – which is what CM was designed for – is tapering off. If you are using a container-based approach, you might use CM to build the initial image; or you might use something simpler like Packer, since you don't need the idempotency features of a CM tool. If your application developers are moving on to something like Serverless, you may find that there is no need at for CM – it may just be used for building out supporting infra. Every tool has an era. The Chef code we wrote in 2012 is very different than the code we write today. I don't know what CM code we'll be writing in 2020, but I imagine it will be different, or less.
- AND on that happy note, that's my time.