Fear of the Bus - Heidi Waterhouse - DevOpsDays Tel Aviv 2016
•Download as PPTX, PDF•
0 likes•224 views
DevOps is such a dynamic and flexible part of an organization that it is the absolute hardest thing to get documented, and the place where it really makes a difference to burnout if you have managed the miracle.
More Related Content
Similar to Fear of the Bus - Heidi Waterhouse - DevOpsDays Tel Aviv 2016
Similar to Fear of the Bus - Heidi Waterhouse - DevOpsDays Tel Aviv 2016 (20)
More from DevOpsDays Tel Aviv
More from DevOpsDays Tel Aviv (20)
Recently uploaded
Recently uploaded (20)
Fear of the Bus - Heidi Waterhouse - DevOpsDays Tel Aviv 2016
- 1. @wiredferret #docsxdevop Fear of the Bus Documentation and Devops DevOpsDays Tel Aviv 2016 Heidi Waterhouse @wiredferret h.waterhouse@gmail.com heidiwaterhouse.com
- 2. @wiredferret #docsxdevop Are you afraid of the bus?
- 3. @wiredferret #docsxdevop Mt. Carmel Forest Fire
- 4. @wiredferret #docsxdevop Racing to stay the same place
- 6. @wiredferret #docsxdevop Bridges rely on redundancy
- 7. #docsxdevop@wiredferret Core principles Everyone can touch the docs Content matters more than presentation Docs reduce burnout
- 8. @wiredferret #docsxdevop Bad data is worse than missing data. Missing data is worse than ugly data. Ugly data saves lives
- 9. #docsxdevop@wiredferret 3 Categories of DevOps Info What you need to keep Automatable Structural Secret
- 13. @wiredferret #docsxdevop Jira is complex, but not thought-provoking
- 16. Secret information risks What could happen if the data is lost? What could happen if the data is found by good actors? What happens if the data is found by bad actors?
- 17. @wiredferret #docsxdevop Who has that password?
- 21. @wiredferret #docsxdevop What do you have, and where is it?
- 22. @wiredferret #docsxdevop What are people using?
- 23. @wiredferret #docsxdevop Post-it notes run the world
- 24. Punchlist People responsible Workflow Support info Dependencies General information types
- 27. @wiredferret #docsxdevop Blue screen of death
- 30. @wiredferret #docsxdevop Data has an expiration date
- 31. @wiredferret #docsxdevop One day of Flickr photos
- 34. @wiredferret #docsxdevop Never call someone in the hospital
- 35. @wiredferret #docsxdevop Docs prevent disaster
- 36. #docsxdevop@wiredferret tl;rt Write it down, because that's easier and safer than remembering it.
- 37. @wiredferret #docsxdevop Fear of the Bus Documentation and Devops DevOpsDays Tel Aviv 2016 Heidi Waterhouse @wiredferret h.waterhouse@gmail.com heidiwaterhouse.com
Editor's Notes
- The title of this talk is derived from some black humor in some companies and IT departments. Would the company fold if any single person got hit by a bus? Sometimes it's less gruesome, and we say "won the lottery", but the principle is the same -- is there anyone in your organization who is literally impossible to replace? If so, I hope you have them insured well enough to dissolve the company gracefully.
- No matter how much someone loves a job or company, there are things that can happen that make them suddenly unavailable -- accidents, medical emergencies, natural disasters, even telecom breakdowns.
- Most operations and devops people are running flat-out to keep up with their work and don't feel that they have the time to talk to someone about their rapidly-changing docs or write the docs themselves.
- That means a lot of mission-critical information gets stored in people's memories or in their messy wiki pages. Essentially, the human is working as a file pointer. But that is volatile memory, RAM. But if something happens to disrupt your power you lose the pointers, and even if the data still exists, you can't find it and it's effectively gone.
- The way to insulate ourselves as organizations is to make sure that no one is irreplaceable. We have to have some way to pick up their tasks, capture their understanding, and keep going with our organization.
- Core principles Everyone can touch the docs Content matters more than presentation Docs reduce burnout and onboarding stress because everyone is empowered to get answers
- Roughly speaking, there are three categories of information we need to capture: Automatable Structural Secret
- Automate the things you can automate. If you build a new style of server, build it with a script and write down descriptions of things that change as part of the process of creation. Automatable data is both a safety net against losing information, and a way to streamline your workflow. Any time you are spending time repeating a process that could be automated, you are boring your team and allowing for the introduction of errors.
- Automation is a lot like teaching your children a chore. At first it's easier to do something yourself rather than automating or teaching it, but over time, your initial investment will pay off.
- Create a structure that encourages the creation of templates. That means that you have a predictable file structure, and a consistently updated set of templates to use. Templates are both a file organization system, and a way that we actually write information. It's much easier to fill out a form or a template than to accurately remember all the information you're supposed to include in something. A set of structures reduces the cognitive load of documenting something by quite a lot.
- For example, when we fill out a bug in a bug tracking system, it's much easier to check all the required and recommended fields than it is to free form remember all the things that we are supposed to remember to include. If I had to do that, I would forget something every time. A little-know secret of template use is that you need to store a copy of the template somewhere people can't find it, because no matter how you teach them not to do it, and lock the file, someone will inevitably use your master template to fill out instead of a copy. So do yourself a favor and hide a clean copy somewhere no one else will look for it.
- Structural information is also where we keep information about workflow. What is the series of steps that need to happen for something to be released? Have you made that into a punch list? If you need to terminate an employee, how many people have to be notified, and what do they need to do about it? That's not just about HR cutting their last paycheck, but about revoking physical and network access, activating post-employment requirements, and making sure everything is done correctly. If you don't store structural information in a place people can find it, they do ad hoc work and again, things get missed.
- Secret information is the hardest kind of thing to document, because you have to do threat modeling on it. If you're in an organization that does maturity models of security and has standing security policies, you probably have someone who handles this for you, but if you're with a smaller group, then you will need to do it yourself.
- What could happen if the data is lost? What could happen if the data is found by good actors? What happens if the data is found by bad actors?
- Secret information is the kind we are most likely to skip documenting because it seems risky, but also the kind that is most important to have exist somewhere outside a single person. For example, what will happen if you don't have a way to access your current website because the host username and password were configured by a marketing person who has left the company? What if it's the password to your Microsoft support account? You don't always want to publish this on an open internal wiki, but you do need to figure where to keep it so the right people can get to it.
- No matter what type of information you're documenting, it needs to be findable, usable, and current.
- Happily, because we are talking about the devops world, we don't have to worry about making anything pretty, fancy, or user-friendly. It's internal so it only needs to be functional.
- Collecting information doesn't just mean telling people they should write down what they're doing, it means making it easy to write things, almost easier than not writing them. It means removing all the obstacles and giving people an easy way to do the right thing. It's a cultural change.
- I suggest that you start with what I call a documentation audit. What do you already have and where is it?
- If you can, install some internal analytics. What pages are people accessing most? You'll want to be sure to address the content on those first.
- Collecting information doesn't have to be about writing a lot of new information, it means getting people to share their personal folders, desktop documents, and sometimes their post-it notes.
- Start the documentation audit by listing all your current documents in a spreadsheet. This doesn't have to be fancy. Make columns for the tasks that you expect your devops team to be able to support, like "Deploy to production" or "Respond to AWS outage". Decide what kind of documents you need for any task. I suggest: Make sure each of your tasks has all the required documents. You will find that there are some that are almost complete and some that are very sketchy. If a task requires Secret information, be sure you put a reference to that in the dependencies document.
- It doesn't do you any good to collect information if no one can find it. You're going to need to use structure, metadata, and internal training to make sure people can actually find the carefully collected documentation you're using.
- One of my favorite recommendations is to put everything in one big flat file. This works in organizations up to a certain size. The advantage is that you can just put new information at the top and push old information down without losing it. When you want to know something, you just search on it. This doesn't even have to be in a wiki or anything that fancy, as long as it's a single hosted file somewhere and not something people are using on their individual systems. If what you care about is making sure things get documented somewhere, use one big page. It's ugly, and it goes against every design principle, but who cares? It's findable, searchable, everyone can have a single bookmark, and no one has excuses. If you're worried about someone using old info, you can archive things after a year. If you have a devops procedure that has been unchanged for a year, I am frankly astonished. Bonus points if you have a hack afternoon every month or two to update, put in anchor tags and headings, and clean things up as a team
- Search is magic and amazing and I think over a decade of completely searchable email has changed how we think about data storage. That said, the reason search works for email is because presumably we read the message at some point and we know what keywords we are looking for. When more than one person is building a collection of documentation, you tend to have keyword fragmentation, just because everyone is coming from a different background. If someone is using the wrong keyword, they won't find the thing they need. For example, what would you call this screen? (BSOD). For years and years, if you search for these terms, you would not get any answers from the Microsoft site, because that's not what Microsoft thinks of that screen as. They do now, and if you search for it, it's not directly in the text of the help, but it is in the meta text so that you can find it and see what Microsoft calls it.
- 18F, a startup embedded in our General Services Administration, uses Slack in an interesting way to collect and find information. When someone says something in a slack message that should be retained, they attach a unique emoji that doesn't get used for anything else. Then you can search on that emoji and add whatever it is next to to the documentation you're trying to build. When I was talking about this at another conference, I found the guy who actually did it. His name is Mike Bland, and he even uploaded the hubot plugin to do it: https://github.com/mbland/hubot-slack-github-issues Every team has their own idiosyncratic ways of communicating. It doesn't matter. What does matter is figuring out what your team is doing and putting documents in their existing path.
- Information has a shelf life. It goes stale, or worse than stale, it gets rotten.
- If I could do it, we would tag all information with a date and when it exceeds a certain age, it gets tossed. But that's a problem for lots of reasons, so instead we have to set things up so that people can "smell" that the data is old and make a considered choice whether or not they want to consume it. If I say this at enough conferences, someone will build it for me. I want a wiki style sheet that gradually ages data by turning it lighter and lighter, until by a year old, it's white text on a white background. You could still read it if you highlighted it, but it would be obvious that this information was worn out and should either be updated or disposed of.
- I have another whole talk about how keeping data is expensive and increases our security threat surface, but I'm not going into that today. What I am going to say is that if someone, like a new hire, is looking at your internal documentation, you don't want them to be able to execute on one thing that was absolutely right two years ago and will now bring down servers, and all the people who are already there know not to use Current_Build_Script, because we all use New_Build_Script. Nothing says that, we just know it. If you don't use it, throw it away, or at least archive it so it doesn't reach out from the past like the hand of a zombie.
- All the hardest problems in computers are people Documentation reduces burnout and daily interruptions, and vastly decreases the cost of on boarding.
- How do you test that you're doing it right? Some of you familiar with the concept of the chaos monkey. The idea is that you select a person on the team, have them hand over their work phone, and take a day off. This is especially nice if they had a nasty on-call weekend. Who wouldn't enjoy a day off that was actually….off? Block them on slack and email, don't let anyone call their home phone. They're not here.
- Handle a set of stressful actions when the world is not actually on fire, when people aren't actually emotionally compromised. If you can't remember where Maria keeps the build files when she's out for the day, you sure as heck won't remember when she's in the hospital and you're waiting to hear if it all went ok.
- Devops docs save time, money, and having to post pictures of the Hindenberg where your uptime stats used to be.