Updating Software in Humans
•Download as PPTX, PDF•
2 likes•79 views
A talk about mental models, war stories and why computers are terrible.
Recommended
More Related Content
Similar to Updating Software in Humans
Similar to Updating Software in Humans (20)
Recently uploaded
Recently uploaded (20)
Updating Software in Humans
- 1. Updating software in humans
- 2. Computers are mostly terrible You have to play dumb to get anything done War stories are fun!
- 13. Computers are mostly terrible
- 14. A fun example - why do you need to suspend curiosity most of the time?
- 16. There are no guarantees when you save a file.
- 18. Computers are mostly terrible ✅ You have to play dumb to get anything done✅ War stories are cool
- 19. “A mental model is an explanation of someone’s thought process about how something works in the real world. It is a representation of the surrounding world, the relationships between its various parts and a person’s intuitive perception about his or her own acts and their consequences.”
- 20. Cool war story
- 21. That unfortunate time we destroyed all of our Elasticsearch clusters.
- 28. Our understanding and mental model of the extent of our automation, and related risks, was incomplete.
- 29. Back to the deployment? 😅
- 31. Some highlights from CI/CD
- 37. Computers are terrible ✅ You have to play dumb to get anything done ✅ War stories are cool ✅
- 38. Updating your mental model: Ask people stuff Documentation (lol) Read source code Chaos experiments
Editor's Notes
- Hello! It’s really cool to be here at such an amazing conference! I really love this room, though the table layout reminds me of a wedding. This is the weirdest wedding I’ve ever been at. This is a talk about mental models that plausibly sounds like it could be part of a software delivery conference! It covers some topics already mentioned in other talks, hopefully I won’t contradict them too much.
- First I’m going to spoil my own talk. Here’s what you will take away from this talk. If “computers are terrible” is a surprise, you are probably at the wrong conference, and you weren’t watching Laura’s talk just there.
- But before I cover those topics, does anybody remember this? Charity suggested on twitter that we should do some deployments on stage! So I’m also going to do a deployment and talk a little bit about why deployments are important to Intercom. Later on I’ll chat through some of the details of our deployment process.
- So I’m going to ship something to Intercom live on stage. This is a terrible idea, live demos can go badly and be very tedious to watch! The deployment will probably work. It will definitely eventually work, our deployment pipeline is generally robust, though it’s not extremely fast and is subject to the occasional wobble. There’s a chance it may not go through by the end of this talk especially if I get a bout of nerves and make a mistake! If it doesn’t work, I promise to keep coming back to this conference to talk every year until I do manage to ship successfully to Intercom during a talk.
- Here’s what I’m going to ship. It’s a simple redirect of app.intercom.io/THE SHIP EMOJI to shipitcon.com. You can test it now to see if it works. After I kick this off, I’ll explain a little about Intercom and why we care so much about shipping software.
- I work at Intercom. Our Dublin office is literally next door, I spend a reasonable amount of time every day looking out of our window onto the roof of this building. Intercom is an Irish software startup who helps online businesses talk to their customers, primarily using a messenger.
- Provide software as a service to our customers. Here’s our messenger. There’s gifs, emojis. There’s also a backend app which has a lot more going on but doesn’t work as well on giant screens!
- Here’s our messenger. There’s gifs, emojis. There’s also a backend app which has a lot more going on but doesn’t work as well on giant screens!
- The messenger space is pretty competitive, there’s a huge opportunity to create a lot of value, and so we have to move fast.
- In the R&D team at Intercom we use principles to drive our work. These allow us to teach, share what we’ve learned and scale how we think about building great product. They’re a bunch of lessons learned. They’re opinionated, and not simply A bunch of truisms. The principles distill how we think about building great product.
- Here are 3 of our principles we use in our R&D organisation. “Ship to learn” is a universal R&D principle. The sooner we ship, the quicker we learn how our product is used. Getting software into users hands lets you understand how its used quickly.
- “What you ship is what matters” is a design principle, used by our design team. Our designers care about what is actually built, delivered and usable by our users, not the artifacts created along the way. The process is important, but the output is the critical part. Sketch? Sigma.
- “Build in small steps” is a direct instruction to our engineers. Make small changes frequently. Break work down into safer, smaller steps. This doesn’t just refer to changes done via code deploys and pull requests, but allo usual modern “testing in production” techniques such As feature flags. In addition to being iterative and assistive for an agile development process, there are secondary benefits such as helping our availability, quality and again lets us understand what actually happens when we ship what we’re building. These aren’t universal truisms. It would be reckless of an infrastructure provider to globally or naively apply these. They also imply a bunch of support that you need to do to allow these to happen.
- Ok, so getting back on track. Computers are great! They can do things like tell the time, save files, do basic arithmetic, run automated tests, deploy software and talk to each other in large distributed systems just so that we can be all agile and stuff when we want to serve our Uber for cats startup. The thing is, computers are terrible at all of those things. Or rather, those things sound simple, and obvious but reality is quite difficult. And things get even worse when you have lots of computers attached to a network!
- Almost all of the time, what we are working with are things that we have imperfect, incomplete and at times mental models. Almost all of the time, this is ok. It is also utterly essential to get anything done that this remains the case. So let’s look at a very basic thing that everybody does with computers every day.
- Haha, what even is this thing? It’s a 3.5 inch floppy disk from the 90s! But it’s also used as the icon to save data everywhere, even though nobody saves anything to disk these days. There are a million tweets along these lines, this is not an original joke. Nobody really uses floppy disks any more, we get it. Detachable storage is generally greatly frowned upon, don’t we all use Dropbox etc.? But not only is this a terrible icon, reliably saving stuff to any type of disk is surprisingly hard.
- The problem is that there are no guarantees what happens when the program you’re writing calls the write() system call. The OS, filesystem, disks themselves have multiple intermediate layers that are crucial for a high performing system but make it hard to reason about where your data actually is.
- The “documentation” doesn’t help. Here’s an extract from “man mount” on a modern Linux system. The “rumour” is from around 2001, nearly 20 years old. There are some common strategies to the limitations of filesystems such as renaming files to achieve atomicity. I’ve done this a load of times in simple situations, but when it comes to actually being able to recover the data in all circumstances. But if you’re writing the parts of MySQL or PostgreSQL or a distributed system like S3 with serious data durability requirements, you need to dig deep here and understand the precise behaviour of the different parts that make up a disk (caches, platters etc.) as well as the drivers, filesystems. For the rest of us? It’s ok to think that when you click the floppy disk and save the file, that the file is saved. It works the vast majority of time and you have to pretend you don’t know what’s happening.
- So we’ve shown that computers are terrible and you have to ignore the reality to actually do anything.
- This is a good quote, lifted from @copyconstruct’s writeup about mental models. The mental model you need to have unless you’re a MySQL or kernel developer is “the computer saved my file”. I guess the trick is knowing when to go deeper. When things break, that’s a great time to go deeper!
- So here’s a war story from Intercom where we had to rebuild our understanding of something.
- Here’s a network diagram of our production setup in the cloud. We’re hosted entirely in Amazon Web Services in us-east-1 (North Virginia). There are different network subnets in our cloud. We use three. This is where our services live. I’m not really dumbing this down, we try to keep things very simple.
- A subnet has different bits of configuration, like the IP addresses it can use. It also has a routing table that tells the computers where to send the packets. This routing table is ALSO VERY SIMPLE - a small number of entries.
- Dumpster fire, everything’s fine dog.
- The same routing table is used across all subnets. We use TERRAFORM, an infrastructure as code tool, to manage a bunch of our AWS infrastructure including our network setup. routing tables etc. It allows us to define our network in code and it translates this to AWS API calls. We added NEW SUBNETS but the way we had configured TERRAFORM was that when it was adding subnets, it would REMOVE AND ADD BACK THE ROUTING TABLES WHEN A NEW SUBNET WAS ADDED. I’ll say that again. When we add a new subnet, the routing table was removed from all subnets and then recreated and added back to all the subnets.This made a conceptually simple change into a complex, dangerous change. You wouldn’t do that in the AWS UI The reason for this was that Terraform’s language is pretty basic and gives very few primitives to program with.
- By now you can probably guess where this is going, though trust me it gets better. We added some new subnets, but got the new IP ranges wrong. They were overlapping with existing ones. The routing table got DELETED by Terraform, it tried to create the new subnets then boom. And then it basically gave up after it couldn’t create the new subnets. Complete network outage of our production cloud environment in AWS for 14 minutes and 57 seconds. Engineers, mostly in SF, did amazing getting us back to a good state. This was on a Friday evening. I was actually in the pub for my 40th birthday watching all this unfold over Slack. It was very impressive to see our Incident Command and global shared on call kick in. Once we were in a major event mode, of course large amounts of engineers joined to help out. This was great to see, especially from the pub :D
- In general, automation is amazing, but sometimes it can really bite you in the ass.
- We use OpsWorks, which is basically hosted Chef, to manage our Elasticsearch clusters. Fully documented but not quite well understood feature of AWS’s Opsworks service. Because the hosts weren’t contactable, Opsworks decided to “autoheal” the hosts by moving them to new hardware. All The search data was stored on the nice fast local disks, gone. So basically opsworks auto healed the shit out of our elasticsearch clusters, leaving us with 10 wonderfully empty clusters.
- Built a list of known automation, examined cloud watch logs, looked for areas where errors could result in damage or there weren’t safeguards from destroying production infrastrucutre. “single bullet gun” This is incomplete - we weren’t formally proving what is our there, but an audit of what we can work with easily is a good start.
- So we’ve shown that computers are terrible and you have to ignore the reality to actually do anything. Since we’ve been looking at that deployment, and this is a conference about delivering software I’m going to show off some of the more interesting parts of our testing process.
- Sometimes we change something in our environment that is not backwards compatible. Like, we upgrade a database or something. This check gives us the ability to force developers to a minimum version so that we don’t annoy them. We’ll tell everybody to rebase. Happens infrequently enough, but a good way to avoid doing a lot of work at times. Also we work off of trunk!!!
- This is from our Docker file. We don’t use Docker in production, but it’s useful to share test artifacts. We use a vendored copy of a Docker base image from CircleCI (who we don’t use for Intercom) based off Ruby 2.5.5 (which we have long since migrated off of) to install a “stretch” Docker image (we don’t use Debian in production, we use Amazon Linux, which is CentOS based). Why do we vendor this? Because upstream changes have broken our deployment pipeline, so we’d rather control this. That’s a lesson learned!
- We use a vendored copy of a Docker base image from CircleCI (who we don’t use for Intercom) based off Ruby 2.5.5 (which we have long since migrated off of) to install a “stretch” Docker image (we don’t use Debian in production, we use Amazon Linux, which is CentOS based). Why do we vendor this? Because upstream changes have broken our deployment pipeline, so we’d rather control this. That’s a lesson learned!
- We use a vendored copy of a Docker base image from CircleCI (who we don’t use for Intercom) based off Ruby 2.5.5 (which we have long since migrated off of) to install a “stretch” Docker image (we don’t use Debian in production, we use Amazon Linux, which is CentOS based). Why do we vendor this? Because upstream changes have broken our deployment pipeline, so we’d rather control this. That’s a lesson learned!
- We use a vendored copy of a Docker base image from CircleCI (who we don’t use for Intercom) based off Ruby 2.5.5 (which we have long since migrated off of) to install a “stretch” Docker image (we don’t use Debian in production, we use Amazon Linux, which is CentOS based). Why do we vendor this? Because upstream changes have broken our deployment pipeline, so we’d rather control this. That’s a lesson learned!
- Here we just install a vendored version of MongoDB Enterprise server (we don’t use enterprise server in production) for some reason at the end of a shell chain that installs a vendored version of OpenSSL. Something to do with the move from jessie. “It works”
- Here’s libeatmydata
- Whirlworld tour through Docker Some Mongo confi, some installing MySQL from scratch (We run AWS’s RDS Aurora in production), some Redis stable (we use ElastiCache Redis in production) And some Ruby stuff! Next, we load the schema from a cache… no I mean the schema cache from a… cache…
- In our modern CI/CD environment, a “green” build has passed all its tests and is therefore safe to ship to production. A quick look under the covers shows that this is often far from the case, for example non-deterministic tests being covered up by retries. Also when we deploy our monolith to over a thousand servers, it doesn’t happen at once. At any moment there might be many software versions running, even on the same host. But for all but a bunch of narrow edge cases, this is totally fine. My point is here that we’ve got a reliable, battle hardened complex build and deploy system. To ship code at Intercom, you’re better off not knowing about any of this stuff! It’s another example of having to act dumb to get your job done. The abstraction here is critical.
- So we’ve shown that computers are terrible and you have to ignore the reality to actually do anything. Updating your knowledge of what’s going on is useful when you realise that what you’re working with doesn’t work the way you expect.
- Once you realise there’s value in updating your model of the world, what can you do about it? For example, my understanding of the main drivers of Intercom’s AWS cloud costs and how they are influenced by how we autoscale isn’t something that can be solved with writing some debugging statements . Analyse data, ingest it into analytical tools, build hypotheses and make decisions on the basis of what you now think you know.
- There are some great materials on a bunch of the topics I’ve covered here that go into a lot more detail and probably ten times more articulate. Copy Construct’s blog post that I already mentioned, Tanya Reilly’s “Nobody could have predicted this”, Dan Luu’s talk from Deconstruct which goes into a lot of detail about how reliably saving data is very hard. I’ll tweet out links and the slide deck after the talk!
- The backend looks like this. It looks similar enough to
- So unlike the proverbial frog in boiling water, we did notice some problems.
- There’s my twitter handle, I hope you enjoyed the talk. Thanks for listening!