The Science Of Troubleshooting
•Download as PPTX, PDF•
2 likes•838 views
From a talk given at Agile 2016 Can we use the Scientific Method to become better troubleshooters?
Recommended
More Related Content
Similar to The Science Of Troubleshooting
Similar to The Science Of Troubleshooting (20)
Recently uploaded
Recently uploaded (20)
The Science Of Troubleshooting
- 1. THE SCIENCE OF TROUBLESHOOTING Adding Rigour to Chaos Shawn Button
- 2. THE SCIENCE OF TROUBLESHOOTING Or “How We Actually Solve Problems, maybe without even realizing it” Shawn Button
- 3. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 4. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 5. CUSTOMARY “WHO AM I?” SLIDE Shawn Button • Developer • Agile Coach • Lever 21 Pokémon Go Trainer
- 6. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 7. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 8. STORY OF A TROUBLESHOOTING PROBLEM
- 10. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 11. To-Do DOING (1) DONE Story of a Team IntroTypes of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 12. TYPES OF REASONING • Deductive • Inductive • Abductive
- 13. DEDUCTIVE REASONING Starts with the assertion of a general rule and proceeds from there to a guaranteed specific conclusion. e.g. Math If x = 4 And if y = 1 Then 2x + y = 9 e.g. All birds have feathers and swans are birds, so swans have feathers.
- 14. INDUCTIVE REASONING Begins with observations and proceeds to a generalized conclusion that is likely, but not certain, in light of accumulated evidence. e.g. All swans that I have seen are white, therefore all swans are white.
- 15. ABDUCTIVE REASONING The generation of new ideas (hypotheses) to explain observations. Begins with an incomplete set of observations and proceeds to the likeliest possible explanation for the set.
- 16. ABDUCTIVE REASONING Examples: Medical diagnosis: given this set of symptoms, what is the diagnosis that would best explain most of them? Criminal trial: given the evidence is the suspect more likely innocent or guilty? Swans get wet when they swim. That swan is wet, therefore it was swimming, and therefore there must be water near here.
- 17. ABDUCTIVE REASONING Abductive reasoning is concerned with imaginative reasoning, a process where new ideas or hypotheses come into existence through observation. We use abductive reasoning all of the time in order to make sense of the world. We build up a mental model of reality that is constructed from hypotheses, which are based on observations.
- 18. TYPES OF LOGICAL REASONING Deductive Reasoning General Rule Specific Conclusion (true if rule is true) Inductive Reasoning Specific Observations General Conclusion (may be true) Abductive Reasoning Incomplete Observations Best Prediction (may be true)
- 19. ELEMENTARY, MY DEAR WATSON
- 21. TYPES OF LOGICAL REASONING Deductive Reasoning General Rule Specific Conclusion (true if rule is true) Inductive Reasoning Specific Observations General Conclusion (may be true) Abductive Reasoning Incomplete Observations Best Prediction (may be true)
- 22. To-Do DOING (1) DONE Story of a Team IntroTypes of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 23. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 24. BACK TO THE TEAM
- 25. WHAT DID WE DO?
- 26. WHAT WORKED?
- 27. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 28. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 30. ]
- 31. THE SCIENTIFIC METHOD OF TROUBLESHOOTING Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 32. OBSERVE PROBLEM Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 33. OBSERVE PROBLEM • A Problem is behavior that you didn’t expect • First thing to do it figure out how it should behave • Focus on external behavior. What are all of the things we should expect to see if it works. • Ask: What input does it take? What output should it give? Etc. • Write down the problem statement! If you can’t create a concise description you probably don’t understand it well enough. • Can we recreate the problem? If not we need to be able to in order to proceed.
- 34. COLLECT OBSERVATIONS Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 35. COLLECT OBSERVATIONS • Now collect some evidence. Write down any observations about what’s happening in the system. • What actually happens when you recreate the problem? What inputs did you use? What are the outputs. Are there log files? Can we see other impacts, for example in a database or service. • Spend some time digging here, because the more you spend the better the quality of the hypotheses you are able to create in the next step.
- 36. CREATE HYPOTHESES Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 37. CREATE HYPOTHESES • Think of as many possible causes for the observed behaviour as you can. • These are your hypotheses. A hypothesis is an attempt to understand and explain what is happening. • Hypotheses can be wrong, and many will be. • Write everything down! • If you can’t form clear hypotheses, you might just not know enough, and need to gather some more data. • There’s nothing wrong with using Stack Overflow / Google to help form hypothesis!
- 38. Ease to Perform Guess of Value WHICH HYPOTHESIS SHOULD WE TEST?
- 39. DESIGN AND CARRY OUT EXPERIMENTS Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 40. DESIGN AND CARRY OUT EXPERIMENTS • Before you start your experiment write down what you are doing, what you expect to see, and what it means if the experiment fails. • It could be a description of the metrics you’re going to look at, code or config you’re going to change, query you’re going to run • An experiment could also just be gathering data or measurements, for example from log files, metrics, other visibility tools you have. For example “if we look in the production logs we should see this log statement before the error” • One way of proving a hypothesis is by looking at the code, or stepping through a debugger. • Again, write it down! • E.g. If I change this line in the config file, then I expect that this error should no longer appear in the log.
- 41. EVALUATE RESULTS Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 42. EVALUATE RESULTS • What happened when you performed the experiment? • For complicated debugging it helps to keep a record of the results, for example save screenshots or copies of log files. • Did the results match your prediction? If not, your hypothesis must be false (unless you made some other mistake). • Do the results suggest a new hypothesis, or refinement to existing hypotheses? • If you changed something (e.g. code or config) and the results aren't what you expected, you should undo the change.
- 43. EVERY EXPERIMENT BUILDS UNDERSTANDING Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 44. EVERY EXPERIMENT BUILDS UNDERSTANDING • If you are able to solving a problem without learning, the problem is just going to reoccur later. • A validated hypothesis doesn’t necessarily mean you’ve fixed the problem. It might just be that you’ve learned something about the system. You might need to go through many experiments loops to learn enough the find the problem. • The scientific method is actually a structured knowledge acquisition process. Solving problem is a happy fringe benefit!
- 45. EVENTUALLY, A SOLUTION! Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 47. EXERCISE! As a table you are going to try out the scientific method. Pick a scenario from the back of the handout, or better, someone volunteer a real problem they have or had.
- 48. EXERCISE! As a table you are going to try out the scientific method. On the big sheet on the table make sections for: Problem Statement Observations Hypotheses
- 49. EXERCISE! As a table you are going to try out the scientific method. Together come up with each of these. For hypotheses come up with the experiment you will run, and how you will know if the hypothesis is validated.
- 50. EXERCISE! Observe Problem Collect Observations Create Hypotheses Design Experiment Perform Experiment Evaluate Results Falsified, so try another hypothesis Supported, so refine hypotheses Our Understanding Of The System Solution!
- 51. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 52. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 56. “First I had to become conscious while programming. I had been programming for years… and I was astonished to discover that, even though programming decisions came smoothly and quickly to me, I couldn’t explain why…. The first step… was slowing down long enough to become aware of what I was thinking, to stop pretending that I coded by instinct” WHY SO DELIBERATE WITH THE PROCESS?
- 57. “I’m not a great programmer. I’m just a good programmer with great habits” WHY SO DELIBERATE WITH THE PROCESS?
- 58. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 59. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 60. ]
- 61. • Your memory isn’t as good as you think it is! • Take notes in a notebook, whiteboard, stickies, or a chat tool • Provides clarity on the problem, hypotheses, experiments and learning • Avoids repeating experiments, or missing connections WRITE EVERYTHING DOWN!
- 62. The speed of running experiments is key. The faster you go through the experiment loop the faster you learn. EXPERIMENT FASTER
- 63. • Isolate the problem in a smaller app that you can run in isolation (Unit test tools like xUnit are fantastic for this) • Focus on faster experiments, for example ones that only use logs from previous runs • Tools help here! Become an expert in your debugger, or in some languages, interactive consoles EXPERIMENT FASTER
- 65. Often the problem is mired in a lot of complexity. Can you find a way to recreate the problem in a less complicated fashion? • Extract the offending code into it’s own method/class • Refactor to clean it up • Comment out extraneous lines • Once you run your experiment you likely want to revert these changes SIMPLIFY
- 66. Empathize with the future person who has to debug the problem (who might be you). • Design in the ability to run experiments. • Good, modular design lets you run things in isolation. Single Responsibility Principle! • Have unit tests. Use Test Driven Development. • Use informative logging statements. • Keep your logs clean. • Good error messages beat documentation. DESIGN FOR DEBUGGING
- 67. • Write Everything Down! • Running Experiments Faster • Simplify The System • Create Systems That Support Troubleshooting • Be Deliberate About The Process and Practice THINGS THAT MAKE THIS EASIER RECAP
- 68. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 69. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 70. ]
- 71. • If you skip steps or try to hurry this, you will immediately slow down. • If you are a manager protect your developers from pressure (and blame) when they are debugging a critical problem. • Have someone to communicate status to anxious management types and customers, to allow the devs time to by systematic • Take a break if you get stuck PRESSURE AND RUSHING SLOWS YOU DOWN
- 72. ]
- 73. If you change more than one thing at a time it is very hard to evaluate the results of your experiment. Take as small steps as you can in order to learn about the system. ONLY CHANGE ONE VARIABLE AT A TIME
- 74. You will unconsciously want to problem to have a certain cause, so pick or exclude valid hypotheses. • You have “gut feelings” or assumptions about what is wrong • There is a part of the system you distrust or you often see problems in a certain area • We have a tendency to think a problem is someone else’s fault BIAS, ASSUMPTIONS, BLAME
- 75. Before you blame someone else, run an experiment that conclusively proves it is someone else’s problem. Then provide them with the BLAME results of that experiment to help them debug the problem.
- 76. • Sometimes a problem can have more than one root cause. • Sometimes failures can interact with each other (or even cancel!) • Sometimes a tested supported hypothesis can lead you down the wrong path. COMPLEX FAILURES
- 77. • Don’t just randomly run experiments! • Hypotheses are created based on evidence. • New hypotheses should be guided and refined by your increasing knowledge of the system you are investigating. TRIAL AND ERROR
- 78. • Pressure and rushing slows you down • Changing Multiple Variables confounds you • Bias, Assumptions, Blame • Complex Failures • Trial and Error WATCH OUT FOR
- 79. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 80. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 81. If you are wanting to learn about a new system, the same technique applies! State your hypothesis about how the system works, and then run your experiment. You build up a mental model of the system through structured experiments. KNOWLEDGE ACQUISITION
- 82. Sometimes, it’s just easier to google the problem. Don’t think this replaces all of the 90 times a day you search Stack Overflow. STACK OVERFLOW AND GOOGLE
- 83. WHERE IS THE CREATIVITY? Scientific method of debugging is a creative process, as you create hypotheses to test. Creativity helps. Sometimes framing the right hypo takes creativity and ingenuity. More people helps. Thinking outside of the “box” Does this feel dry and mechanical? Remember Abductive reasoning is a creative process. Finding the right hypothesis and drawing connections between experiments takes creativity, ingenuity, experience. More people helps (Mobbing ftw!)
- 84. • Knowledge Acquisition • When Not To Use The Scientific Method • Where Is The Creativity? OTHER THINGS
- 85. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 86. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 87. PAIRING AND MOBBING The best way to teach this is to demonstrate it in action. When a problem happens get everyone in a room and walk through the process. Use a big white-board or sticky notes on a wall to record your notes.
- 88. TOYOTA KATA Scientific method of debugging is a creative process, as you create hypotheses to test. Creativity helps. Sometimes framing the right hypo takes creativity and ingenuity. More people helps. Thinking outside of the “box”
- 89. TROUBLESHOOTING KATA QUESTIONS Scientific method of debugging is a creative process, as you create hypotheses to test. Creativity helps. Sometimes framing the right hypo takes creativity and ingenuity. More people helps. Thinking outside of the “box” Questions About The Problem 1. What do we know about the problem? 2. Is this enough to form hypotheses? 3. What are our hypotheses? Are we missing any? 4. Which hypothesis is most likely to advance us towards a solution? Questions For The Experiment 1. What experiment can we run to test our hypothesis? 2. What do we expect to see? 3. What were the results? Did we see what we expected? 4. What have we learned about the system?
- 90. TEACHING THIS Scientific method of debugging is a creative process, as you create hypotheses to test. Creativity helps. Sometimes framing the right hypo takes creativity and ingenuity. More people helps. Thinking outside of the “box” • Pairing and Mobbing is an excellent way to share this process and discipline • Use the Troubleshooting Kata questions to start people thinking in a scientific way
- 91. OBLIGATORY XKCD
- 92. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 93. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 97. BONUS! EXPERIMENTAL MINDSET Once you get into the habit of thinking about things in terms of experiments you will find it is applicable everywhere. Being more explicit about experimentally testing what you learn and believe makes you a more powerful thinker.
- 98. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 99. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 100. USEFUL REFERENCES “Debugging With Scientific Method” - @stuarthalloway talk at Clojure/conj - https://www.youtube.com/watch?v=FihU5JxmnBg “The Scientific Method of Troubleshooting: A FutureTalk with Blithe Rocher” - https://blog.newrelic.com/2015/08/19/futuretalk-scientific-method-blithe-rocher/ “Scientific Debugging” - http://yellerapp.com/posts/2014-08-11-scientific-debugging.html Game Coding Complete - Mike McShaffry “Scientific debugging: Finding out why your code is buggy” - http://www.embedded.com/design/debug-and-optimization/4418635/Scientific-debugging-- Finding-out-why-your-code-is-buggy---Part-1 “Each necessary, but only jointly sufficient” - http://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ Implementation Patterns – Kent Beck
- 101. CREDITS “Pair Programming” - Lisamarie Babik - https://commons.wikimedia.org/wiki/File:Pair_programming_1.jpg “Sherlock” - http://today.uconn.edu/wp-content/uploads/2014/07/sherlock-holmes-basil- rathbone.jpg ”Life Is A Mastermind Game” - https://adsoftheworld.com/taxonomy/brand/mastermind The Four Stages of Competence - https://focuspocusnow.files.wordpress.com/2013/02/4- stages-of-competence.png ”Lab Puppy” - https://pixabay.com/en/puppy-labrador-purebred-retriever-1082141/ “Springfield Blame” - https://c2.staticflickr.com/8/7473/15650030866_f236377785.jpg
- 102. FIND US HERE! Shawn Button shawn@leanintuit.com @shawnbut
- 103. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References
- 104. To-Do DOING (1) DONE Story of a Team Intro Types of Reasoning Back to the Team Scientific Method Deliberate Process These Make It Easier Watch Out For These Other Uses For Experiments Teaching This Bonus Material! Closing and References Thanks for Coming! Catch ‘Em All!
Editor's Notes
- - team asked to start working on an existing system. - Java, only a few years old, but already becoming legacy. big enterprise framework, no tests, little documentation - team didn’t get support - different location; different management hierarchy with different priorities.- - trying to build, deploy, run - strong deadline and pressure - 3 very smart developers spent 1.5 days, and they just couldn’t get it running. we mobbed, found the problem, and got it running in 45 minutes
- this started me on a path. Why were we successful when they had failed before? We weren’t successful just because we got in a room. Collaboration isn’t a silver bullet. I didn’t have some knowledge of the system or expertise they team didn’t have I think we were successful because of the process we used. I think sometimes people think good troubleshooting is innate, or due to expertise. But is effective debugging also something we could teach? Is it possible that I could get better at it if I was more explicit about the process
- Please bare with me, as this part is a little technical, but I think it’s important background.
- Effectively, it is a process of choosing the hypothesis, which would best explain the available evidence. Think of abduction as creating and evaluating competing hypotheses. Whichever hypothesis is stronger, for whatever reason, wins the abduction.
- Sherlock actually used inductive or abductive reasoning. I don’t know why Connan-Doyle used the term “deduction” I think maybe because it sounds silly to say you are “inducing” or “abducting” something? Pause for riotous laughter.
- Okay, back to the team They had been given a code repository, and were able to build It’s a Tomcat app. big-enterprise java, complicated configuration of the app required, mysterious imported enterprise framework. There were four configuration files in the repo. application wouldn’t start up with them. error was thrown on the console. asked the team that was “supporting” them they were emailed different versions of configuration files. different error was shown on the console team was randomly trying different combinations of the configuration files. They’d drop in a new file and try to restart it. Rinse and repeat. Each restart of app took 10 minutes.
- first thing we did was slow down reverted to the original set of configuration files We looked at the error on the console and logs. Came up with hypotheses about what the possible causes of the errors could be We then came up with a list of things we could try in order to test these hypotheses. Experiments. We then picked one and tried it. When we ran the experiment we learned something about the system. Either way, pass or fail. Sometimes we’d have no impact on the error, introduce a new error, and then that experiment didn’t work. We’d undo the changes and try another hypothesis.
- Eventually we discovered that the configuration files that had two lines that were causing problems. We found that we needed to take property values from different files to make it work. There were two problems, which is part of the reason why the team couldn’t find the problem. In their experiments they’d change a bunch of things, fix the one error but introduce a different error. They were running such large and unfocused experiments that they didn’t learn anything.
- First, a There’s a lot of philosophy of science, and it’s fascinating stuff. I highly advise you to read all of it. But for the purposes of this presentation we are going to skip Kuhn’s The Structure of Scientific Revolutions Regardless the scientific method has served us pretty well for the last couple of hundred years, so let'
- explain game This can be solved by using the scientific. The problem is the hidden pegs. Your hypothesis is a guess a the colours. You experiment by placing the colours and receiving the white and black pegs as the result of your experiment. You narrow down to the correct solution by running experiments I’ll come back to mastermind
- Explain overview
- Slide: Observe failure behavior that you didn’t expect. In this case the team was unable to start up tomcat First thing to do it figure out how it should behave. people miss that step, especially if this problem is coming from a bug report. Focus on external behavior. What are all of the things we should expect to see if it works. Ask: What input does it take? What output should it give? And so on. Write down the problem statement. If you can’t create a concise description you probably don’t understand it well enough. Take time with your original problem description. Try and be as precise as you can. If you mess up the problem definition, it’s easy to get stuck, or confused. Can we recreate the problem? If not we need to be able to in order to proceed.
- Slide: Observe failure behavior that you didn’t expect. In this case the team was unable to start up tomcat First thing to do it figure out how it should behave. people miss that step, especially if this problem is coming from a bug report. Focus on external behavior. What are all of the things we should expect to see if it works. Ask: What input does it take? What output should it give? And so on. Write down the problem statement. If you can’t create a concise description you probably don’t understand it well enough. Take time with your original problem description. Try and be as precise as you can. If you mess up the problem definition, it’s easy to get stuck, or confused. Can we recreate the problem? If not we need to be able to in order to proceed.
- Write down any observations about it. Now collect some evidence. What actually happens. What inputs did you use? What are the outputs. Are there log files? Can we see other impacts, for example in a database or service. Spend some time digging here, because the more you spend the better the quality of your next step which is
- Think of as many possible causes for the observed behaviour as you can. These are your hypotheses. A hypothesis is an attempt to understand and explain what is happening. Hypotheses can be wrong, and many will be. Write everything down! If you can’t form clear hypotheses, you might just not know enough, and need to gather some more data. There’s nothing wrong with using StackOverflow/Google to help form hypothesis!
- Now select the hypothesis that you want to test. This doesn’t have to be the most likely, it might just be the easiest to test. You’re looking for hypotheses that you think will give you good data, or perhaps just easy ones to run. Pick experiments which are designed to to give you as much information as possible, for the lowest effort. Experienced developers have an expert, intuitive sense for problems. They’ve done it so much that they eliminate unlikely problems without even knowing why. Like a chess master only looking at the best choices. (could use quote about difference between master and expert in chess, that they actually look at less options).
- Before you start your experiment write down what you are doing, what you expect to see, and what it means if the experiment fails. It could be a description of the metrics you’re going to look at, code or config you’re going to change, query you’re going to run An experiment could also just be gathering data or measurements, for example from log files, metrics, other visibility tools you have. For example “if we look in the production logs we should see this log statement before the error” One way of proving a hypothesis is by looking at the code, or stepping through a debugger. Again, write it down! E.g. If I change this line in the config file, then I expect that this error should no longer appear in the log.
- What happened when you performed the experiment? For complicated debugging it helps to keep a record of the results, for example save screenshots or copies of log files. Did the results match your prediction? If not, your hypothesis must be false (unless you made some other mistake). Do the results suggest a new hypothesis, or refinement to existing hypotheses? A validated hypothesis doesn’t necessarily mean you’ve fixed the problem. It might just be that you’ve learned something about the system. If you changed something (e.g. code or config) and the results aren't what you expected, you should undo the change.
- If you are able to solving a problem without learning, the problem is just going to reoccur later. A validated hypothesis doesn’t necessarily mean you’ve fixed the problem. It might just be that you’ve learned something about the system. You might need to go through many experiments loops to learn enough the find the problem. The scientific method is actually a structured knowledge acquisition process. Solving problem is a happy fringe benefit.
- If you are able to solving a problem without learning, the problem is just going to reoccur later. A validated hypothesis doesn’t necessarily mean you’ve fixed the problem. It might just be that you’ve learned something about the system. You might need to go through many experiments loops to learn enough the find the problem. The scientific method is actually a structured knowledge acquisition process. Solving problem is a happy fringe benefit.
- Finally, happy day! You’ve learned enough about the system to fix the problem!
- Snowboard story Switched from skiing to snowboarding about 10 year ago. After about 5 years it was the beginning of winter and as I put on my board at the top of the hill I realized I had no idea how to snowboard. My body did, which way to lean, how to react, but intellectually I had no idea how to snowboard. I decided to become conscious of what I was doing. Pay attention to how I did it. Once I did that I could actually begin to improve! I could experiment with different techniques, deliberatly try to get better. I quickly improved. Still not good, but better.
- When you do something for a while you become competent, but perhaps unconsciously competent I’m not fond of this model because I think there is a fifth stage, where by becoming more conscious you can increase your competence You do this by paying attention to what you are doing, in order to conscious improve. Dare I say, experiment with different ways of doing things?
- Another way is because Teaching a class, where you have to learn when you teach I know more french grammar than english
- So, I would say “I’m not a great troubleshooter, I’m just a good troubleshooter with great habits”
- Okay, lets come back to MasterMind for a minute. Imagine that you couldn’t see the clues for previous experiments. How much harder would this be? It would be a huge amount of hubris to think we could remember all of the results of the experiments, but that’s what we do in programming all of the time. We run many experiments, but don’t record the results! Write your experiments down! Follow the process, and be deliberate at every step. Btw, I borrowed this metaphor from a talk stuart halloway gave at Clojure/conj
- bugs like this require attention to your process - just wildly hacking away will lead you to wandering in circles, confused about the state of the problem, and the system Write everything down. If you are doing this on your own keep a lab notebook. If you are working with others or a team write on a whiteboard or sticky notes or Slack Write down because: you avoid repeating a hypothesis that you’ve already discarded (or should not even consider), because you forgot that you tested it make debugging harder because your “fixes” cause new problems
- Compare to speed of build-measure-learn loop in lean startup Speed of running experiments is key. If it takes many many minutes (or days) to run an experiment then you will be tempted to take big steps Sometimes you can only test your hypotheses by going through a complicated or long-lived procedure, for example starting up WebLogic. Sometimes you can only test your hypotheses by promoting to production! Isolate the problem in a smaller app that you can run in isolation (Unit test tools like xUnit are fantastic for this) Focus on faster experiments, for example ones that only use logs from previous runs Tools help here! Become an expert in your debugger, or in some languages, interactive consoles
- In my experience you can almost always figure out how to simplify and speed up the experiment, it just takes creativity and effort. It usually pays off
- For example it might suck to work in Java, but if you have to then take advantage of the very mature infrastructure! I am surprised when I work with devs and they don’t know of the cools things their interactive developer can do. Using these tools you can quickly gather data, or run many experiments in seconds, rather than minutes.
- Might be same as fast Isolating the problem Refactoring code to simplify so that we could see what it was doing….
- The team I mentioned before had this problem. They were under pressure, so they hurried, so they changed many things at once to make it “faster”. If you skip steps or try to hurry this you will immediately slow down. If you are a manager protect your developers from pressure (and blame) when they are debugging a critical problem. Assign someone to communicate status to anxious management types and customers, to allow the devs time to by systematic
- Because of the way that the game works we tend to change multiple things between experiments. We always run multi-variate experiments, which makes the results of the experiments in mastermind difficult to interpret. You need to combine the results of multiple experiments in order to figure out what gave us the black or white peg. Avoid this at all costs!
- Example was the team I mentioned before. They were changing multiple things at the same time, so while they fixed one error they introduced another
- The most important thing when you are starting a problem is to forgot what you already know! Often we will unconsciously think “I really want it to be this cause” . Your understanding of the system is incomplete, and your guesses as to what the issue is can confound your experimental results. The only real way to guard against this is to watch for it, or if you can, get somebody else to repeat your experiments. In particular we have a tendency to blame others for our failure. It’s good form to never send a bug to someone else unless you have evidence that it is their problem. Blame tools, frameworks, other teams Your original problem state will have some assumptions in it. Sometimes these will turn out to be just plain false. So some of your experiments should investigate those assumptions and check that they’re correct. Assumptions – stuck on idea
- Complex Failures Complex systems fail in complex ways. Often the failures interact with each other. Assuming that there’s a single root cause is an easy route to misdiagnosis. Instead look for combinations of failures that together explain the issue at hand. Sometimes more than 1 root cause means that you need to learn more about the system by performing experiments to learn how it works. A tested supported hypothesis MIGHT be a dead-end. Two failures might cancel each other out Dangers of reductivism. “5 whys”for example, could hide cases where there are multiple root causes, each of which is contributing, necessary but not sufficient.
- Avoid “trial and error programming” balance scientific method with investigation/leaning so you can form good hypotheses Hypotheses and experiments should contribute to your understanding of the system, which informs your hypotheses! Unfortunately, a lot of debugging uses this approach today. Turn off the machine. Reboot. Try rearranging the order of your statements,
- There are things you need to watch out for.
- Knowledge Aquisition Using the scientific method for learning about a system is the best way to do it. For example learning a language/api by writing tests against it. Structured application of the scientific method.
- I think sometimes people think scientific process is dry and lacks creativity. It’s a mechanical process. Quite the contrary, this is a process that requires discipline and rigour, but also ingenuity and creativity. The real problem can be very different than the appearance. For example an outOfMemory error that appeared as something else, or an error due to an incorrect version of a library. It can take creativity. That’s the imaginative reasoning part of Abductive Reasoning.
- Knowledge Aquisition Using the scientific method for learning about a system is the best way to do it. For example learning a language/api by writing tests against it. Structured application of the scientific method.
- Mobbing has entered my toolset, and is now my default approach when there are lots of unknowns, lack of knowledge, a knowledge gap, divergence between team members, or high criticality.