Top 5 Reasons Why Improvement Efforts Fail
•
1 like•604 views
This is my story of lessons learned on why our improvement efforts fail... I had a great team. We were disciplined about best practices and spent tons of time on improvements. Then I watched my team slam into a brick wall. We brought down a fully-ramped semiconductor factory three times in a row, then couldn't ship again for a year. Despite our best efforts with CI, unit testing, design reviews, and code reviews, we lost our ability to understand the system. I discovered our mistakes weren't caused by technical debt. Most of the problems were caused by human factors. We failed to improve because we didn't solve the right problems. To learn, we need a feedback loop. To improve, we need a feedback loop with a goal. There's five different ways our project feedback loop can break: * **Broken Target** - Our definition of "better" is broken. * **Broken Visibility** - We don't see the pain, so we take no action. * **Broken Clarity** - We don't understand what's causing the pain. * **Broken Awareness** - We don't know how to avoid the pain. * **Broken Focus** - We see the pain, but our attention is focused on something else. Find out how to repair the broken feedback loops on your software project.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to Top 5 Reasons Why Improvement Efforts Fail
Similar to Top 5 Reasons Why Improvement Efforts Fail (20)
Recently uploaded
Recently uploaded (20)
Top 5 Reasons Why Improvement Efforts Fail
- 1. Janelle Klein openmastery.org @janellekz Improvement Efforts FAIL Top 5 Reasons Why
- 2. , Developer, Consultant, CTO @ Specialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (data geek) Continuous Delivery infrastructure, automation strategy & technical mentorship Janelle Klein Who Am I? How to Measure the PAIN in Software Development Janelle Klein Author of “Idea Flow” leanpub.com/ideaflow Founder of newiron.com
- 3. What is this talk about? Mistakes
- 4. We were trying to do all the “right” things. About 8 Years Ago…
- 5. We were building a factory automation system...
- 6. We shipped to production...
- 7. We shipped to production... (again)
- 8. We couldn’t reproduce the problem.
- 9. We shipped to production... (AGAIN) The rollback failed!
- 10. Project FAILURE
- 11. The Situation 10 year old software project, 1.5M LOC, 24/7 uptime, programmable statistical processing engine Brought down production three releases in a row 12 developers on the team, disciplined with best practices, constantly working on improvements
- 12. The Retrospective “What are we going to do?!” Our biggest problem “Well, we know we’ve got a quality problem right?”
- 13. The Retrospective “What are we going to do?!” Our biggest problem “The problem is we don’t have enough test automation!”
- 14. So the Test Automation Began… Our regression testing took 3-4 weeks… Let’s automate the tests!
- 15. 20% Percent Automated: So the Test Automation Began…
- 16. 50% Maintenance started getting PAINFUL… Let’s refactor the tests! Percent Automated: So the Test Automation Began…
- 17. 80% It was still really PAINFUL… “Well, at least our regression cycle is faster, right?” Our regression cycle still took 3-4 weeks! Percent Automated: So the Test Automation Began…
- 18. ApplicationTests We noticed a Pattern in Our Test Suite
- 19. ApplicationTests We noticed a Pattern in Our Test Suite Most of the bugs were found during exploratory testing.
- 20. The First Mistake Our biggest problem “Well, we know we’ve got a quality problem right?” “The problem is we don’t have enough test automation!” What’s the mistake we made?
- 21. What does “better” really mean? What qualifies as an “improvement”?
- 22. “Better” following best practices (solution-focused) Start with the Solution (problem-focused) “Better” solving the problems Start with the Problem What does “better” really mean?
- 23. The Retrospective Our biggest problem “The problem is we don’t have enough test automation!” Solution-Focused “Problems”
- 24. “What problem am I trying to solve?” Mistake #1: Starting with the Solution
- 25. ApplicationTests Most of the bugs were found during exploratory testing. Our Test Suite
- 26. The Retrospective “What are we supposed to do?!” Our biggest problem “The problem is all the technical debt is causing us to make mistakes.”
- 27. The Retrospective “What are we supposed to do?!” Our biggest problem “Let’s brainstorm a list of all the problems!”
- 28. Brainstorm a List Filled up Tech-Debt Backlog
- 29. The Retrospective “What are we supposed to do?!” Our biggest problem “Well, if we don’t understand a problem, we should collect data.”
- 30. The Retrospective Our biggest problem “What data would help us understand the problem?” “What are we supposed to do?!”
- 31. Technical Debt Mistakes I thought the problem was Technical Debt
- 32. ? Most of our mistakes were in the most well-written parts of the code. Mistakes
- 33. We made significantly more mistakes in code that we didn’t write ourselves. Lower Familiarity More Mistakes= There had to be more to the story...
- 34. Complex( So*ware( PAIN This is what I knew... What made development feel painful?
- 35. Unexpected Behavior Problem Resolved Tracking Painful Interaction with the Code (Friction) Troubleshooting Progress 5 hours and 18 minutes of troubleshooting... PAINFUL
- 36. The amount of PAIN was caused by… Likeliness(of(( Unexpected( Behavior( Cost(to(Troubleshoot(and(Repair( High(Frequency( Low(Impact( Low(Frequency( Low(Impact( Low(Frequency( High(Impact( PAIN(
- 37. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger Most of the pain was caused by human factors. What causes PAIN?
- 38. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Most of the pain was caused by human factors. Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm
- 39. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Most of the pain was caused by human factors.
- 40. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? PAIN is a consequence of how we interact with the code. Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm
- 41. PAIN occurs during the process of understanding and extending the software Complex( So*ware( PAIN Not the Code. Optimize “Idea Flow”
- 42. ApplicationTests Most of the bugs were found during exploratory testing. Our Test Suite
- 43. ApplicationTests The bugs weren’t in the code, the bugs were in our Conceptual Models Our Test Suite
- 44. The Fix: We built a Data Dictionary The bugs weren’t in the code, the bugs were in our Conceptual Models
- 45. The “Coat of Armor” Principle Software Coat of Armor More tests result in better bug protection It’s not that simple!
- 46. “Alarm Pain” Six hours of troubleshooting and repairing tests when nothing is broken.
- 47. “Green Washing” — Kerry Kimbrough
- 48. “Green Washing” — Kerry Kimbrough If we don’t understand the system, we fix the tests incorrectly.
- 49. The “Waxy Coating” Principle Optimize the signal to noise ratio. Software (before) Software (after) Tests are like a waxy coating poured over the code.
- 50. Cost & Risk are a Function of Increased Difficulty Cost & Risk Difficulty of Work The Difficulty of Doing Our Jobs Human Limitations
- 51. Likelihood)of)) Unexpected) Behavior) Cost)to)Troubleshoot)and)Repair) High)Frequency) Low)Impact) Low)Frequency) Low)Impact) Low)Frequency) High)Impact) PAIN) “Better” = Reduce the Risk Reduce Likeliness = Mistake Proofing Reduce Recovery Cost = Mistake Tolerance
- 52. The Second Mistake Our biggest problem “The problem is all the technical debt that’s causing us to make mistakes.” What’s the mistake we made?
- 53. Complex( So*ware( PAIN Mistake #2: Assuming the problems are inside the code. What pain are we experiencing when we interact with the code?
- 54. Our biggest problem “Let’s brainstorm a list of all the problems!” The Third Mistake What’s the mistake we made?
- 55. “What’s the best opportunity for improvement? “The awful email template engine code!” Our biggest problem The Retrospective
- 56. “Fill in missing unit tests!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
- 57. “We should clean up the database code!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
- 58. “Let’s improve maintainability of our test framework!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
- 59. Just because a problem comes to mind, doesn’t mean it’s an important problem to solve. Our biggest problem The Retrospective “What’s the best opportunity for improvement?
- 60. Our biggest problem What do I feel the most intensely about? Daniel Kahneman Thinking Fast and Slow The Retrospective “What’s the best opportunity for improvement?
- 61. “The awful email template engine code!” Recency Bias Our biggest problem The Retrospective “What’s the best opportunity for improvement?
- 62. Guilt Bias “Fill in missing unit tests!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
- 63. Known Solution Bias Our biggest problem The Retrospective “What’s the best opportunity for improvement? “We should clean up the database code!”
- 64. Sunk Cost Bias “Let’s improve maintainability of our test framework!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
- 65. Mistake #3: Assuming the biggest problems will come to mind. How do I know the right problems are on the list? What do I feel the most intensely about? Daniel Kahneman Thinking Fast and Slow
- 66. The Deadline Effect Rushing before the deadline increases risk.
- 67. The Retrospective “What are we supposed to do?!” Our biggest problem “We should stop rushing before the deadline.” “Yes. Fixed deadline. Variable scope.”
- 68. Urgency Leads to High-Risk Decisions 7:01 Iterative Validation with Unit Tests 7:010:00 14:230:00 Skipping Tests and Validating at the End We gamble to save time: If I make no mistakes I save ~2 hours. If I make several mistakes I lose ~8 hours.
- 69. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits Problems
- 70. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits Problems
- 71. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits Problems
- 72. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits Problems
- 73. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits Problems
- 74. Likelihood)of)) Unexpected) Behavior) Cost)to)Troubleshoot)and)Repair) High)Frequency) Low)Impact) Low)Frequency) Low)Impact) Low)Frequency) High)Impact) PAIN) PAIN = Frequent Severe Explosions
- 75. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
- 76. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
- 77. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
- 78. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
- 79. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
- 80. Product( Requirements( So1ware(Task( Possible(Decision(Paths( So1ware( Software is a Reflection of our Decisions
- 81. Product( Requirements( So1ware(Task( High%Risk% Decision%Habits% So1ware( (and(all(our(problems)( ( RESET Software is a Reflection of our Decisions
- 82. Product( Requirements( So1ware(Task( High%Risk% Decision%Habits% So1ware( (and(all(our(problems)( ( Technical Debt is just a Symptom of the Problem Software is a Reflection of our Decisions
- 83. The Deadline Effect Rushing before the deadline increases risk.
- 85. We deleted our automated test suite… The automation wasn’t solving the problems!
- 86. What are the specific risks in this release? How can we mitigate the risks? Handful of manual tests Custom Tools +
- 87. Release Size (weeks) Time Defect Rates RISK-Bound Releases Continuous Delivery is about learning how to effectively manage risk
- 88. We built TONS of automation. The only real difference: We didn’t design a single solution without a specific problem in mind.
- 89. Our biggest problem “We should stop rushing before the deadline.” “Yes. Fixed deadline. Variable scope.” The Forth Mistake What’s the mistake we made?
- 90. Mistake #4: Over-simplification and hand-waving. How are you adapting your decision habits? Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety'
- 91. Then I got into consulting… The Software Rewrite Cycle Start% Over% Unmaintainable% So0ware%
- 92. We Start with the Best of Intentions High Quality Code Low Technical Debt Easy to Maintain Good Code Coverage
- 94. PAIN The Classic Story of Project Failure Problems get deferred Builds start breaking Releases get chaotic Productivity slows to a crawl Developers begging for time It’s never enough Project Meltdown
- 95. Our “Solution” “We should just quit our jobs.” “Yeah, it’s hopeless.” What are we supposed to do?
- 96. What if we could measure our PAIN? 1. Test Data Generation 2. Merging Problems 3. Repairing Tests 1000 hours/month The Biggest Problem: ~700 hours/month generating test data
- 97. “Better” “Better” What if we could get managers and developers all pulling the same direction? Managers Developers
- 98. The Biggest Cause of FAILURE in our Industry: Next Talk: Stop Getting Crushed By Business Pressure
- 99. Mistake #5: Feeling helpless instead of taking responsibility. Leadership is not a title, it’s a choice.
- 100. Summary
- 101. The First Mistake Our biggest problem “Well, we know we’ve got a quality problem right?” “The problem is we don’t have enough test automation!” What’s the mistake we made?
- 102. “What problem am I trying to solve?” Mistake #1: Starting with the Solution
- 103. The Second Mistake Our biggest problem “The problem is all the technical debt that’s causing us to make mistakes.” What’s the mistake we made?
- 104. Complex( So*ware( PAIN Mistake #2: Assuming the problems are inside the code. What pain are we experiencing when we interact with the code?
- 105. Our biggest problem “Let’s brainstorm a list of all the problems!” The Third Mistake What’s the mistake we made?
- 106. Mistake #3: Assuming the biggest problems will come to mind. How do I know the right problems are on the list? What do I feel the most intensely about? Daniel Kahneman Thinking Fast and Slow
- 107. Our biggest problem “We should stop rushing before the deadline.” “Yes. Fixed deadline. Variable scope.” The Forth Mistake What’s the mistake we made?
- 108. Mistake #4: Over-simplification and hand-waving. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' How are you adapting your decision habits?
- 109. The Fifth Mistake “We should just quit our jobs.” “Yeah, it’s hopeless.” What are we supposed to do?
- 110. Mistake #5: Feeling helpless instead of taking responsibility. Leadership is not a title, it’s a choice.
- 111. So what should we do instead?
- 112. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow Improve Quality of Decisions Data-Driven Software Mastery
- 113. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow Target - The direction of “better” Target: Optimize the Rate of Idea Flow Data-Driven Software Mastery
- 114. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow Input - The constraints that limit our short-term choices… Data-Driven Software Mastery
- 115. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow Output - The pain signal we’re trying to improve Data-Driven Software Mastery
- 116. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow Focus on the biggest pain… F ocus! Data-Driven Software Mastery
- 117. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow 1. Visibility - Identify the specific patterns 1. Visibility Data-Driven Software Mastery
- 118. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow 2. Clarity - Understand cause and effect 2. Clarity Data-Driven Software Mastery
- 119. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow 3. Awareness 3. Awareness - Stop and think to adjust habits Data-Driven Software Mastery
- 120. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow 4. Run Experiments to Learn What Works Data-Driven Software Mastery
- 121. If any part of the feedback loop is broken:
- 122. My team spent tons of time working on improvements that didn’t make much difference. We had tons of automation, but the automation didn’t catch our bugs.
- 123. My team spent tons of time working on improvements that didn’t make much difference. We had well-modularized code, but it was still extremely time-consuming to troubleshoot defects.
- 124. The hard part isn’t solving the problems it’s identifying the right problems to solve. “What are the specific problems that are causing the team’s pain?”
- 125. Retrospective: “Mastery Circle” Circle Leader Circle Member Focus: What’s the problem to solve? Observe: Ask questions about the facts Conclude: Breakdown causes into patterns Optimize: Discuss strategies for improvement Learn: Run experiments to learn what works Observation Questions Make a F.O.C.O.L. Point!
- 126. What’s Next?
- 127. Emergent Practice Everything I’m showing today:
- 128. Stop Getting Crushed By Business Pressure Next Talk:
- 129. Talk #3: Let’s Make the PAIN Visible! #OpenDX An Open Standard for Measuring PAIN (Specification for Data Collection) Developer Experience
- 130. Talk #4: Learn Your Way to AWESOME. Input: Decision Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. Visibility 2. Clarity 3. Awareness F ocus! Output: “Friction” in Idea Flow
- 131. LEARN YOUR WAY TO AWESOME. Free to Join Industry Peer Mentorship Network openmastery.org
- 132. Industry Peer Mentorship Network Companies Community Groups HQ in Austin Open Mastery Austin meetup.com/Open-Mastery-Austin
- 133. Janelle Klein openmastery.org @janellekz Check out openmastery.org for details. Read my Book. Think About It. FREE with MembershipBuy It How to Measure the PAIN in Software Development Janelle Klein
Editor's Notes
- The really awesome thing about doing technical assessment and mentorship for a living, is for the last 5 years, I’ve been working on a research project on codifying the art of software development into a teachable craft
- [Read]
- We were trying to do all the right things. We had CI, unit testing, design reviews, code reviews. All that stuff you’re supposed to do.
- We were building this factory automation system that was responsible for detecting manufacturing problems then shutting down the tool responsible. I’d been on the project about 6 months, we were working through the final testing of a major release, tied a bow on it, shipped to production.
- Later that night we were on this conference call with IT. And I hear this guy just screaming in the background. Apparently, we had shut down every tool in the factory. So we rolled back the release and tried to figure out what happened. There was a configuration change that didn’t quite make it to production. We all felt terrible, but there wasn’t much we could do at this point. So we fixed the problem, and shipped to production... again.
- Back on the conference call with IT. And guess what... the same thing happened. What were we supposed to say… oops? So once again, we rolled back the release.
- We couldn’t reproduce the problem. We spent months trying to figure it out, and we were just completely stumped. We tried everything we could think of. Meanwhile, our development team was pretty much idle so management just told them to go ahead with the next release. Everyone was just working like nothing was wrong, but we couldn’t ship anything. We had a whole nother release in the queue before we finally figured it out. Guess what was wrong? We were scared to death to try again, but we didn’t really have a choice. So we cross our fingers, and shipped to production again.
- Back on the conference call with IT. We were all watching these real time activity charts and holding our breath. Finally everything seemed to be ok. I was so relieved that things would finally be back to normal again. And then about 3am, my phone rang. It was my team lead calling... he asked me about some code that I’d wrote and I knew exactly what happened. I’d made some improvements to the code that caused a memory leak. And my changes ground the system to a screeching halt. This time, it was my fault. So once again, we rolledback the release. This time the rollback failed.
- Fully ramped semiconductor factory. 50k wafer starts a day. Completely offline. My fault. I felt so horrible. I was in my bosses office, just sobbing.
- I thought the main obstacle was all the technical debt building up in the code base that was causing us to make mistakes. and if we made changes in the code that had more technical debt, we’d be more likely to make mistakes. So I got this idea to build a tool that could detect high-risk changes, and tell us where we needed to do more testing -- but what I found wasn’t what I found wasn’t what I expected at all.
- Our bugs were mostly in the code written by the senior engineers on the team where the design actually got the most scrutiny. It’s not like we didn’t have any awful crufty code -- but that’s not where the bugs were. The correlation I did find in the data was this...
- [read] And while that made some sense, I couldn’t help but think, there had to be more to the story...
- When I had to work with complex code, it was really painful. [read]
- So I started keeping track of all my painful interaction with the code and visualizing it on a timeline like this. The pain started [] when I ran into some unexpected behavior and ended [] when I had the problem resolved. So that was 5 hours and 18 minutes of troubleshooting, I think everyone would agree that’s pretty painful.
- The amount of pain was driven by two factors...
- So I started breaking down the problems into categories. And when I did this, I realized that most of the pain was actually caused by human factors.
- This is when I have an idea in my head about how the code is supposed to work, but it doesn’t work that way anymore.
- This is when your running an experiment, and there’s multiple possibilities for how a behavior can occur, and you make a bad assumption, and down the rabbit hole you go. These aren’t really problems with the code itself, [read]
- These aren’t really problems with the code itself… [read]
- The pain isn’t something inside the code, pain occurs during the process of interacting with the code. So I started optimizing for… and I did that, with the help of a data driven feedback loop.
- On our project, we ended up [read] For almost a year! [read]
- [read] Then we started asking []
- [read] That’s when everything changed [] We were finally able to turn the project around. And I learned one of the most valuable lessons in my career. [read]
- The pain was caused by problems building up in the code.
- A typical improvement effort usually starts with brainstorming a list [slow] We think about the things that bugged us recently, how we’re not following best practices, or the code that just makes us feel shameful. [] -- Then all that goes into our technical debt backlog, and we chip away at improvements for months. But just because a problem comes to mind, doesn’t mean it’s an important problem to solve When we’re brainstorming, [] we can easily miss our biggest problems then [our improvements don’t make...]. [] Don’t do this.
- From the outside it looks like we’re trying to drive a car without a steering wheel. We line up the car’s trajectory based on our ideals, then close our eyes and floor the gas pedal.
- Really wanted to help.
- Really wanted to help.
- ©2014 New Iron Group
- ©2014 New Iron Group
- ©2014 New Iron Group
- ©2014 New Iron Group
- ©2014 New Iron Group
- ©2014 New Iron Group
- ©2014 New Iron Group
- ©2014 New Iron Group
- The other thing I think we really need to question is [read] All of these things are really a means to an end. If you’ve ever spent hours and hours troubleshooting a bug in that one line of magic code, it really makes you wonder [read]. [read] What are we aiming for as an industry? So let me summarize what Idea Flow brings to the table.
- Since best practices are solution-focused, we’re always start with the hammer and looking for the nail. Test automation is our favorite hammer. Instead we need to be characterizing all the different types of nails,
- Since best practices are solution-focused, we’re always start with the hammer and looking for the nail. Test automation is our favorite hammer. Instead we need to be characterizing all the different types of nails,
- Really wanted to help.
- Really wanted to help.
- Find out what the substitution thing is called.
- Find out what the substitution thing is called.
- We optimize for execution time, even when the time spent on human cycles can completely dwarf the execution time. Why do you think that is?
- When it comes to solving these really complex problems, our intuition is just wrong. It leads us astray.
- The pain isn’t something inside the code, pain occurs during the process of interacting with the code. The problems I focused on fundamentally changed.
- Avoiding Pain
- Rework Risk is driven by the likelihood... Things like... bad assumptions about the architecture or design or bad assumptions about customer requirements. The longer we delay before making corrections, the greater the rework.
- This is from a project about 10 months old where we actively focused on reducing troubleshooting time. With our everyday problem-solving effort, we still spent about 10-20% of our time on friction.
- So in this first case study, there was a huge mess inherited by a really great team, it was a 12 year old project, where all the original developers had left. This is what it looks like when 90% of your time figuring out what to do, and 10% of your time actually doing stuff. The lack of familiarity has an enormous impact on how much friction we experience.
- So there were tons of problems, and the team wasn’t sure what to focus on, so they set a goal to raise unit test coverage by 5%. If you start adding up all the problems across the team [], these guys were spending about 700 hours per month generating test data to support whatever task they were working on. But oddly, in all the retrospective meetings, this problem didn’t even come up. It was just part of the work.
- This second case studies was with a massive rewrite effort. They had this big monolith application that they rewrote completely from scratch, with micro services, a continuous delivery pipeline, the whole nine years. And what really surprised me about this project, is that after only 18 months, they were already spending 40-60% of their development capacity troubleshooting problems.
- They had this design for the architecture, that looked good on paper, but then once they distributed the design across teams, and discovered the architecture had some flaws, they were stuck. The good ol’ Conway’s law effect, and they couldn’t seem to adapt. So I got involved with the team, just as they were getting into the thrashing stage, and starting to lose control.
- You could see this pattern of pain building up over time, that we always talk about, but have never been able to measure. So I don’t have quite enough data to make a chart like this, but these are some of the patterns you could see.
- First, learning is front-loaded while the team figures out what to do.
- Then there’s this rush before the deadline where validation ends up deferred.
- Then the pain builds, and you see the baseline friction level rising over time.
- Then finally chaos reigns, and the unpredictable work stops fitting in the timebox. So I’m measuring capacity hours over time, so even though all these releases are the same size, you can see how the team had to work twice as many hours to get the release out the door.
- Then management got resentful because nothing was actually getting better.
- Now I want to point out something. For all three projects, these tasks all took one to three days. Generally speaking, as the problems build, we can still break down the work into bite-sized chunks. but what we work on during that time dramatically changes. [read] even when the problems are severe. So if you thought about how much time you spend doing troubleshooting, learning and rework. What percentage of time do you think it would be? Which do you think is the biggest? What do you think the biggest causes are of troubleshooting time?
- So If I wanted to know what was causing the pain I needed to understand the things that caused these 2 factors. A lot of the problems had more to do with human factors than anything going on with the code. Stale Memory mistakes, Ambiguous Clues. But once I understood what was causing the pain, [read -- most of the problems were easy to avoid] For example...
- Really wanted to help.
- Really wanted to help.
- We’d always get a different set of bugs. What would you do?
- Troubleshooting Risk we’ve already talked about, it’s driven by the likelihood...
- So If I wanted to know what was causing the pain I needed to understand the things that caused these 2 factors. A lot of the problems had more to do with human factors than anything going on with the code. Stale Memory mistakes, Ambiguous Clues. But once I understood what was causing the pain, [read -- most of the problems were easy to avoid] For example...
- For the problem categories -- I use hashtags in the Idea Flow Maps, then add up the durations for each hashtag.
- For the problem categories -- I use hashtags in the Idea Flow Maps, then add up the durations for each hashtag.
- For the problem categories -- I use hashtags in the Idea Flow Maps, then add up the durations for each hashtag.
- [read] And while that made some sense, I couldn’t help but think, there had to be more to the story...
- [read] And while that made some sense, I couldn’t help but think, there had to be more to the story...
- We’d always get a different set of bugs. What would you do?
- Really wanted to help.
- Really wanted to help.
- So we’re all familiar with the haystack principle…
- Then I realized George was missing a key conceptual model.
- N
- Rework Risk is driven by the likelihood... Things like... bad assumptions about the architecture or design or bad assumptions about customer requirements. The longer we delay before making corrections, the greater the rework.
- Used thinking checklists to codify a decision-making process… let me show you what I mean.
- Focus on one decision principle until you have it down.
- It’s not that best practices are bad, or wrong, they’re just backwards.
- Really wanted to help.
- Really wanted to help.
- We start off with the best of intentions… We're going to write high quality code that’s low in technical debt, easy to maintain and of course, has good code coverage.
- So I started working 60-70 hour weeks for about 6 months strait. And my team started working 60 hour weeks for 6 months strait. Then the releases started falling apart. Things were just constantly going wrong.
- Crazy deadlines, and I tried to explain to management that we needed to go slower, but they threatened to outsource the project if we didn’t get it done. This project was my baby.
- When we fall into urgency mode, we start compromising safety for speed. We make decisions that don’t seem like a big deal at the time, but they create a hazardous work environment. Instead of taking a little more time to put our toys away, we end up falling down the stairs and in the hospital.
- First, we ignore the risks, basically ignoring tango -- a lot of times because the risk isn’t obvious.
- First, we ignore the risks, basically ignoring tango -- a lot of times because the risk isn’t obvious.
- First, we ignore the risks, basically ignoring tango -- a lot of times because the risk isn’t obvious.
- First, we ignore the risks, basically ignoring tango -- a lot of times because the risk isn’t obvious.
- First, we ignore the risks, basically ignoring tango -- a lot of times because the risk isn’t obvious.
- Troubleshooting Risk we’ve already talked about, it’s driven by the likelihood...
- When we fall into urgency mode, we start compromising safety for speed. We make decisions that don’t seem like a big deal at the time, but they create a hazardous work environment. Instead of taking a little more time to put our toys away, we end up falling down the stairs and in the hospital.
- Next thing you know we’re working late nights and weekends, choking down red bull to stay awake... hacking out last minute fixes and hoping that nothing else breaks. Who’s done this before? We make jokes about programmers running on caffeine and pizza... but this problem is really serious. When our project is on the line, we give up a lot -- we’re skipping our kids recitals, missing our annivery dinners, we get sick, we gain weight stress deteriorates our health and can tear apart our relationships. Just because we’re not bleeding doesn’t mean we don’t get hurt by all this. We can’t run a sustainable business by compromising the safety of the people doing the work.
- I watched my project get crushed.
- [read] What do you guys think? What are the biggest obstacles?
- If our feedback loop is broken, we don’t respond.
- Troubleshooting Risk we’ve already talked about, it’s driven by the likelihood...
- Learning Risk is driven by the likelihood... Things like... lots of 3rd party libraries, complex frameworks, a really large code base, or high turn-over rate -- all these things can cause extra learning work.
- Rework Risk is driven by the likelihood... Things like... bad assumptions about the architecture or design or bad assumptions about customer requirements. The longer we delay before making corrections, the greater the rework.
- We make decisions that save a few hours that lead to side effects that cost several hours. When we try to go faster, we do things that increase the likelihood of mistakes and the cost to recover when things go wrong. We’ve been in this pattern for the last 2 years, and now we’re here.
- So If I wanted to know what was causing the pain I needed to understand the things that caused these 2 factors. A lot of the problems had more to do with human factors than anything going on with the code. Stale Memory mistakes, Ambiguous Clues. But once I understood what was causing the pain, [read -- most of the problems were easy to avoid] For example...
- We’ve been collecting lots of data and have identified our biggest problems. I think we can dramatically reduce risk with some focused effort. I'd like to propose a 3-month trial [] with one person working full-time on these problems. The team will make decisions on the improvement work, and I’ll share our progress and lessons learned with you each month. I know we can do this, but I need your help. Will you help me make this happen?
- Really wanted to help.
- From the outside it looks like we’re trying to drive a car without a steering wheel. We line up the car’s trajectory based on our ideals, then close our eyes and floor the gas pedal.
- [read] That’s when everything changed [] We were finally able to turn the project around. And I learned one of the most valuable lessons in my career. [read]
- Iterative clarify then implement “better”
- Iterative clarify then implement “better”
- If you want to join me, then read the book, and think about the ideas, see if this is something you want to be a part of. You can either buy the book, or if you start a reading group for Idea Flow, I’ll provide free e-books for all the attendees. Check out openmastery.org for details.
- And if you don’t want to take my word for it, you should read Idea Flow because Rene and Matt said so.