Learn Your Way to AWESOME.
•Download as PPTX, PDF•
1 like•757 views
Once we make our pain visible with Idea Flow Mapping, we've got a data-driven feedback to learn what works. Objective data enables us to do something we've never been able to do before in our industry: science. This talk is about how to do science in software development. The Lean Startup community has pioneered the art of everyday science to reduce the risk of building the wrong product by running customer experiments to learn what works. By mapping these same basic scientific principles to technical risk management, we can run experiments to learn our way to AWESOME! Edit Archive Delete In this talk we'll cover: How science is used in the Lean Startup world to run business model experiments How science is used in the Lean Manufacturing world to support process control & supply chain optimization How we can apply science in software development to systematically learn what works. If you want to start learning and improving faster than ever before, you won't want to miss this talk.
Recommended
More Related Content
What's hot
What's hot (13)
Viewers also liked
Viewers also liked (11)
Similar to Learn Your Way to AWESOME.
Similar to Learn Your Way to AWESOME. (20)
Recently uploaded
Recently uploaded (20)
Learn Your Way to AWESOME.
- 1. Janelle Klein openmastery.org @janellekz Point A Point AWESOME. Learn Your Way to AWESOME.
- 2. Focus of this Talk: Using Science to Fuel Software Mastery “Data-Driven Software Mastery”
- 3. Overview How can we measure development pain? How do we create a data-driven feedback loop? What does this look like in practice?
- 4. Overview How can we measure development pain? How do we create a data-driven feedback loop? What does this look like in practice?
- 5. Great Team Disciplined with Best Practices Constantly Working on Improvements+ Project FAILURE About 8 years ago…
- 6. Technical Debt Mistakes I thought the problem was Technical Debt
- 7. Most of our mistakes were in the most well-written parts of the code. Mistakes
- 8. We made significantly more mistakes in code that we didn’t write ourselves. Lower Familiarity More Mistakes= There had to be more to the story...
- 9. PAIN This is what I knew... What made development feel painful?
- 10. Unexpected Behavior Problem Resolved Tracking Painful Interaction with the Code (Friction) Troubleshooting Progress 5 hours and 18 minutes of troubleshooting... PAINFUL
- 11. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger Most of the pain was caused by human factors. What causes PAIN?
- 12. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Most of the pain was caused by human factors. Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm
- 13. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Most of the pain was caused by human factors.
- 14. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? PAIN is a consequence of how we interact with the code. Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm
- 15. PAIN occurs during the process of understanding and extending the software PAIN Not the Code. Optimize “Idea Flow”
- 16. My team spent tons of time working on improvements that didn’t make much difference. We had tons of automation, but the automation didn’t catch our bugs.
- 17. My team spent tons of time working on improvements that didn’t make much difference. We had well-modularized code, but it was still extremely time-consuming to troubleshoot defects.
- 18. The hard part isn’t solving the problems it’s identifying the right problems to solve. “What are the specific problems that are causing the team’s pain?”
- 19. PAIN occurs during the process of understanding and extending the software PAIN Not the Code. Optimize “Idea Flow”
- 20. The Rhythm of “Idea Flow” Write a little code. Work out the kinks. Write a little code. Work out the kinks. Write a little code. Work out the kinks.
- 21. Write a little code. Work out the kinks. The Rhythm of “Idea Flow” Does the behavior match our expectations?
- 22. The Rhythm of “Idea Flow” Write a little code. Work out the kinks. Adjustment LoopProgress Loop
- 23. “Friction” in Idea Flow Write a little code. Work out the kinks. Adjustment LoopProgress Loop
- 24. Idea Flow Mapping Tools (Open Source, Supported GA ~August 2016) github.com/ideaflow/tools
- 26. “What caused the pain in this case?” Categorize the Problems with #HashTags #ReportingEngine #Hibernate #MergeHell
- 27. 1. Problem A 2. Problem B 3. Problem C Add up the Pain by Category What’s the biggest problem to solve?
- 28. Overview How can we measure development pain? How do we create a data-driven feedback loop? What does continuous improvement look like? What should we be trying to optimize for?
- 29. The Lean Startup Eric Reis Pioneers in the Art of “Everyday Science”
- 30. “The Art of the Scientist” Workshop, Ash Maurya Ash: “What’s the most important thing that scientists do?” “Run Experiments!” “Analyze Data!” Getting Involved with Lean Startup
- 31. “The Art of the Scientist” Workshop, Ash Maurya NO! ? ? ? ? ? ? ? Getting Involved with Lean Startup
- 32. “The Art of the Scientist” Workshop, Ash Maurya Ash: “The most important things scientists do is build models.” Getting Involved with Lean Startup
- 33. Ceiling of Achievement, Ash Maurya Lots of learning, but not learning the “right things.”
- 34. Floor of Improvement? Lots of improving, but not improving the “right things.”
- 35. Modeling (Inductive) Experimentation (Deductive) Empirical Data We can’t ignore this part. The Scientific Method
- 36. Experimentation = Deductive Reasoning
- 37. Modeling = Inductive Reasoning
- 38. Modeling (Inductive) Experimentation (Deductive) Empirical Data Creating an explicit model makes our beliefs testable and our discoveries additive. Science is a Feedback Loop We can’t ignore this part.
- 39. “The Customer Factory”, Ash Maurya Optimize the Rate of Happy Customer Flow
- 40. Unaware Visitors Users Retention User Repeats Experience Currency (not necessarily money) Customers Referrals Customer Value > Customer Cost > Your Costs Amusement Park Interested Prospects
- 41. “The Art of the Scientist” Lean Startup Workshop, Ash Maurya Ash: “The goal isn’t learning, the goal is traction.”
- 42. We can measure flow through the Customer Factory. How do we know if we’re getting traction?
- 44. Focus on Limiting Constraint Optimize the Rate of Flow in a Supply Chain
- 45. “The Customer Factory”, Ash Maurya Optimize the Rate of Happy Customer Flow Identify the Limiting Constraint
- 46. The Three Steps of Science 1. Measure the “What” 2. Explain the “Why” 3. Experiment to Learn “How”
- 47. Lean Startup: Science-Driven Refactoring Target: Optimize Rate of Factory Flow Pain Signal: “Friction” in Flow
- 48. This is “Out of Control” Lower Variability = Better Control Process Control in Lean Manufacturing Quality Target Upper Control Limit Lower Control Limit
- 49. Symptom Process Control in Lean Manufacturing Symptom Symptom Cause Cause Fix Fix Fix FixCause Cause Symptom
- 50. Symptom Symptom Symptom Cause Cause Fix Fix Fix FixCause Cause Symptom What Why How Process Control in Lean Manufacturing
- 51. Average Friction per Customer This is “Traction” Optimize for Customer Happiness High Friction Min Friction “No” Threshold Customer Happiness Categorize the “No” Type
- 52. How do we use science to create a feedback loop in software development?
- 53. Problems Look Avoidable in Hindsight z 14:230:00 Big haystack of changes will make troubleshooting difficult “That was a dumb thing to do, I’ll avoid that next time…
- 54. Imagine your brain is a decision-making engine written in code. Breakpoint Human Brain = Decision-Making Machine
- 55. The Challenge We tend to make auto-pilot decisions and do what’s familiar How do we modify decision habits?
- 56. Breakpoint Stop and Think! Turn Auto-Pilot Decisions into Conscious Decisions The Awareness Challenge: We have to recognize the situation and predict the pain in order to modify the decision. Now we can modify the decision code!
- 57. Improve Quality of Decisions Data-Driven Software Mastery
- 58. Target - The direction of “better” Data-Driven Software Mastery
- 59. Input - The constraints that limit our short-term choices… Data-Driven Software Mastery
- 60. Output - The pain signal we’re trying to improve Data-Driven Software Mastery
- 61. Focus on the biggest pain… Data-Driven Software Mastery
- 62. 1. Visibility - Identify the specific patterns 1. Visibility Data-Driven Software Mastery
- 63. 2. Clarity - Understand cause and effect 2. Clarity Data-Driven Software Mastery
- 64. 3. Awareness 3. Awareness - Stop and think to adjust habits Data-Driven Software Mastery
- 65. 4. Run Experiments to Learn What Works Data-Driven Software Mastery
- 66. Average Pain per Incident This is Traction Target Flow Control Limit Categorize the Pain Type “Pain Control” in Software Development
- 67. Symptom Symptom Symptom Cause Cause Fix Fix Fix FixCause Cause Symptom What Why How Refactor BrainPain Type
- 68. Breakpoint Stop and Think! We have to recognize the situation and predict the pain in order to modify the decision. The Awareness Challenge: What —> Why —> How
- 69. George Lakoff Metaphors We Live By Jeff Hawkins On Intelligence + The Foundational Theory Theory of Understanding Theory of Prediction
- 70. What —> Why
- 71. Observe Pattern “Do I know this pattern?” Metaphors are patterns of conceptual “Shapes”
- 72. “What is Clojure?” Lisp property: Functional Language Groovy property: JVM Language property: Object-Oriented Language Clojure! property: JVM Language property: Functional Language How Our Brain Creates a New Shape Recipe
- 73. Three Different Types of Metaphors Object Patterns (Thing) Context/Relationship Patterns (Space) Process Patterns (Time)
- 74. Technical Debt Debt is a “Thing”
- 75. “This looks like a tech debt thing.” 1. Categorize patterns by concept type
- 76. “This looks like a painful tech debt thing.” 2. Categorize variations by intensity. Pain level = 8
- 77. = Little Debt = Medium Debt = Big Debt Beliefs About Developer Pain
- 78. PAIN occurs during the process of understanding and extending the software PAIN Pain occurs over TIME.
- 79. What do you think happens when we make this transition? Process Patterns (Time) Object Patterns (Thing)
- 80. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Take a Closer Look at the Patterns Familiarity Mistakes Stale Memory Mistakes Semantic Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger Most of the pain was caused by human factors.
- 81. Journey Beginning End Think about the Developer’s Experience as a Problem-Solving Journey
- 82. Journey Constraints What were the biggest challenges on the journey?
- 83. Friction Journey What were the biggest challenges on the journey?
- 84. Journey Strategy What were the biggest challenges on the journey?
- 85. Journey Uncertainty What were the biggest challenges on the journey? ???
- 86. 8:150:00 George’s Painful Experience Look for Patterns in Experience Reviews What pain did you experience? What situation caused the pain? What strategies did you try? What strategies would you try if you could do it again?
- 87. Situation Strategy Consequence Developer Journeys Clarity = Understand Why Breakdown the Journeys into Patterns
- 88. Past StrategiesSituation Consequence Prediction Machine “Gut Feel” arises through pattern reinforcement Clarity = Understand Why Breakdown the Journeys into Patterns
- 89. Symptom Symptom Symptom Cause Cause Fix Fix Fix FixCause Cause Symptom What Why How Refactor BrainPain Type
- 90. Why —> How
- 91. Breakpoint Stop and Think! Turn Auto-Pilot Decisions into Conscious Decisions The Awareness Challenge: We have to recognize the situation and predict the pain in order to modify the decision. Now we can modify the decision code!
- 92. The Checklist Manifesto Atul Gawande Scaling Lean Ash Maurya + Change Habits with Checklist Experiments
- 93. The Haystack Principle Lots of unvalidated changes Easier to find the needle. The Haystack Principle Optimize for small manageable haystacks.
- 94. 14:230:00 I want to avoid this… Thinking Checklist Is my current approach likely to cause a big haystack? Situation: start of subtask Let’s Make a Checklist! “What question could I ask my future self to recognize similar risks in the future?” “In what situation would I ask the question?”
- 95. 0:00 Stop and Think: Is my current approach likely to cause a big haystack? Predict: Small haystack Low Friction Checklist Experiments
- 96. 18:120:00 Stop and Think: Is my current approach likely to cause a big haystack? False Prediction Predict: Small haystack Low Friction Checklist Experiments
- 97. 18:120:00 Stop and Think: Is my current approach likely to cause a big haystack? False Prediction High-Risk Situations 1. Unraveling sweater 2. Integration-heavy change 3. High state variation 4. Minimum scope is big Q: Is my current approach likely to cause a big haystack? Start of Subtask Checklist Experiments
- 98. Strategy Types (“do”) 1. DependencyAnalysis 2. IncrementalIntegrationTest 3. DataDrivenTest 4. IsolateHardToTestCode Situation Types (“see”) 1. UnravelingSweater 2. HeavyIntegrationLogic 3. HighStateVariation 4. CoupledExternalDependencies Haystack Decisions
- 99. Adhoc Predictions While You Work I predict this <strategy> will avoid this <pain> General rule of thumb: Predict the pain and throw as many patterns as you can in a sentence I see a risk of this <situation>, and I predict this <strategy> will help me avoid it
- 100. Symptom Symptom Symptom Cause Cause Fix Fix Fix FixCause Cause Symptom What Why How Refactor BrainPain Type
- 101. Improve Quality of Decisions Data-Driven Software Mastery
- 102. If any part of the feedback loop is broken: Our improvements don’t make much difference.
- 103. Average Pain per Incident This is Traction Target Flow Control Limit Categorize the Pain Type “Pain Control” in Software Development
- 104. Overview How can we measure development pain? How do we create a data-driven feedback loop? What does this look like in practice?
- 106. 1. Clarify the Strategy
- 107. 1. Clarify the Strategy
- 108. 1. Clarify the Strategy
- 109. Look for #Patterns. 1. Clarify the Strategy
- 110. Janelle’s Painful Experience 2. Record an Idea Flow Map Make adhoc predictions as you work
- 111. 3. Reflect on Decisions (Experience Review)
- 112. 3. Reflect on Decisions (Experience Review)
- 113. 3. Reflect on Decisions (Experience Review)
- 114. Look for #Patterns. 3. Reflect on Decisions (Experience Review)
- 116. Focus Meeting: Mastery Circle Circle Leader Circle Member Focus: What pain did we experience? Observe: Ask questions about the facts Conclude: What major factors caused pain? Optimize: What strategies to reduce the pain? Learn: What experiments could we run? Observation Questions Make a F.O.C.O.L Point! Summarize lessons learned in a #HashTagged Journey. (focused problem-solving discussion)
- 117. Dedicated Capacity for Improvement Work Long-term feedback loop
- 118. LEARN YOUR WAY TO AWESOME. Free to Join Industry Peer Mentorship Network openmastery.org
- 119. Developer Insight Tools Platform Idea Flow Mapping Tools Team Insight Tools Team Joe Sally Mark Eric Community Insight Tools Anonymized Data (REST) Shared Taxonomy of Patterns & Principles (with example data)
- 120. Janelle Klein openmastery.org @janellekz Check out openmastery.org for details. Read my Book. Think About It. FREE with MembershipBuy It How to Measure the PAIN in Software Development Janelle Klein
Editor's Notes
- Hi everyone, this is Janelle Klein, founder of Open Mastery, and author of Idea Flow Learning Framework. So I wanted to tell you about Business Mastery Learning Framework, the new research project we’re working on, so you can see if this sounds like something you’d want to be apart of.
- I thought the main obstacle was all the technical debt building up in the code base that was causing us to make mistakes. and if we made changes in the code that had more technical debt, we’d be more likely to make mistakes. So I got this idea to build a tool that could detect high-risk changes, and tell us where we needed to do more testing -- but what I found wasn’t what I found wasn’t what I expected at all.
- Our bugs were mostly in the code written by the senior engineers on the team where the design actually got the most scrutiny. It’s not like we didn’t have any awful crufty code -- but that’s not where the bugs were. The correlation I did find in the data was this...
- [read] And while that made some sense, I couldn’t help but think, there had to be more to the story...
- When I had to work with complex code, it was really painful. [read]
- So I started keeping track of all my painful interaction with the code and visualizing it on a timeline like this. The pain started [] when I ran into some unexpected behavior and ended [] when I had the problem resolved. So that was 5 hours and 18 minutes of troubleshooting, I think everyone would agree that’s pretty painful.
- So I started breaking down the problems into categories. And when I did this, I realized that most of the pain was actually caused by human factors.
- This is when I have an idea in my head about how the code is supposed to work, but it doesn’t work that way anymore.
- This is when your running an experiment, and there’s multiple possibilities for how a behavior can occur, and you make a bad assumption, and down the rabbit hole you go. These aren’t really problems with the code itself, [read]
- These aren’t really problems with the code itself… [read]
- The pain isn’t something inside the code, pain occurs during the process of interacting with the code. So I started optimizing for… and I did that, with the help of a data driven feedback loop.
- On our project, we ended up [read] For almost a year! [read]
- [read] Then we started asking []
- [read] That’s when everything changed [] We were finally able to turn the project around. And I learned one of the most valuable lessons in my career. [read]
- The pain isn’t something inside the code, pain occurs during the process of interacting with the code. So I started optimizing for… and I did that, with the help of a data driven feedback loop.
- There’s this decision point when we validate the code and ask ourselves, “does the behavior match our expectations?”
- When we try to make progress, learning gets in the way. Especially if we’re unfamiliar with the system. And with conflict loops we have to troubleshoot the problem, then rework our solution before we get back to a stable state.
- Quick demo.
- Then by categorizing all the different types of problems I was running into, we could start adding up the time spent for each category, and we could figure out what was causing our biggest problems.
- So If I wanted to know what was causing the pain I needed to understand the things that caused these 2 factors. A lot of the problems had more to do with human factors than anything going on with the code. Stale Memory mistakes, Ambiguous Clues. But once I understood what was causing the pain, [read -- most of the problems were easy to avoid] For example...
- A little history here…
- That one sentence, fundamentally changed the trajectory of the next 3 years of my life.
- So we [read] measure the pain, explain what’s causing the pain, then run experiments to systematically learn what works. And for the model, we’re breaking down the lessons learned into patterns and what I call “decision principles”, the optimization rules for how to make trade-off decisions that reduce friction.
- So we [read] measure the pain, explain what’s causing the pain, then run experiments to systematically learn what works. And for the model, we’re breaking down the lessons learned into patterns and what I call “decision principles”, the optimization rules for how to make trade-off decisions that reduce friction.
- Ash talked about how the entrepreneurs he was working with were doing lots of learning, but spinning their wheels and getting nowhere. So he’s like: [read] And he wrote this incredible book, Scaling Lean which is some of the most brilliant material I’ve seen in years. Essentially describing the art of traction.
- This is a feedback loop.
- Rework Risk is driven by the likelihood... Things like... bad assumptions about the architecture or design or bad assumptions about customer requirements. The longer we delay before making corrections, the greater the rework.
- N
- Rework Risk is driven by the likelihood... Things like... bad assumptions about the architecture or design or bad assumptions about customer requirements. The longer we delay before making corrections, the greater the rework.
- Rework Risk is driven by the likelihood... Things like... bad assumptions about the architecture or design or bad assumptions about customer requirements. The longer we delay before making corrections, the greater the rework.
- Metaphors and language shape the way we see everything in the world around us. We don’t even realize how many of them affect us. Look for patterns in things, instead of look for patterns in processes.
- If we don’t make time to deal with emerging risks and emerging risks, we will never get out of this cycle.
- An attribute of a process is fundamentally different than an attribute of a thing. It’s the difference between a data file, and a program that generates data files.
- This happens all the time in software development. Imagine the idea you had conceptually mapped to an “it” becomes a “function for producing its.”
- Then you start recognizing the patterns, “aha! this is one of those situations, or… I should keep on eye on that.” If there’s one thing that will accelerate your learning faster than anything else, it’s this. Need to also mention Open Mastery Online. People in the Mentorship program we’re going to build a statical process control system for software development.
- Then you start recognizing the patterns, “aha! this is one of those situations, or… I should keep on eye on that.” If there’s one thing that will accelerate your learning faster than anything else, it’s this. Need to also mention Open Mastery Online. People in the Mentorship program we’re going to build a statical process control system for software development.
- Then you start recognizing the patterns, “aha! this is one of those situations, or… I should keep on eye on that.” If there’s one thing that will accelerate your learning faster than anything else, it’s this. Need to also mention Open Mastery Online. People in the Mentorship program we’re going to build a statical process control system for software development.
- Then you start recognizing the patterns, “aha! this is one of those situations, or… I should keep on eye on that.” If there’s one thing that will accelerate your learning faster than anything else, it’s this. Need to also mention Open Mastery Online. People in the Mentorship program we’re going to build a statical process control system for software development.
- Rework Risk is driven by the likelihood... Things like... bad assumptions about the architecture or design or bad assumptions about customer requirements. The longer we delay before making corrections, the greater the rework.
- Used thinking checklists to codify a decision-making process… let me show you what I mean.
- We optimize for execution time, even when the time spent on human cycles can completely dwarf the execution time. Why do you think that is?
- Focus on one decision principle until you have it down.
- Then I realized George was missing a key conceptual model.
- So here’s what I’m doing about it, because I really give a shit about these problems getting solved. I built an organization to carry the torch.
- With anonymized data, that we can all study together, solve problems together, and codify our knowledge into patterns and principles backed by data rather than anecdote and gut feel.
- If you want to join me, then read the book, and think about the ideas, see if this is something you want to be a part of. You can either buy the book, or if you start a reading group for Idea Flow, I’ll provide free e-books for all the attendees. Check out openmastery.org for details.