No (Lab) Jacket Required: Designing Experiments for Learning [2019 Flowcon]
•
1 like•261 views
Slides as presented during the workshop with Karl Scotland at Flowcon, Paris, Dec. 12, 2019. https://www.flowcon.fr/ Hypothesis-Driven Development is thinking about the development of new ideas, products and services – even organizational change – as a series of experiments to determine whether an expected outcome will be achieved, so we need to know how to design and run experiments properly. This session helps participants understand the importance of using experiments to help teams and organizations learn and improve, while giving hands-on practice in designing experiments to yield measurable evidence for that learning. We’ll even play a couple of inductive-logic games to get us in the mindset of how to write tests to validate – and invalidate – our hypotheses in order to acquire knowledge. Whether you’re in product development or organizational improvement, for those wanting to move from projects to experiments in the quest of continuous learning and evolutionary improvement, this session will provide the tools and mindset for going beyond simply calling something an experiment, to conducting experiments with the proper rigor to maximize learning.
Recommended
More Related Content
Similar to No (Lab) Jacket Required: Designing Experiments for Learning [2019 Flowcon]
Similar to No (Lab) Jacket Required: Designing Experiments for Learning [2019 Flowcon] (20)
More from Matthew Philip
More from Matthew Philip (20)
Recently uploaded
Recently uploaded (20)
No (Lab) Jacket Required: Designing Experiments for Learning [2019 Flowcon]
- 1. @kjscotland #FlowCon @mattphilip Karl Scotland Matt Philip NO (LAB) JACKET REQUIRED: DESIGNING EXPERIMENTS FOR LEARNING
- 4. @kjscotland #FlowCon @mattphilip ”LET’S EXPERIMENT!”
- 5. @kjscotland #FlowCon @mattphilip WHY EXPERIMENTS? In complex environments, you can’t follow recipes or conduct detailed analysis to understand the situation. Rather, you must experiment (probe). Cynefin Improve collaboratively, evolve experimentally. Kanban Method Pivoting …. a structured course correction designed to test … hypothesis about the product, strategy, and engine of growth. Lean Startup Each feature includes a benefit hypothesis. SAFe Complex Complicated Simple Chaotic Relationship between cause and effect clear only in hindsight Relationship between cause and effect requires analysis Relationship between cause and effect obvious to all No relationship between cause and effect at system level “If … maybe?” “If… then… then” “Now that…” “If… then”
- 6. @kjscotland #FlowCon @mattphilip ELEUSIS EXPEDITIOUS A GAME ABOUT HYPOTHESES
- 7. @kjscotland #FlowCon @mattphilip IDEA ◆ This is a variation on Eleusis Express; a card game of inductive reasoning which itself is a modification of Robert Abbott’s game Eleusis. ◆ One player has a secret rule for which cards can be played. ◆ For example: Each card has to be a different color from the card before it. ◆ Other players lay down cards to discover the pattern. If a player lays down a card correctly, he or she can try to guess the rule.
- 8. @kjscotland #FlowCon @mattphilip OBJECT ◆ A game consists of one or more rounds. ◆ A different player is chosen as oracle of each new round, and it is the oracle who knows the secret rule. The oracle does not play any cards that round. ◆ All plays are made to a central layout that grows as the round progresses. A layout consists of a horizontal mainline of correct cards that follow the secret rule. Below the mainline are vertical sidelines of cards that did not follow the rule. These sideline cards can be overlapped to save space.
- 9. @kjscotland #FlowCon @mattphilip ORACLE ◆ The oracle has a secret rule. ◆ He or she places the pile of cards face down, turns over the top card, and puts it on the table. This will be the start of the mainline. ◆ Before play starts, the oracle may give a hint about the rule. ◆ The player to the left of the oracle goes first, then the play continues around to the left.
- 10. @kjscotland #FlowCon @mattphilip PLAYERS ◆ A player turns the top card from the deck and lays it on the table. He or she asserts whether it is correct or not, and the oracle accepts or rejects. ◆ If correct, the card goes to the right of the last card on the mainline. ◆ If incorrect, the card goes below the last correct card (it either starts a sideline or it adds to a sideline).
- 11. @kjscotland #FlowCon @mattphilip GUESSING ◆ Whenever a player makes a correct assertion, he or she can guess the rule. Everyone must hear the guess. ◆ The oracle then says whether the player is right or wrong. If wrong, the game continues. If right, the round ends.
- 13. @kjscotland #FlowCon @mattphilip LEAN MASTERMIND
- 14. @kjscotland #FlowCon @mattphilip 1. Choose who is who. One player becomes the codemaker, the others are the codebreakers 2. Choose 4 digits. The codemaker chooses a pattern of four digits; for example 1 2 3 4. 3. Try guessing the pattern. The codebreakers try to guess the pattern, in both order and number, within 12 turns. Each guess is made by writing a row of four. 4. Provide feedback. After each guess the codemaker provides feedback by placing from zero to four key dots next to the guess in the following order. 1. A full key dot [●] indicates the existence of a code number which is correct in both number and position. 2. An empty dot [○] indicates the existence of a correct number placed in the wrong position. 5. Keep guessing. Once feedback is provided, another guess is made. Guesses and feedback continue until either the codebreakers guess correctly, or 12 incorrect guesses are made. INSTRUCTIONS
- 15. @kjscotland #FlowCon @mattphilip As before, with the following additions: • Before a guess: Write down the intent of the guess – what it is you want to learn. • After feedback: Write down the result and what you actually learned. WORKSHEET
- 16. @kjscotland #FlowCon @mattphilip RULES (FOR ORACLE’S EYES ONLY) 1.If the last card was red, play a black card. If the last card was black, play a red card. 2.If the last card was a spade, play a heart; if last card was a heart, play a diamond; if last was diamond, play club; and if last was club, play a spade. 3.The cards on the mainline must follow this pattern: three red cards, then three black, then three red, then three black, etc. 4.If the last card is an odd-numbered card, play an even-numbered card; if the last is even, play an odd. [When numbers are involved, ace is 1 (odd), jack is 11 (odd), queen is 12 (even), and king is 13 (odd). 5.If the last card is among the cards ace to 7, play a card 8 to king. If last is among 8 to king, play ace to 7. 6.Play a card with a number that is 1, 2, or 3 higher than the number of the last card. The numbers can “turn-the-corner.” 7.If the last card is an odd-numbered card, play a red card. If the last card is even, play a black card. 8.The card played must be the same suit or the same number as the last card. 9.If the last card is black, play a card with a number that is equal to or lower than the number of the last card. If the last card is red, play a card equal to or higher than the last card.
- 18. @kjscotland #FlowCon @mattphilip • Define a question • Gather information and resources (observe) • Form an explanatory hypothesis • Test the hypothesis by performing and experiment and collecting data in a replicable manner • Analyze the data • Interpret the data and draw conclusions that serve as a starting point for a new hypothesis • Publish results SCIENTIFIC METHOD
- 20. @kjscotland #FlowCon @mattphilip INFORMATION THEORY CURVE
- 21. @kjscotland #FlowCon @mattphilip HOW MUCH LEARNING? @mattphilip #AgileMW2019
- 23. @kjscotland #FlowCon @mattphilip AGILE-DELIVERY VALUE CURVE: NO FEATURE EVER FAILS?
- 25. @kjscotland #FlowCon @mattphilip VALIDATED LEARNING?
- 27. @kjscotland #FlowCon @mattphilip WHAT IS “DONE” FOR YOU? -- John Cutler
- 28. @kjscotland #FlowCon @mattphilip GENERATING EXPERIMENTS
- 30. @kjscotland #FlowCon @mattphilip • Company policy to give a bonus to employees who refer new hires • Implementing SAFe • Rolling out a new team-leadership model • Work-from home policy • [Your own example] HOW WOULD YOU TEST THESE?
- 31. @kjscotland #FlowCon @mattphilip • Psychological safety to fail • Learning is the outcome • Time-based • Measurable • Single variable (Control!) • Provable (and disprovable) hypothesis ELEMENTS OF GOOD EXPERIMENTS
- 32. @kjscotland #FlowCon @mattphilip HYPOTHESIS-DRIVEN DEVELOPMENT
- 33. @kjscotland #FlowCon @mattphilip • Language: “Fail Fast” ! “Prove/disprove hypothesis” • Psychological Safety • Leaders: Establish the boundaries (SETT) • Organizational experiments board • “Goldratt User Stories” • Finding your optimal WIP limit • Retrospective actions • A/B tests OTHER CONSIDERATIONS
- 35. @kjscotland #FlowCon @mattphilip “CURATOR OF EXPERIMENTS”
- 36. @kjscotland #FlowCon @mattphilip SOURCES AND RESOURCES ◆ Karl Scotland (https://availagility.co.uk/) ◆ https://yorkesoftware.com/2018/03/09/goldratt-user-stories/ ◆ http://cognitive-edge.com/methods/safe-to-fail-probes/ ◆ Eleusis Express http://www.logicmazes.com/games/eleusis/express.html ◆ Freakonomics Radio: 370. How to Fail like a Pro ◆ Unlearn book by Barry O’Reilly ◆ Agile BOSSA nova book ◆ How to Measure Anything book ◆ Pretty much anything from John Cutler ◆ The Surprising Power of Online Experiments (https://hbr.org/2017/09/the-surprising-power-of-online- experiments) ◆ https://exp-platform.com/Documents/2015-08OnlineControlledExperimentsKDDKeynoteNR.pdf ◆ http://www.theagilefactor.com/ ◆ Thinking in Bets by Annie Duke ◆ https://barryoreilly.com/how-to-implement-hypothesis-driven-development/ ◆ Online Experimentation at Microsoft ◆ Elizabeth Ayer, FlowCon 2019 - Beyond the Black Hole: Product Management for Continuous Delivery ◆ http://mattphilip.wordpress.com