This class will present hypothesis-driven development, the cutting-edge paradigm for evolving validated products. We’ll dive into how to frame hypotheses, design experiments, and use A/B testing to gather data to prove or disprove our ideas.

1. i290 lean/agile product management
unit 6: experimental product development
@jezhumble
https://lapm.continuousdelivery.com/
humble@berkeley.edu
This work © 2015-2016 Jez Humble
Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

2. identify experiments to test hypotheses
understand how to do outcome-based planning
describe hypothesis-driven development
understand why small batches are important
define A/B testing and the culture it enables
learning outcomes

3. Epic
Theme
Story

4. impact mapping
Gojko Adzic, Impact Mapping

5. working backwards
http://www.allthingsdistributed.com/2006/11/working_backwards.html

6. @jezhumbleJeff Gothelf “Better product definition with Lean UX and Design” http://bit.ly/TylT6A
hypothesis-driven delivery
We believe that
[building this feature]
[for these people]
will achieve [this outcome].
We will know we are successful when we see
[this signal from the market].

8. COST OF EXPERIMENTS
8
Production
Software
SPEED
COST
new services
feasibility spike
service
substitution
integration
Quantitative
forecasting
real-time price
experiment
Data sampling
and modeling tests
Sketches &
Paper Prototypes
Interactive
Prototype
Software
demo
Interviews
& surveys
micro-niche
Wizard of Oz
VIABILITY (BUSINESS) | DESIRABILITY (CUSTOMER) | FEASIBILITY (TECH)

9. exercise
• choose a hypothesis from your assignment
• design an experiment to test your hypothesis
• what do you expect the results to be?
• what result will confirm your hypothesis?
• what result will disprove your hypothesis?
• how soon can we get the result?

10. “Etsy’s Product Development with Continuous Experimentation”
Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI

11. “Etsy’s Product Development with Continuous Experimentation”
Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI

12. “Etsy’s Product Development with Continuous Experimentation”
Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI

13. Jon Jenkins, “Velocity Culture, The Unmet Challenge in Ops” 2011 | http://bit.ly/1vJo1Ya

14. do less
“Evaluating well-designed and executed
experiments that were designed to
improve a key metric, only about 1/3 were
successful at improving the key metric!”
“Online Experimentation at Microsoft” | Kohavi et al | http://stanford.io/130uW6X

15. “I think building this culture is the key to
innovation. Creativity must flow from
everywhere. Whether you are a summer intern
or the CTO, any good idea must be able to seek
an objective test, preferably a test that exposes
the idea to real customers. Everyone must be
able to experiment, learn, and iterate.”
http://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html

16. @jezhumble
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES
WORK IN
SMALL
BATCHES

17. higher quality
you can stop at any time with a working system
faster feedback (assuming people pay attention)
higher motivation
less rework
working in small batches
Don Reinertsen, Principles of Product Development Flow, ch5.

18. further reading
https://www.infoq.com/presentations/controlled-experiments
http://xp123.com/articles/resources-on-set-based-design/
Tom DeMarco & Tim Lister, Waltzing with Bears
Humble et al, Lean Enterprise ch 9
Gojko Adzic, Impact Mapping