Product development is inherently risky. While lean and agile methods are praised for supporting rapid feedback from customers through experiments and continuous iteration, teams could do a lot better at prioritizing using basic modeling techniques from finance. This talk will focus on quantitative risk modeling when developing new products or services that do not have a well understood product/market fit scenario. Using modeling approaches like Monte Carlo simulations and Cost of Delay scenarios, combined with qualitative tools like the Lean Canvas and Value Dynamics, we will explore how lean innovation teams can bring scientific rigor back into their process.

1. Stop Flying Blind!
Quantifying Startup Product Risks with Monte Carlo Simulation.
Sam McAfee, Principal - @sammcafee

2. Slides available at
bit.ly/riskiest_assumption

3. Who is this guy?
• Dot-com boom survivor.
• Ran a dev shop for 10 years after that.
• Agile has always been the “norm”.
• Read ‘4 Steps’ before #LeanStartup.
• Participated in 3 different startups. All dead.
• Ran Change.org engineering.
• Learned LeanUX from @clevergirl.
• Wore plaid and facial hair before it was cool.
Tweet every word: @sammcafee

4. A Typical Scenario
Maybe you’ve experienced something like this?

5. A typical scenario
The CEO

6. “ ”
Our competitors
have it, so we should
have it too!

7. A typical scenario
Product Management

8. It takes too long to
get new features out. “ ”

9. A typical scenario
Engineering

10. “ ”
We have to refactor
all the things we rushed
out the door last time.

11. How do we prioritize
competing objectives?

12. Understanding Risk
It’s kind of important for startups.

13. Understanding Risk
Why Is Risk Important?

14. Startups are risky!
You are participating in one of the highest risk
business endeavors there is.
Risk is a business term.
You are in business. Let’s use terminology that
business people use to make decisions.
Ignore it at your peril.
Choosing to ignore risk will not protect you from it.

15. Understanding Risk
What Is Risk?

16. Risk =
Impact * Probability

17. Understanding Risk
Stop Using Your Gut!

18. “ ”
All people, including
experts and managers,
are very bad at assessing
the probabilities of events.
— Doug Hubbard

19. Risk =
Impact * Probability

20. How Do We
Define Impact?
Hint: For startups, it can probably be found on the bottom line.

21. Economic framework.
Allows you to compute the impact of any change in
the system into a single unit of measure.
Start with your P&L.
Your CEO or finance people already have a
framework for you to start with. Include them.
Apply risk scenarios.
Calculate the impact of likely scenarios on the total
product life-cycle profits, or similar KPI.

22. Scenario 1
Cost of Delay

23. Cost of Delay Scenarios
• Projects will have different cost of delay curves.
• What is the effect on total life-cycle profits for each unit of delay?
• Delay in product development is overwhelmingly affected by
queues between steps rather than job duration at a given step.
• Cost of Delay enables you to calculate the cost of queues.

24. Scenario 2
Sequence of Work

25. Work Sequence Scenarios
• The order in which you do work can have a substantial impact on
the total cost of queues.
• If cost of delay is homogeneous, do the shortest job first.
• If job duration is homogeneous, do the highest cost of delay first.
• You can combine job duration and cost of delay using weighted
shortest job first.

26. Tangent: which is the shortest job?
• Job duration is an estimate.
• Estimates are probabilistic, not deterministic.
• Use cycle time and throughput to compute a probability
distribution for likelihood of job duration.

27. Scenario 3
Capacity vs. Queue Cost

28. Capacity Utilization Scenarios
• You can reduce the cost of queues by adding additional capacity.
Obviously, additional capacity has a cost too.
• Additional capacity increases transaction costs, but lowers
holding (delay) costs.
• Is the cost of adding additional capacity justified by the gains in
product development throughput and lower queue costs?

29. How Do We Define
Probability?
Use your metrics, Luke!

30. Use business metrics.
You are probably already tracking tons of data about
how you acquire new customers and how they enter
and leave your product funnels.
Value each conversion.
Use your historical data to calculate values for each
step in a product funnel.
Note areas of variance.
The steps in your funnel that have consistent,
regular conversion rates are unlikely to change.
They represent lower information content.

31. How Do We Value
Information?
Hint: It’s probabilistic too.

32. Information Has Value
• 10,000 visitors, 10% sign-up, $20/month.
• Each visitor is worth $2/month (10% x $20/month).
• A test to increase rate to 15% is worth:
• An additional $10/month * probability of test success.

33. Experiments Have Value
• Experiments attempt to capture new information.
• Experiment value = expected benefit - cost of running the test.
• Expected benefit = increase in KPI * probability of success.
• Cost of running the test = cost of delay * job duration.

34. Your backlog is a
prioritized list of
experiments.

35. Monte Carlo Simulations
It’s not just for insurance geeks anymore!

36. Monte Carlo Simulation
• A series of dependent variables, each with probability
distributions.
• Randomly selects a value from each variable, and computes
output. Repeats thousands of times.
• End result is another probability distribution, in the unit of
measure that you care about.
• It’s 2014. You can do this in a spreadsheet, in about an hour.

37. Basic funnel
Using time series data from your funnel, create
probability distributions for each step toward
capturing revenue.
Basic histogram
After 5,000 or so simulations, display a histogram of
the expected output.
1
2
1
2

38. References
+

39. Thank you!

40. Slides available at
bit.ly/riskiest_assumption