The document emphasizes the importance of experimentation in product development, highlighting its impact on decision-making and outcomes through measurements over time. It outlines steps to cultivate a strong experimentation culture, advocating for distributed decision-making, hypothesis-driven approaches, and the use of effective tools to enhance data accessibility. Lastly, it illustrates the progression of experimentation from small-scale initiatives to larger, impactful endeavors across teams.

4. Why experiment?

5. Why experiment?
Where to?

6. The impact of experimenting cannot be understated
HAS DESIRED
IMPACT
SHOULD BE
TURNED OFF
REQUIRES
ITERATION
Source: Vineeth Madhusudanan, Statsig
Your product metric over time

7. Shipped feature (X)
Shipped feature (Y)
Shipped feature (Z)
Source: Vineeth Madhusudanan, Statsig
Your product metric over time
HAS DESIRED
IMPACT
SHOULD BE
TURNED OFF
REQUIRES
ITERATION
The impact of experimenting cannot be understated

8. HAS DESIRED
IMPACT
SHOULD BE
TURNED OFF
REQUIRES
ITERATION
Shipped feature (X)
Didn’t ship
feature (X)
Didn’t ship
feature (Y)
Didn’t ship
feature (Z)
Shipped feature (Y)
Shipped feature (Z)
Source: Vineeth Madhusudanan, Statsig
Your product metric over time
The impact of experimenting cannot be understated

9. What does good look like?

11. What makes for a great experimentation culture?
Data is seen as empowering
Outcomes measured on both a short and long-term time horizon
Distributed decision-making at scale
01
02
03

12. What makes for a great experimentation culture?
Distributed decision-making at scale
01

13. What makes for a great experimentation culture?
Outcomes measured on both a short and long-term time horizon
Distributed decision-making at scale
01
02

14. What makes for a great experimentation culture?
Data is seen as empowering
Outcomes measured on both a short and long-term time horizon
Distributed decision-making at scale
01
02
03

15. statsig.com
BUT. Great experimentation cultures
aren’t built overnight.

16. statsig.com
Source: Vineeth Madhusudanan, Statsig
Walk
Choose your tools,start
measuring impact
Jog
Individual team experimentation
culture develops
Run
10x experiments
across many teams
Fly
Empowered, distributed
execution at scale
Start small…

17. Walk
Choose your tools,start
measuring impact

18. Build your data baseline

19. A good experimentationtool
will…
Scale your Data Science team
Create the right hooks and
daily/weekly/monthlybehaviors
Democratizeaccessto
experimentation
Choose & implement tooling that automates data-
driven development

20. Jog
Individual team experimentation
culture develops

21. Gate every feature; only launch your winners
Shipped feature (X)
Didn’t ship
feature (X)
Didn’t ship
feature (Y)
Didn’t ship
feature (Z)
Shipped feature (Y)
Shipped feature (Z)
Source: Vineeth Madhusudanan, Statsig
Your product metric over time

22. Introduce hypothesis-driven experimentation

23. Run
10x experiments
across many teams

24. The team is 10x-ing the # of experiments run
REAL STATSIG CUSTOMER’S
EXPERIMENTATION GROWTH
FROM FALL ‘22 TO SPRING ‘24

25. Data informs every part of the product development cycle
STATSIG
VIRTUOUS CYCLE
Analytics
Increase the number of
insights and feature ideas
Feature Flags &Experiments
Grows the number of features being
built, experimented with, and released
in a managed way
Metrics &Events
Grows the amount of product
data to analyze for insights
and feature ideas

26. Fly
Empowered, distributed
execution at scale

27. Holdouts measure long-term impact

28. statsig.com
And you’re off to the races!
Source: Vineeth Madhusudanan, Statsig
Walk
Choose your tools,start
measuring impact
Jog
Individual team experimentation
culture develops
Run
10x experiments
across many teams
Fly
Empowered, distributed
execution at scale