The document discusses the essential building blocks for a successful experimentation program, emphasizing the importance of buy-in, process, and visibility. It outlines steps for enabling experimentation, defining success through metrics, and continuous optimization, while advocating for a culture that embraces A/B testing. The content also highlights various maturity levels of experimentation within organizations and the significance of effective communication and collaboration in driving experimentation efforts.

1. Nils Koppelmann
Building Blocks of a strong
Experimentation Program

2. "Yes, I think so too!
I just have the feeling that the topic of A/B testing is hopefully more and more
anchored in our culture. I think in the beginning it was a bit "annoying"
because it's extra work, but more and more I think everyone understands the
added value of it!
But on the anchoring of the culture we can hopefully soon work together
more intensively 🥳"
"Just a quick feedback, because we just call more often again, it's always fun
with you and your team!"

3. People running experiments
Agenda 01
2. The building blocks
3. 1 – Buy In
4. 2 – Process
5. 3 – Visbility
6. Experimentation Maturity
07 Takeaways

4. This is me… (or at least my LinkedIn profile)

5. Singular
Some
∞
Controlled or singular
A single person runs experiments, or experiments require
leadership approval.
Centralized team
A centralized team is responsible for running
experiments, often limited to marketing.
Can also be an agency.
Literally everyone
Everyone in the company is allowed to run
commission experiments.
How many people are running
experiments in your company?

6. Enable Drive Grow
Building Blocks
Three building blocks is all it takes.

7. Enable
Buy In Process Visibility
Drive Grow
Building Blocks
Three building blocks is all it takes.

8. Enable
Buy In

9. Buy In
→ reporting and meeting cadence
Think about:
• People
• Training
• Tools
• Agencies (possibly)
→ Goal tree map
01
Get initial buy in
Get levels of leadership to buy
into the idea of experimentation
02
Set clear goals
Communicate and align on
transparent goals of your
experimentation programme
03
Get budget
Without budget, there is no
experimentation programme
04
Report regularly
Create a reporting system that is relevant
to leadership and
management and supports your cause

10. “If you can't describe what you are doing as a
process, you don't know what you are doing”
– W. Edwards Deming

11. Drive
Process

12. Defining success
We define success based on
corporate goals and measurable
KPIs.
UX Audit &
Research
We identify bottlenecks, analyze the
website and user behavior with
qualitative and quantitative
methods.
Hypotheses -
Backlog
Formulate hypotheses to
positively influence user
behavior as well as metrics.
Prioritization of hypotheses.
Plan &
run tests
The test ideas with the highest
priority are included in the testing
roadmap, designed, implemented,
and tested. We also take care of
continuous monitoring of ongoing
tests.
Phase 1 Phase 2 Phase 3 Phase 4 Phase 5
Full process
Evaluation &
Implementation
We evaluate a completed test and
generate specific action
recommendations and learnings that
contribute to the prioritization of
further test ideas.
Process

13. Defining success
We define success based on
corporate goals and measurable
KPIs.
UX Audit &
Research
We identify bottlenecks, analyze the
website and user behavior with
qualitative and quantitative
methods.
Hypotheses -
Backlog
Formulate hypotheses to
positively influence user
behavior as well as metrics.
Prioritization of hypotheses.
Plan &
run tests
The test ideas with the highest
priority are included in the testing
roadmap, designed, implemented,
and tested. We also take care of
continuous monitoring of ongoing
tests.
Evaluation &
Implementation
We evaluate a completed test and
generate specific action
recommendations and learnings that
contribute to the prioritization of
further test ideas.
Phase 1 Phase 2 Phase 3 Phase 4 Phase 5
Full Process
Testingprocess
Process

14. A/B-Testing
Retainer Projects

15. Start
Q2
Discovery
Kickoff
Optimization Sprint
Optimization Sprint 1
Q3
Optimization Sprint 2 Optimization Sprint 3 Optimization Sprint 4
What the process looks like
Detailed project planning
Defining success
Getting setup, setting and
aligning on goals.
Technical and tracking setup.
Diving deep
We identify bottlenecks, analyze
the website and user behavior
with qualitative and quantitative
methods.
A road to test on
Formulate hypotheses to
positively influence user
behavior and metrics.
Prioritizing hypotheses to
create a testing and
optimization pipeline.
Experimenting
The highest ranked ideas get
designed, developed and tested
using both controlled
experiments and user testing.
Analyzing and reporting
We analyze completeded tests and
generate concrete
recommendations for action and
learnings that contribute to the
prioritization of further test ideas.
Onboarding
Kickstart
Discovery Planning Execution Analysis
Optimization Sprint
May June July August September

16. Start
Onboarding
• Document Handover
• Client Questionnaire
• Stakeholder Interviews
• Access Provisioning
What the process looks like
Individual work packages
Technical Setup
• Analytics Health Check
• Tracking Setup
• Tool Setup
• Reporting Setup
Kickoff Research
• Full Funnel Audit
• Conversion Audit
• Copy Teardown
• Deconstruction
• Heatmap/Recording Analysis
• Polls and Surveys on-site
• Customer Voice Analysis
• Behaviour Hypotheses
• User Testing
Onboarding Discovery
Operations
• Continuous optimization of the
customer journey
• Plan & execute in 2-week sprints
• Concept, design, development
• Pre-defined number of test p.m.
Optimization Sprint
Continuous Research
• Funnel Analysis
• Customer Voice Analysis
• Polls and Surveys on-site
• Heatmap/Recording Analysis
• Meta Analysis
• User Testing
Reporting
• Per experiment reporting
• Monthly reporting
• Weekly virtual Check-ins with the
team
• As needed meetings

17. Hypotheses
Your hypotheses and strategy are the
backbone of the success of your
program.
Automation
Automate away anything that keeps
people from doing value- add work.
Build your process on a strong foundation
+12.5%
-3.5%
+6.3%
+/-0.5%
Research
Create your test ideas based on sound
research. The better you understand
your customers, the better your ideas
will get. e.g. in Airtable
Documentation
Make sure your entire process is well
documented.

18. Client
Agency
Tool Landscape Example
Connecting services
event-based
Task
s
Notifications
Custom Analytics
Connector

19. Grow
Visibility

20. Visibility: Meetings and Touchpoints
Experimentation Weekly
Current experiments,
ideas and score cards
Strategy
💡New ideas
Questions &
Discovery
• Setting: weekly or bi-weekly
• bring new people in
• share the learnings with the
company
• encourage bottom-up

21. "Yes, I think so too!
I just have the feeling that the topic of A/B testing is hopefully more and more
anchored in our culture. I think in the beginning it was a bit "annoying"
because it's extra work, but more and more I think everyone understands the
added value of it!
But on the anchoring of the culture we can hopefully soon work together
more intensively 🥳"
"Just a quick feedback, because we just call more often again, it's always fun
with you and your team!"

22. Where are we actually?
Experimentation Maturity

23. Factors that contribute to Experimentation Maturity
Initial
Definition
Integration
Management
& Measurement
Optimization
01
02
03
04
05

24. Singular
Some
∞
Controlled or singular
A single person runs experiments, or experiments require
leadership approval.
Centralized team
A centralized team is responsible for running
experiments, often limited to marketing.
Can also be an agency.
Literally everyone
Everyone in the company is allowed to run
commission experiments.
How many people are running
experiments in your company?
01
02
03
04
05
00

25. Factors that contribute to Experimentation Maturity
Culture
A culture of experimentation that encourages risk-taking, hypothesis
testing, and continuous learning.
Resources
Sufficient resources, such as funding, time, and personnel, dedicated to
experimentation.
Infrastructure
The infrastructure and tools required to support
experimentation.
Methodology
A consistent and rigorous methodology for designing and conducting
experiments.
Integration
Integration of experimentation into decision-making processes.

26. Buy In
Enable
Process
Drive
Visibility
Grow
Takeaways
Take these home and your success is (almost) guaranteed.

27. Let's get into the Q&A
Thanks

28. Q&A

29. References
We work with eCom, SaaS and marketplaces