Run more experiments with fewer resources

1
Q4-2023
GAP Analyses.
Mark de Winter
Mark de Winter – Clickvalue - 29-1-2024

2
COMPANY’S
VELOCITY
TARGETS
CRO
RESOURCES

3
COMPANY’S
VELOCITY
TARGETS
CRO
RESOURCES
Problems when scaling
experimentation programs:
o Resources don’t grow at same pace as
(change) ambitions.
o Quality becomes an issue >10
experiments / month over multiple
teams.
o Cumbersome tasks are boring, and
demotivating.

5
Concepting & creation
Data & Analytics
Conversion
optimization
Online marketing
B2B market
Online marketing
B2C market
E-mail marketing &
marketing automation
24
CXO Experts (incl.
developers)
1000
AB tests / year
+15
Years of experience in
CXO
50
Countries

6
Our Mission:
“Growing our clients’ business by supporting them in
making better decisions on their customers’
experiences”

9
CXO Trends in 2024*
(UX) Research
✓ Sustainability
✓ Digital accessibility
✓ Customer centricity
✓ Predictive analysis
✓ Centralized insights
Solution Design
✓ Immersive experiences
✓ Mobile first CX
✓ Optimizing voice
interactions
✓ VR / AR
Experimentation
✓ Automation
✓ Exp. data quality &
Dashboarding
✓ Hyper personalization
PM / Strategy
✓ Scaling of experimentation
in organizations
✓ Leveraging AI
✓ Data privacy & trust
✓ Omnichannel harmony
*Sources: Contentsquare / CMSWire / + Clickvalue benchmarks

11
Step 1: A structured process.

12
“If you don’t have a process, you don’t know what you’re doing”
E. Deming

13
Step 1: Start with a structured process
1) Ad-hoc experimentation 2) Dedicated experimentation 3) Advanced experimentation 4) Embedded experimentation
Maturity state
Single touchpoint Multiple touchpoints Full customer journey All (marketing & product related)
Experimentation scope
1 person Small team(s) Multiple organized teams Embedded (no CXO team)
Teams
Basic Structured process Advanced / automated Democratized
Process
1: Start structuring the
program

14
Ensure a solid workflow

15
Templates Subtasks Sprint management

16
Step 2: Start automizing standardized
tasks
Maturity state
Teams
Process
2: Take your first
automation steps

17
Automated Program Dashboarding:
Filter out all results
and experiment data
Quickly find all
your reports
based on
advanced
filters
View results from whole
program or specific team(s)
Succesdata
Compare
success ratios
to understand
what works
and what not

AI to increase efficiency
Generate quick ideas or substantiate
existing ideas with generative
content or create your own
database for trend & meta-analyses
1. AI Ideation
Generative AI can help create
assumptions on how to interpret
data and create hypotheses.
2. AI enriched
opportunities
AI can help with inspiration and quickly
generate multiple versions (viewports /
devices) of existing designs
3. Faster design
AI can help to create assumptions on
how the data can be interpreted in user
behavior and attitude to generate follow
up ideas.
4. Rich validation

19
19
Example:
Help with
creating
assumptions
in ChatGPT
https://chat.openai.com/g/g-9CEOma1P1-cx-cognitive-insight
Prompt with: “What
can explain the
effect…”

21
Step 3: Automation by default
Maturity state
Teams
Process
3: Automation at the
core of the program

22
Experiment data management & monitoring:
Basic decision
support on
what to do.
Only show
the
experiments
you need
Splitwatch+ shows all
experiments and its relevant
data at one glance
See all
running
experiments
and select KPI
to call

23
Experiment database:
Summary of results +
option to download
full report
Build custom filtering
& dashboarding for
specific research topics
Multiple variables available
to easily search through it
A database
with all
experiments

24
Experiment database:
Job
Completion
Surveys Funnel
analysis
Return
rate
analysis
Customer
Journey
Mapping
A/B test
analysis
Landing
page traffic
analysis
Refined
opportunity
Moderated
usability
study
Expert
review
Competitor
scan
Prototype
test results
Funnel
analysis
Exit
survey
results
Preference
test results
1: One place for the
entire organisation to
log insights
2. A uniform way to
add new insights using
a clearly defined data
model & forms
3: The ability to easily find
and combine insights.

25
Team training:
Individual training
tracks: Analyst (Asana)
Knowledge base in
Document360

26
26
Example: AI
customer
database

27
CXO Ecosystem:
Research
inspiration
Better predictions
due to prior data
& insights
Design
inspiration
Automated Program
Output overview
Splitwatch+
automated data &
monitoring tool
Automated
reporting
Structured
Opportunity’s,
Areas, Solutions
and their
relations (OST)
Faster “meta-
analyses”
Experiment repository stores all
important experiment data

28
Summary
Maturity state
Teams
Process
1: Start structuring the
program

29
Summary
Maturity state
Teams
Process
2: Take your first
automation steps

30
Summary
Maturity state
Teams
Process
3: Automation at the
core of the program

31
Benefits of automation in experimentation
√ Scaling will become easier and less resource heavy
√ Less quality loss due to automated data-prep and decision support
√ Better, clearer overview of all CXO output, results, impact and ROI
√ Save time for deeper analyses, designs, interpretations and follow ups.
√ More fun! As repetitive tasks are mostly automated.
But
➢ Beware of rubbish in – rubbish out: Prioritize clear process standards
➢ Standardisation v.s. customization
➢ AI isn’t trustworthy (yet)

32
COMPANY’S
VELOCITY
TARGETS
CRO
RESOURCES

Run more experiments with fewer resources

Recommended

Recommended

More Related Content

Similar to Run more experiments with fewer resources

Similar to Run more experiments with fewer resources (20)

More from VWO

More from VWO (20)

Recently uploaded

Recently uploaded (20)

Run more experiments with fewer resources