Go behind the curtain with WiderFunnel's Director of Strategy, Nick So, as he shares exclusive learnings from his experience building successful experimentation programs for companies like HP, Asics, Magento, and IBM. During this webinar, you’ll get an insider look at the frameworks and strategies that WiderFunnel developed to help clients scale experimentation and accelerate through the phases of maturity.

1. Profitable ‘A-ha!’ moments everyday.™

2. Scaling Experimentation: The 5 Key Pillars of Maturity
Nick So, Director of Strategy at WiderFunnel

3. ● Use the general chat panel to ask questions or make
comments - these will be answered during the Q&A segment
at the end of the presentation
● This presentation deck and recording will be available to you after
the webinar
Before we get started:

4. • Oversees the team dedicated to the ideation, design,
development, and execution of experiments
• Leads the experimentation research and strategy for several
of WiderFunnel’s Fortune 500 clients
• Has spoken at several conferences including eduWeb and
eCommerce Summit Dallas
Nick So, Director of Strategy, WiderFunnel

5. First, an introduction...

6. Conversion Rate Optimization

7. Experimentation

8. Experimentation fuels the world’s leading companies.
“Our success is a function of
how many experiments we
do per year, per month, per
week, per day.”
Jeff Bezos
“Instead of saying, ‘I have
an idea’, what if you said,
‘I have a new hypothesis,
let’s go test it’”.
Satya Nadella
“Our company culture
encourages experimentation
and the free flow of ideas.”
Larry Page

9. © 2007-2018 WiderFunnel Marketing Inc.

10. © 2007-2018 WiderFunnel Marketing Inc.

12. Most organizations are only scratching the surface when it comes to
experimentation maturity.

13. How do you actually achieve
experimentation maturity?

14. A quick story
A client’s journey from just getting started to ongoing,
scalable, organizational experimentation program.

15. The situation when we started
▪ High volume website with a one-person optimization team.
▪ Mish-mash of technologies for digital experimentation:
Ad server, Optimizely, and Adobe Analytics.
▪ ‘Hunt and peck’ optimization planning.
▪ 4-8 experiments per month.
▪ Little confidence in test results.
▪ Unknown optimization opportunity.

16. They needed:
A growth program that produces a never-ending
stream of profitable insights.

17. Here’s where DMV.org is now
▪ The right process – the Infinity Optimization Process.
▪ The right velocity – now running 500+ experiments per year.
▪ The right results – revenue doubled four years in a row.
▪ The right team – integrated DMV.org and WiderFunnel teams.
▪ The right tech – with Adobe and Optimizely integrated.

18. So...where do I start?

19. The 5 Pillars of Experimentation Maturity
Process
Accountability
Culture
Expertise
Technology

20. One of the most difficult pillars to strengthen:
Process

21. Source: https://www.gartner.com/technology/research/methodologies/hype-cycle.jsp

23. Use landing page
templates
Follow conversion
“best practices”
Choosing tools before
strategy / tool-driven
testing.
No experiments
Start personalization without a
personalization strategy
Misunderstand statistical
significance
Pollute test results with improper
technical setup
Little cognitive
bias training
Unaware of Design of
Experiments (DOE)

24. The most common mistake:
Implementing “gut feeling” ideas without
framework thinking or experiment validation.

25. Even in the most mature
experimentation
organizations, optimizers say
experiments are being
derailed by gut-feeling.
Source: “State of Experimentation Maturity 2018” research study by WiderFunnel, widerfunnel.com/research

27. Hype phase
▪ Tool-triggered testing
▪ Ad-hoc testing
▪ Low-hanging fruit
▪ ‘Best practices’ testing
▪ Isolated testing team
▪ Concluding tests before 7 days or
running for 6+ weeks

28. ▪ Initial wins from low-hanging fruit
▪ Reporting results based on
secondary conversion metrics
▪ Interest and hype from other
departments
▪ Excitement to scale rapidly

29. ▪ Logjam of ideas
▪ No idea where or what to test first
▪ Inability to scale
▪ Diminishing wins and ROI

30. “We had been struggling with doing optimization
ourselves internally and with other partners but
getting no meaningful results. We needed help
from someone who could bring a structured
approach to our optimization efforts.”

31. Avoid the trough of disillusionment:
The importance of framework thinking.

32. Framework Thinking:
Frameworks are organized structures of thinking that allow individuals to think
and act for themselves, while collaborating with others to solve problems or
achieve common goals and objectives.

34. The Infinity Optimization Process

35. The Experimentation Mindset

36. Yin
Creative
Inspired
Fuzzy
QualitativeYang
Logical
Proven
Solid
Quantitative

37. The Infinity Optimization Process

38. The LIFT Model

41. The right Value Proposition
communicated with Clarity
in a context of Relevance
avoiding Anxiety and Distraction
and adding high Urgency.

42. CLARITY: Headline is long, wrapping 3
lines.
RELEVANCE: “Try the Demo” – not
referenced in headline on demo page.
DISTRACTION: Unnecessary copy
persuading visitors to “try the demo”.
CLARITY: Details of what the demo
includes hidden within the block of copy.
URGENCY: No mention of “instant
access”
CLARITY: No mention of “Free” on the
page.
LIFT Analysis

43. LIFT Analysis
DISTRACTION: “Try the Demo” CTA still
at the top of the page.
DISTRACTION: Unnecessary text block.
CLARITY: First form field is not
auto-focused.
CLARITY: Are all fields required?
CLARITY: “Submit” is the worst possible
CTA button copy.
RELEVANCE: Inconsistent CTA design
compared to previous step.

44. Winning merged page design

45. How do I decide where to start?
It’s as easy as PIE

47. PIE Prioritization Framework
Potential
How much
improvement can be
made?
Ease
How easy would it be
to actually make these
improvements?
Importance
How much impact can
these improvements
have on our primary
business goal?

48. “Zone” prioritization: Where to experiment
Lift Zone #1 8 7 7 7.3
Lift Zone Potential Importance Ease PIE Score
Web analytics
Heuristic & first impression analysis
Voice of customer
Revenue impact
Experimentation velocity
Cost of acquisition
Technical
“Political”

49. Hypothesis prioritization: What to experiment on
Hypothesis #1 3 7 9 6.3
Hypothesis Potential Importance Ease PIE Score
Evidence from Digital Analytics
Evidence from User Testing & Research
Based on Persuasion Principles
Conversion KPI
Revalidates Past Insight
Informs Business Insight
Technical
“Political”

50. Business Prioritization
Inputs:
▪ Unique visitors / users
▪ Identified conversion goal
▪ Conversion rate
▪ Value per conversion

51. Business Prioritization
Inputs:
▪ Unique impressions
▪ Identified conversion goal
▪ Conversion rate
▪ Value per conversion

52. Achieving organizational experimentation maturity

53. The 5 Pillars of Experimentation Maturity
Process
Accountability
Culture
Expertise
Technology

54. Process
Accountability
Culture
Rigorous process guides
ideation, prioritization,
execution & analysis.

55. 90% of organizations at the
“Scaling” maturity level ran 21+
experiments last year.
These organizations are more likely to
have a dedicated optimization team, a
more rigorous experimentation process.
Source: “State of Experimentation Maturity 2018” research study by WiderFunnel, widerfunnel.com/research

56. Accountability
Accountability
Culture
Experimentation strategy and
metrics are directly linked to
key company metrics.

57. Executive teams at 63% of
more mature organizations
are involved in
metric-setting for
experimentation, pointing
to Senior-level buy-in.
Source: “State of Experimentation Maturity 2018” research study by WiderFunnel, widerfunnel.com/research

58. Culture
Accountability
A culture that values every
experiment (win or lose) as a source
of quality insights and results.

59. 100% of more mature organizations rely on analytics and on insights from previous
experiments to develop new experiment hypotheses.
Source: “State of Experimentation Maturity 2018” research study by WiderFunnel, widerfunnel.com/research

60. Technology
Accountability
Right tech stack and set
development process for
testing & implementation.

61. Source: “State of Experimentation Maturity 2018” research study by WiderFunnel, widerfunnel.com/research
When building a team for
experimentation,
organizations at every
maturity level are highly
focused on Web
Development and QA.

62. Expertise
Accountability
Experimentation is a shared
company priority, with dedicated
people & resources.

63. Source: “State of Experimentation Maturity 2018” research study by WiderFunnel, widerfunnel.com/research
While Web Development
and QA skills are a priority
for every organization
surveyed, organizations at
the “Scaling” maturity level
are equally focused on
experimentation strategy.

64. Optimization is a shared company
priority, with dedicated people &
resources.
Rigorous process guides
ideation, prioritization,
execution & analysis.
Foundations for Experimentation Maturity
Optimization strategy and
metrics are directly linked to
key company metrics.
Right tech stack and set
development process for
testing & implementation.
Defined methodology &
protocol of experimentation
agreed across the organization.
A culture that values every
experiment (win or lose) as a source
of quality insights and results.

65. ▢ Conversion metrics have direct impact on
revenue or on major business goals.
▢ There is a dollar value attributed to a
successful conversion.
▢ Conversion metrics are prioritized in
order of importance to business.
▢ Conversion “success” is clearly defined.
▢ Overall experimentation program success
is linked to operational metrics.
Checklist: Identify impactful success metrics

66. ▢ A system enabling employees to submit
hypotheses and test ideas.
▢ Set objective evidence factors for
hypothesis prioritization.
▢ List of agreed upon conversion metrics.
▢ Hypotheses prioritization framework in
place and understood.
Checklist: Idea & Hypothesis Gathering

67. Checklist: Results Sharing & Dissemination
▢ Executive summary reports and quarterly
reviews.
▢ Accessible real-time data reports.
▢ Posted results reports or progress graphs
displayed openly.
▢ Internal results and insights database.
▢ Cross-departmental results & insights
presentations.

68. “Experimentation is important because
change is the only constant. Whether
it’s consumer behavior or new technology,
things are always dynamic in our world.
You can never assume that what worked
yesterday will work tomorrow.”
- Gretchen Gary, Product Manager

69. Q&A

70. Get the Recording:
widerfunnel.com/hello/scaling-experimentati
on-webinar/
Thank you!

71. Profitable ‘A-ha!’ moments everyday.™