Featuring speakers from The Next Web, Moz
Martijn Scheijbeler, Lead Growth, SEO & Analytics, The Next Web
Andy Nelson, Director of Growth Marketing, Moz
Khattaab Khan, Strategic Consultant, Optimizely
If you’re running A/B tests consistently, you’re already on the right track. But how can you take the next step and turn your testing efforts into a well-oiled growth engine that scales? Moz, charity: water, and The Next Web have all achieved tremendous success within their optimization programs and have figured out how to effectively run hundreds of experiments a year. Learn how these companies increased the impact of their testing programs by leveraging tools and processes that scale.
Cybersecurity Awareness Training Presentation v2024.03
Opticon 2015-Scaling Your Testing Program for Maximum Impact
1. #opticon2015
Scaling your Testing Program
for Maximum Impact
Andy Nelson
Director of Growth
Marketing, Moz
@theandynelson
andy.nelson@moz.com
Martijn Scheijbeler
Lead Growth, SEO &
Analytics, The Next Web
@martijnsch
martijn@thenextweb.co
m
Khattaab Khan
Strategy Consultant,
Optimizely
@khattaabkhan
khattaab@optimizely.com
2. Increase Your Impact
Without More Resources
Andy Nelson
Director of Growth Marketing, Moz
@theandynelson
andy@moz.com
#opticon2015
11. Elements of good prioritization
Core
• Potential impact to success metric
• Estimated runtime required
• Estimated level of effort
12. Elements of good prioritization
Modifiers
• Alignment with strategy
• Support from VOC, analytics, prior
testing
• Urgency
Core
• Potential impact to success metric
• Estimated runtime required
• Estimated level of effort
33. #opticon2015
How The Next Web’s Testing
Program Scaled
Martijn Scheijbeler
Lead Growth, SEO & Analytics - The Next Web
@martijnsch
martijn@thenextweb.com
34. Yearly 5.000 attendees, 75.000 products sold.
Monthly 6 million unique users, 12 million pageviews.
The Next Web: 2006 > 2015
39. What did we learn along the way?
• Swim lanes, KPIs, Goals.
• Segments: traffic sources, etc.
• Idea creation: backlog + prioritisation.
• Scaling > Analysing results.
• Do it live!
41. New v.s. Returning - Europe v.s. USA - Facebook v.s. SEO - Mobile v.s. Desktop
Type, Regions, Source, Device
42. Idea creation v.s. Planning
• Different templates + segments.
• How much time per test? Run tests for a fixed predefined time.
• The maximum of your program.
• Take away: create a backlog!
54. Scaling our testing program to the next level?
• Even more templates, across more projects.
• 25+ tests a month with current growth of traffic.
• Bigger team? We can’t keep up with coding ;-)
• More automation? Push test variant to GTM.
• Mix it up with personalisation/ behaviour.
• Push our experiments to Slack for sharing.
• Create a better style guide + Bootstrap guidelines.
55. Summary & Take aways
• Know your KPI framework, expectations and limitations.
• Know your segments, CR% so you know how to prioritise.
• Follow your planning, priorities.
• Scale = 40% - 50% automation: analyse fast!
• Be able to push code live!
56. #opticon2015
Have fun & Thank you!
Martijn Scheijbeler
Lead Growth , SEO & Analytics - TNW
@martijnsch
martijn@thenextweb.com
57. Strategies Essential to Achieve
Testing Program Scale
#opticon2015
Khattaab Khan
Strategy Consultant, Optimizely
@khattaabkhan
khattaab@optimizely.com
60. 44% 42%
8%
52%
87%
81%
41%
31%
44%
32%
11%
12%
15%
27%
48%
16%
7%
Homepage to
Category
Category to Product Product to Shopping
Cart
Shopping Cart to
Checkout
Checkout to
Payments
Payments to
Conversion
Continue down
funnel
Visit different
page
Exit website
X
A
M
P
LE
Homepage
Category
Page
Product
Page
Shopping
Cart
Checkout
Page
Payment
Page
With a conversion funnel, we could analyze all steps
to see how and where users drop offEvaluate User Path
61. 44% 42%
8%
52%
87%
81%
41%
31%
44%
32%
11%
12%
15%
27%
48%
16%
7%
Homepage to
Category
Category to Product Product to Shopping
Cart
Shopping Cart to
Checkout
Checkout to
Payments
Payments to
Conversion
Continue down
funnel
Visit different
page
Exit website
X
A
M
P
LE
Homepage
Category
Page
Product
Page
Shopping
Cart
Checkout
Page
Payment
Page
With a conversion funnel, we could analyze all steps
to see how and where users drop off
Focus Testing on Problem Areas that are Drivers
of Business Value
65. Custom Prioritization Scoring
Pauline Marol
Lead PM, Site Optimization
“I don’t want to be
the one making
decisions on what
experiments I run
and don’t run
because I am
biased.”
66. Results Analysis
Results Page Best Practices
• Evaluate opportunity costs
• Analyze segment performance
• Shareable results link
67. Results Sharing
Communicating Results + Roadmap Iteration
• Hold team accountable to strategy that inspired variation design
• Validate/invalidate hypothesis to evaluate goal effectiveness
• Define where results output will live (e.g. database, wiki, etc.)
69. Simultaneous Testing Use Cases
Experiments on same page
Experiments on same flow
(e.g. checkout / registration flow)
Multivariate experiment
70. Beware of Interaction Effects
Elements on same page
Elements on subsequent pages
in a funnel
When multiple points of user interaction influence the
same goal, interaction effects can taint data
73. Evolution into
Optimization Center of Excellence
MEASURE
RESULTS
Analytics
Dev + QADesign
Program
Manager
Product
Manager
Product
Manager
Dedicated optimization
resources serving
multiple BUs
74. Leverage data to focus on goals that directly
drive business value
Hold your team accountable with
effective planning documents
Know when to run simultaneous tests
Scale your program structure to support
multiple business units
Takeaways
76. #opticon2015
Andy Nelson
Director of Growth
Marketing, Moz
@theandynelson
andy.nelson@moz.com
Martijn Scheijbeler
Lead Growth, SEO &
Analytics, The Next Web
@martijnsch
martijn@thenextweb.co
m
Khattaab Khan
Strategy Consultant,
Optimizely
@khattaabkhan
khattaab@optimizely.com
Now for some Q&A