Featuring speakers from The Next Web, Moz Martijn Scheijbeler, Lead Growth, SEO & Analytics, The Next Web Andy Nelson, Director of Growth Marketing, Moz Khattaab Khan, Strategic Consultant, Optimizely If you’re running A/B tests consistently, you’re already on the right track. But how can you take the next step and turn your testing efforts into a well-oiled growth engine that scales? Moz, charity: water, and The Next Web have all achieved tremendous success within their optimization programs and have figured out how to effectively run hundreds of experiments a year. Learn how these companies increased the impact of their testing programs by leveraging tools and processes that scale.

1. #opticon2015
Scaling your Testing Program
for Maximum Impact
Andy Nelson
Director of Growth
Marketing, Moz
@theandynelson
andy.nelson@moz.com
Martijn Scheijbeler
Lead Growth, SEO &
Analytics, The Next Web
@martijnsch
martijn@thenextweb.co
m
Khattaab Khan
Strategy Consultant,
Optimizely
@khattaabkhan
khattaab@optimizely.com

2. Increase Your Impact
Without More Resources
Andy Nelson
Director of Growth Marketing, Moz
@theandynelson
andy@moz.com
#opticon2015

4. ~2M visits/month

5. #1 Go Big

6. Evolution vs. Revolution

8. Quality > Quantity

9. #2 Prioritize Intelligently

11. Elements of good prioritization
Core
• Potential impact to success metric
• Estimated runtime required
• Estimated level of effort

12. Elements of good prioritization
Modifiers
• Alignment with strategy
• Support from VOC, analytics, prior
testing
• Urgency
Core
• Potential impact to success metric
• Estimated runtime required
• Estimated level of effort

13. http://bit.ly/TestPrioritization

14. #3 Swim Lanes

16. Homepage
Blog
Conversion Funnel
In Product

17. Visitors
1-25
Visitors
26-50
Visitors
51-75
Visitors
76-100
Homepage

19. #4 Multiple Streams of Work

21. 1 week
1 day1 day

22. Checkout Optimization
Product Page Optimization
Test Analyze
Design/Build
Test 1
Test 2
Test 3
Test 4
Test Analyze
Design
Test
Design/Build

23. #5 Prevent Busted Tests
(or at least catch them early)

25. Chrome
Safari
IE
Firefox

26. Chrome
Safari
IE
Firefox
BUSTED

27. Have a process
Pre-launch
QA
Final
Sign-Off
Post-Launch
QA
Weekly
Check-Up

28. Test across browsers/devices

29. 0
22500
45000
67500
90000
5/3 5/10 5/17 5/24 5/31 6/7
Watch for irregular events

30. Test big, bold changes
Improve your prioritization
Run multiple tests
Eliminate dead time
Improve your QA
Takeaways

31. http://bit.ly/TestPrioritization

32. #opticon2015
Thank you!
Andy Nelson
Director of Growth Marketing, Moz
@theandynelson
andy@moz.com

33. #opticon2015
How The Next Web’s Testing
Program Scaled
Martijn Scheijbeler
Lead Growth, SEO & Analytics - The Next Web
@martijnsch
martijn@thenextweb.com

34. Yearly 5.000 attendees, 75.000 products sold.
Monthly 6 million unique users, 12 million pageviews.
The Next Web: 2006 > 2015

36. What’s happened in the last 12 months?
Scaling our testing program

37. That clearly wasn’t good enough
1 test a month :(

38. Currently 5-6 tests a week, how did we get there?
15-20 tests a month :)

39. What did we learn along the way?
• Swim lanes, KPIs, Goals.
• Segments: traffic sources, etc.
• Idea creation: backlog + prioritisation.
• Scaling > Analysing results.
• Do it live!

40. Unique Users, Page Views, Emails & Revenue.
Goals: UU, PV, Emails & $
Related stories, pagination, sharing, email signups.
KPI: CTR, Engagement, Funnels

41. New v.s. Returning - Europe v.s. USA - Facebook v.s. SEO - Mobile v.s. Desktop
Type, Regions, Source, Device

42. Idea creation v.s. Planning
• Different templates + segments.
• How much time per test? Run tests for a fixed predefined time.
• The maximum of your program.
• Take away: create a backlog!

43. The part that we can’t scale yet

44. Analysing > Scaling!

45. Well, this is both the most boring and most exciting part!
1-2 main KPIs, 15 secondary KPIs

46. Rails, Google Analytics API
Built with love by the TNW Marketing team.
TNW Analytics

47. TNW Analytics was born!

48. Adding a new idea

49. Schedule

50. Test

51. Saving 20% of time per test!

52. Take away: Be able to push code!

53. What’s next?

54. Scaling our testing program to the next level?
• Even more templates, across more projects.
• 25+ tests a month with current growth of traffic.
• Bigger team? We can’t keep up with coding ;-)
• More automation? Push test variant to GTM.
• Mix it up with personalisation/ behaviour.
• Push our experiments to Slack for sharing.
• Create a better style guide + Bootstrap guidelines.

55. Summary & Take aways
• Know your KPI framework, expectations and limitations.
• Know your segments, CR% so you know how to prioritise.
• Follow your planning, priorities.
• Scale = 40% - 50% automation: analyse fast!
• Be able to push code live!

56. #opticon2015
Have fun & Thank you!
Martijn Scheijbeler
Lead Growth , SEO & Analytics - TNW
@martijnsch
martijn@thenextweb.com

57. Strategies Essential to Achieve
Testing Program Scale
#opticon2015
Khattaab Khan
Strategy Consultant, Optimizely
@khattaabkhan
khattaab@optimizely.com

58. Focus on Optimization
Opportunities that Matter

60. 44% 42%
8%
52%
87%
81%
41%
31%
44%
32%
11%
12%
15%
27%
48%
16%
7%
Homepage to
Category
Category to Product Product to Shopping
Cart
Shopping Cart to
Checkout
Checkout to
Payments
Payments to
Conversion
Continue down
funnel
Visit different
page
Exit website
X
A
M
P
LE
Homepage
Category
Page
Product
Page
Shopping
Cart
Checkout
Page
Payment
Page
With a conversion funnel, we could analyze all steps
to see how and where users drop offEvaluate User Path

61. 44% 42%
8%
52%
87%
81%
41%
31%
44%
32%
11%
12%
15%
27%
48%
16%
7%
Homepage to
Category
Category to Product Product to Shopping
Cart
Shopping Cart to
Checkout
Checkout to
Payments
Payments to
Conversion
Continue down
funnel
Visit different
page
Exit website
X
A
M
P
LE
Homepage
Category
Page
Product
Page
Shopping
Cart
Checkout
Page
Payment
Page
With a conversion funnel, we could analyze all steps
to see how and where users drop off
Focus Testing on Problem Areas that are Drivers
of Business Value

62. Intelligent Documentation
for
Intelligent Prioritization

63. Roadmap

64. Roadmap + Resource Planning

65. Custom Prioritization Scoring
Pauline Marol
Lead PM, Site Optimization
“I don’t want to be
the one making
decisions on what
experiments I run
and don’t run
because I am
biased.”

66. Results Analysis
Results Page Best Practices
• Evaluate opportunity costs
• Analyze segment performance
• Shareable results link

67. Results Sharing
Communicating Results + Roadmap Iteration
• Hold team accountable to strategy that inspired variation design
• Validate/invalidate hypothesis to evaluate goal effectiveness
• Define where results output will live (e.g. database, wiki, etc.)

68. Simultaneous Testing

69. Simultaneous Testing Use Cases
Experiments on same page
Experiments on same flow
(e.g. checkout / registration flow)
Multivariate experiment

70. Beware of Interaction Effects
Elements on same page
Elements on subsequent pages
in a funnel
When multiple points of user interaction influence the
same goal, interaction effects can taint data

71. Evolution of
Testing Org Structure

72. Core Testing Team
MEASURE
RESULTS
Analytics
Dev + QADesign
Program
Manager
CRO Analyst
optional

73. Evolution into
Optimization Center of Excellence
MEASURE
RESULTS
Analytics
Dev + QADesign
Program
Manager
Product
Manager
Product
Manager
Dedicated optimization
resources serving
multiple BUs

74. Leverage data to focus on goals that directly
drive business value
Hold your team accountable with
effective planning documents
Know when to run simultaneous tests
Scale your program structure to support
multiple business units
Takeaways

75. #opticon2015
Thank you!
Khattaab Khan
Strategy Consultant, Optimizely
@khattaabkhan
khattaab@optimizely.com

76. #opticon2015
Andy Nelson
Director of Growth
Marketing, Moz
@theandynelson
andy.nelson@moz.com
Martijn Scheijbeler
Lead Growth, SEO &
Analytics, The Next Web
@martijnsch
martijn@thenextweb.co
m
Khattaab Khan
Strategy Consultant,
Optimizely
@khattaabkhan
khattaab@optimizely.com
Now for some Q&A